Information retrieval methods, model training methods, devices, equipment and storage media

Abstract

This application discloses an information retrieval method, model training method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring input information, where the information modality of the input information is a first modality; calling an event representation prediction model to predict and process the input information to obtain an input event representation; calculating the event similarity between the input event representation and the retrieved event representation, where the information modality of the retrieved information is a second modality; and determining the retrieved information corresponding to the retrieved event representation as the retrieval result of the input information when the event similarity exceeds a similarity threshold. This application, by acquiring the input event representation and the retrieved event representation, fully extracts the semantic information from both the input and retrieved information; by comparing the differences between the input event representation and the retrieved event representation, it avoids discrepancies between the input and retrieved information caused by different information modalities, thus improving the retrieval effect across modal information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an information retrieval method, a model training method, a device, an equipment, and a storage medium. Background Technology

[0002] Text-based video retrieval refers to searching for video information related to text information, thereby achieving cross-modal search between text and video information.

[0003] In related technologies, video depth features of video information are typically extracted using a video feature extractor, and text depth features of text information are extracted using a text feature extractor. The extracted video depth features and text depth features are then used to train a video-text matcher to achieve cross-modal retrieval between video and text information.

[0004] However, the video depth features and text depth features extracted by video feature extractors and text feature extractors cannot fully exploit the semantic information contained in video and text information. How to further improve the retrieval effect between cross-modal information is an urgent problem to be solved. Summary of the Invention

[0005] This application provides an information retrieval method, a model training method, an apparatus, a device, and a storage medium, the technical solution of which is as follows:

[0006] According to one aspect of this application, an information retrieval method is provided, the method comprising:

[0007] Acquire input information, wherein the information mode of the input information is a first mode;

[0008] The input information is processed by calling an event representation prediction model to obtain an input event representation of the input information, which is used to indicate the event information in the input information.

[0009] Calculate the event similarity between the input event representation and the retrieved event representation, wherein the retrieved event representation is the event representation corresponding to the retrieved information, and the information modality of the retrieved information is the second modality;

[0010] If the similarity of the events exceeds a similarity threshold, the retrieval information corresponding to the retrieval event will be determined as the retrieval result of the input information.

[0011] According to one aspect of this application, a method for training an event representation prediction model is provided, the method being used to train the event representation prediction model in the above-mentioned information retrieval method, the method comprising:

[0012] Obtain a sample information group, which includes a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information correspond one-to-one with the n second modal information, where n is an integer greater than 1.

[0013] The first information group is subjected to prediction processing to obtain a first event group of the first information group, and the second information group is subjected to prediction processing to obtain a second event group of the first information group. The first event group is used to indicate the event information in the first information group, and the second event group is used to indicate the event information in the second information group.

[0014] Based on the prediction error between the first event group and the second event group, the event representation prediction model is trained to obtain the trained event representation prediction model.

[0015] According to another aspect of this application, an information retrieval device is provided, the device comprising:

[0016] The acquisition module is used to acquire input information, wherein the information mode of the input information is a first mode;

[0017] The prediction module is used to call the event representation prediction model to perform prediction processing on the input information to obtain the input event representation of the input information, and the input event representation is used to indicate the event information in the input information;

[0018] The calculation module is used to calculate the event similarity between the input event representation and the retrieved event representation, wherein the retrieved event representation is the event representation corresponding to the retrieved information, and the information mode of the retrieved information is the second mode;

[0019] The determination module is used to determine the retrieval information corresponding to the retrieval event representation as the retrieval result of the input information when the event similarity exceeds a similarity threshold.

[0020] In an optional design of the application, the event representation prediction model includes a first prediction network corresponding to the first modality;

[0021] The prediction module is also used for:

[0022] When the information mode of the input information is the first mode, the first prediction network is invoked to perform prediction processing on the input information to obtain the input event representation of the input information.

[0023] In an alternative design of the application, the first prediction network includes a first modality encoder and a first event generator, and the input information includes at least two input sub-information;

[0024] The prediction module is also used for:

[0025] When the information mode of the input information is the first mode, the first mode encoder is invoked to encode at least two of the input sub-information information one by one to obtain the input feature sequence of the input information, wherein the input feature sequence includes at least two input feature representations corresponding to at least two of the input sub-information information;

[0026] The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the input event representation of the input information, wherein the input event representation includes the representation information of at least one first modality event of the input information.

[0027] In an optional design of the application, the prediction module is further used for:

[0028] The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the input weight information of the input information. The input weight information is used to describe the weights of at least two input sub-information in the input information in at least one first modal event.

[0029] The input event representation of the input information is determined based on the input weight information and the input feature sequence.

[0030] In an optional design of the application, the event representation prediction model further includes a second prediction network corresponding to the second modality;

[0031] The prediction module is also used for:

[0032] When the information modality of the retrieved information is the second modality, the second prediction network is invoked to perform prediction processing on the retrieved information to obtain the retrieval event representation of the retrieved information.

[0033] In an alternative design of the application, the second prediction network includes a second modality encoder and a second event generator, and the retrieval information includes at least two retrieval sub-information;

[0034] The prediction module is also used for:

[0035] When the information mode of the input information is the second mode, the second mode encoder is invoked to encode at least two of the retrieval sub-information one by one to obtain the retrieval feature sequence of the retrieval information, wherein the retrieval feature sequence includes at least two retrieval feature representations corresponding to at least two of the retrieval sub-information;

[0036] The second event generator is invoked to perform prediction processing on the retrieval feature sequence to obtain the retrieval event representation of the retrieval information. The retrieval event representation includes the representation information of at least one second modality event of the retrieval information.

[0037] In an optional design of the application, the prediction module is further used for:

[0038] The second event generator is invoked to perform prediction processing on the retrieval feature sequence to obtain retrieval weight information of the retrieval information. The retrieval weight information is used to describe the weight of at least two retrieval sub-information in the retrieval information in at least one second modality event.

[0039] Based on the retrieval weight information and the retrieval feature sequence, the retrieval event representation of the retrieval information is determined.

[0040] In an alternative design of the application, the input event representation includes representation information of at least one first modal event of the input information; the retrieval event representation includes representation information of at least one second modal event of the retrieval information;

[0041] The computing module is also used for:

[0042] Calculate the correlation score between the representation information of the first modal event and the representation information of the second modal event;

[0043] The event similarity is constructed based on the relevance score.

[0044] In one alternative design of the application, the first modality is a text modality and the second modality is a video modality;

[0045] The acquisition module is also used for:

[0046] Retrieve text information;

[0047] The prediction module is also used for:

[0048] The event representation prediction model is invoked to perform prediction processing on the text information to obtain the text event representation of the text information.

[0049] The computing module is also used for:

[0050] Calculate the event similarity between the text event representation and the video event representation, wherein the video event representation is the event representation corresponding to the video information;

[0051] The determining module is also used for:

[0052] If the similarity of the events exceeds a similarity threshold, the video information corresponding to the video event representation will be determined as the retrieval result of the text information.

[0053] According to another aspect of this application, a training apparatus for an event representation prediction model is provided, the apparatus comprising:

[0054] The acquisition module is used to acquire sample information groups, which include a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information correspond one-to-one with the n second modal information, where n is an integer greater than 1.

[0055] The prediction module is used to perform prediction processing on the first information group to obtain a first event group of the first information group, and to perform prediction processing on the second information group to obtain a second event group of the first information group. The first event group is used to indicate the event information in the first information group, and the second event group is used to indicate the event information in the second information group.

[0056] The training module is used to train the event representation prediction model based on the prediction error between the first event group and the second event group, so as to obtain the trained event representation prediction model.

[0057] In an alternative design of the application, the first event group includes n first event representations corresponding to n first modal information, and the second event group includes n second event representations corresponding to n second modal information;

[0058] The training module is also used for:

[0059] Calculate a first prediction error between the first event representation and the second event group; and calculate a second prediction error between the second event representation and the first event group;

[0060] The event representation prediction model is trained based on the first prediction error and the second prediction error to obtain the trained event representation prediction model.

[0061] In an optional design of the application, the acquisition module is further configured to acquire a verification sample pair, the verification sample pair including first verification information and second verification information, wherein the first verification information and the second verification information have different information modalities;

[0062] The prediction module is further configured to call the trained event representation prediction model to perform prediction processing on the first verification information to obtain a first verification representation, and to call the trained event representation prediction model to perform prediction processing on the second verification information to obtain a second verification representation. The first verification representation includes representation information of at least two first verification events, and the second verification representation includes representation information of at least two second verification events.

[0063] The update module is further configured to verify the trained event representation prediction model based on the prediction error between the first verification representation and the second verification representation, and update the first number of the first verification events in the predicted first verification representation and the second number of the second verification events in the second verification representation.

[0064] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the information retrieval method as described above, and / or the training method for an event representation prediction model.

[0065] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the information retrieval method as described above, and / or the training method for an event representation prediction model.

[0066] According to another aspect of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor reads from the computer-readable storage medium and executes the computer instructions to implement the information retrieval method and / or the training method for the event representation prediction model described above.

[0067] The beneficial effects of the technical solution provided in this application include at least the following:

[0068] By acquiring input event representations of input information and retrieval event representations of retrieval information, the semantic information contained in the input and retrieval information is fully extracted, expanding the ways to describe semantic information. By comparing the differences between input event representations and retrieval event representations, the semantic information in information of different information modalities is compared, avoiding the differences between input and retrieval information caused by different information modalities, and improving the retrieval effect between cross-modal information. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application;

[0071] Figure 2 This is a schematic diagram illustrating a method of using an event representation prediction model provided in an exemplary embodiment of this application;

[0072] Figure 3 This is a schematic diagram of a training method for an event representation prediction model provided in an exemplary embodiment of this application;

[0073] Figure 4 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0074] Figure 5 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0075] Figure 6 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0076] Figure 7 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0077] Figure 8 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0078] Figure 9 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0079] Figure 10 This is a flowchart of an information retrieval method provided in an exemplary embodiment of this application;

[0080] Figure 11 This is a flowchart of a training method for an event representation prediction model provided in an exemplary embodiment of this application;

[0081] Figure 12 This is a flowchart of a training method for an event representation prediction model provided in an exemplary embodiment of this application;

[0082] Figure 13 This is a flowchart of a training method for an event representation prediction model provided in an exemplary embodiment of this application;

[0083] Figure 14 This is a structural block diagram of an information retrieval device provided in an exemplary embodiment of this application;

[0084] Figure 15 This is a structural block diagram of a training apparatus for an event representation prediction model provided in an exemplary embodiment of this application;

[0085] Figure 16 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0086] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0089] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the input information and search information involved in this application were obtained with full authorization.

[0091] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0092] Figure 1 A schematic diagram of a computer system provided in one embodiment of this application is shown. This computer system can implement a system architecture for training a method for event representation prediction models and / or information retrieval methods. The computer system may include a terminal 100 and a server 200. The terminal 100 may be an electronic device such as a mobile phone, tablet computer, vehicle-mounted terminal (vehicle system), wearable device, PC (Personal Computer), or unmanned reservation terminal. A client application for a target application may be installed and running on the terminal 100. This target application may be a training and / or information retrieval application for an event representation prediction model, or other applications that provide training and / or information retrieval functions for an event representation prediction model; this application does not limit the specific application. Furthermore, this application does not limit the form of the target application, including but not limited to Apps (Applications), mini-programs, etc., installed on the terminal 100, and may also be in web page form. The server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 200 may be a backend server for the aforementioned target application, used to provide backend services to the client of the target application.

[0093] The training method and / or information retrieval method for the event representation prediction model provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1Taking the implementation environment of the scheme shown as an example, the training method and / or information retrieval method of the event representation prediction model can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executing the training method and / or information retrieval method of the event representation prediction model), or the training method and / or information retrieval method of the event representation prediction model can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it. This application does not limit this.

[0094] Furthermore, the technical solution of this application can be combined with blockchain technology. For example, the training method and / or information retrieval method of the event representation prediction model disclosed in this application, wherein some data (first image, first pixel block, second pixel block, etc.) can be stored on the blockchain. The terminal 100 and the server 200 can communicate through a network, such as a wired or wireless network.

[0095] Next, the event representation prediction model in this application will be introduced:

[0096] Figure 2 A schematic diagram illustrating a method of using an event representation prediction model provided in one embodiment of this application is shown.

[0097] The event representation prediction model 300 is a trained network model; the event representation prediction model 300 includes: video prediction network 330 and text prediction network 340;

[0098] The video prediction network 330 includes: a video encoder 332 and a video event generator 334.

[0099] The computer device acquires video information 310. It then calls a video encoder 332 to encode the video information 310, obtaining a video feature sequence 310a. Specifically, the video information 310 includes at least two video frames, and correspondingly, the video feature sequence 310a includes at least two video feature vectors, with each video frame corresponding to one video feature vector. Further, by encoding each of the at least two video frames in the video information 310 one by one, video feature vectors corresponding to the at least two video frames are obtained, and these two video feature vectors constitute the video feature sequence 310a.

[0100] The computer device invokes the video event generator 334 to perform prediction processing on the video feature sequence 310a one by one, obtaining the video event representation 310b. Specifically, the video event representation 310b includes representation information of at least two video events, giving full attention to the fact that the video information 310 is composed of video events.

[0101] For example, video event representation 310b is used to describe event information in video information 310, and describes the semantic information of video information 310 through event information in video information 310; further, video event representation 310b includes representation information of at least two video events, each video event being related to at least one video frame in video information 310, and is used to describe the semantic information in the related video frame. In one example, the video event extracts at least one of the following information from at least one video frame related to the video event: time, location, participating roles, participating actions, etc.

[0102] The text prediction network 340 includes: a text encoder 342 and a text event generator 344.

[0103] The computer device acquires text information 320. It then calls a text encoder 342 to encode the text information 320, obtaining a text feature sequence 320a. Specifically, the text information 320 includes at least two word groups, and correspondingly, the text feature sequence 320a includes at least two text feature representations, with each word group corresponding to one text feature representation. Further, by encoding each of the at least two word groups in the text information 320 one by one, text feature representations corresponding to the at least two word groups are obtained, and these two text feature representations constitute the text feature sequence 320a.

[0104] The computer device invokes the text event generator 344 to perform prediction processing on the text feature sequence 320a one by one, obtaining the text event representation 320b. Specifically, the text event representation 320b includes representation information of at least two text events, giving full attention to the fact that the text information 320 is composed of text events.

[0105] For example, text event representation 320b is used to describe event information in text information 320, and describes the semantic information of text information 320 through event information in text information 320; further, text event representation 320b includes representation information of at least two text events, each text event being related to at least one phrase in text information 320, and is used to describe the semantic information in the related phrase. In one example, the text event extracts at least one of the following information from at least one phrase related to the text event: time, location, participating role, participating action, etc.

[0106] The computer device calculates a similarity score of 352 based on the difference between video event representation 310b and text event representation 320b, describing the degree of similarity between video information 310 and text information 320.

[0107] Figure 3 A schematic diagram of a training method for an event representation prediction model provided in one embodiment of this application is shown.

[0108] Obtain a sample information group, which includes a video information group 410 and a text information group 420; wherein, the video information group 410 includes 128 video information items, and the 128 video information items correspond to the 128 text information items in the text information group 420.

[0109] First, the video prediction network 430 is introduced; the video prediction network 430 includes a video encoder 432 and a video event generator 434.

[0110] The video encoder 432 is invoked to encode each of the 128 video information pieces in the video information group 410, resulting in a video feature group 410a. The video feature group 410a includes 128 video feature sequences corresponding to the 128 video information pieces in the video information group 410.

[0111] The video event generator 434 is invoked to perform prediction processing on each of the 128 video feature sequences to obtain video event group 410b; video event group 410b includes 128 video event representations corresponding to the 128 video information in video information group 410.

[0112] Next, we introduce the text prediction network 440; the text prediction network 440 includes a text encoder 442 and a text event generator 444.

[0113] The text encoder 442 is invoked to encode each of the 128 text messages in the text information group 420, resulting in the text feature group 420a. The text feature group 420a includes 128 text feature sequences corresponding to the 128 text messages in the text information group 420.

[0114] The text event generator 444 is invoked to predict each of the 128 text feature sequences to obtain the text event group 420b; the text event group 420b includes the 128 text event representations corresponding to the 128 text information in the text information group 420.

[0115] Calculate the first prediction error between the video event representation and the text event group 420b; calculate the second prediction error between the text event representation and the video event group 410b; construct the prediction error 452 based on the first and second prediction errors, train the event representation prediction model, and obtain the trained event representation prediction model.

[0116] Figure 4 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. The method can be executed by a computer device. The method includes:

[0117] Step 510: Obtain input information;

[0118] For example, the information modality of the input information is the first modality; the information modality is used to indicate the information form of the input information, and / or the information source of the input information.

[0119] In one example, the information format of the input information includes, but is not limited to, at least one of the following: text, audio, video, animation, and image. The information source of the input information indicates the method of acquiring the input information. Taking image information as an example, the information source of the input information indicates the method of acquiring the image information. For example, the information source of the input information includes, but is not limited to, at least one of the following: charge-coupled device (CCD), charge injection device (CID), complementary metal-oxide semiconductor (CMOS), and computed tomography (CT).

[0120] Step 520: Call the event representation prediction model to perform prediction processing on the input information to obtain the input event representation of the input information;

[0121] For example, an input event representation is used to indicate event information in the input information; an input event representation is used to represent semantic information implied in the input information. In one example, the input event representation of the input information indicates the semantic information of the input information by extracting at least one of the information such as time, location, participating role, and participating action from the input information to indicate the aforementioned semantic information. In one example, the input information is text information, which includes at least two phrases, and the input event representation of the input information indicates the semantic information implied in at least one phrase.

[0122] Step 530: Calculate the event similarity between the input event representation and the retrieved event representation;

[0123] A retrieval event representation is an event representation corresponding to the retrieved information, and the information mode of the retrieved information is the second mode; similarly, the retrieval event representation is used to represent the semantic information contained in the retrieved information. In one example, the retrieved information is video information, which includes at least two video frames, and the retrieval event representation of the retrieved information indicates the semantic information contained in at least one video frame.

[0124] It should be noted that the information modality of retrieved information is different from that of input information. In one example, the information format of retrieved information and input information are different, and / or the information sources are different.

[0125] Event similarity is used to indicate the difference between input event representations and retrieved event representations. Event similarity is obtained by comparing input event representations and retrieved event representations.

[0126] In one alternative implementation, the retrieved information can be information from a retrieval database, which includes information from at least two second modalities. The input information is used to retrieve information from the retrieval database whose information modality is a second modality.

[0127] Step 540: If the event similarity exceeds the similarity threshold, determine the retrieval information corresponding to the retrieval event representation as the retrieval result of the input information;

[0128] For example, the event similarity threshold can be preset or obtained by training the event representation prediction model. This embodiment does not make any restrictive provisions.

[0129] For example, if the event similarity exceeds a similarity threshold, indicating a correlation between the retrieved information and the input information, the retrieved information will be used as the retrieval result for the input information. The retrieved information can be one or more pieces of information; when there are multiple pieces of information, no restrictions are placed on their order.

[0130] In summary, the method provided in this embodiment fully extracts the semantic information contained in the input and retrieval information by acquiring the input event representation of the input information and the retrieval event representation of the retrieval information, thus expanding the ways to describe semantic information. By comparing the differences between the input event representation and the retrieval event representation, the semantic information in information of different information modalities is compared, avoiding the differences between input and retrieval information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0131] Next, we will further introduce the event representation prediction model:

[0132] In one alternative implementation of this application, the event representation prediction model includes a first prediction network.

[0133] Further, optionally, the event representation prediction model may also include a second prediction network.

[0134] First, let's introduce the first prediction network in the event representation prediction model:

[0135] Figure 5 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 4 In the illustrated embodiment, step 520 can be implemented as step 522:

[0136] Step 522: When the information mode of the input information is the first mode, call the first prediction network to perform prediction processing on the input information to obtain the input event representation of the input information;

[0137] For example, the event representation prediction model includes a first prediction network corresponding to the first modality.

[0138] For example, the first prediction network is used to perform prediction processing on input information with the first information modality to obtain the input event representation of the input information. Taking text information as an example, the first prediction network can be implemented as a text prediction network to extract the semantic information contained in the text information and predict the text event representation of the text information.

[0139] It should be noted that this embodiment only introduces the first prediction network and uses text information as an example for illustration. It is understood that the input information can be implemented as information in other modalities. In an optional implementation, the event representation prediction model may include only the first prediction network, or it may include other neural networks besides the first prediction network.

[0140] In summary, the method provided in this embodiment uses a first prediction network to predict and process the input information to obtain an input event representation, which fully extracts the semantic information contained in the input information and expands the ways of describing semantic information. By comparing the differences between the input event representation and the retrieved event representation, the semantic information in information of different information modalities is compared, avoiding the differences between input information and retrieved information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0141] Figure 6 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 6 In the illustrated embodiment, step 522 can be implemented as steps 522a and 522b:

[0142] Step 522a: When the information mode of the input information is the first mode, call the first mode encoder to encode at least two input sub-information one by one to obtain the input feature sequence of the input information;

[0143] This embodiment further describes the first prediction network, which includes a first modality encoder and a first event generator. The first modality encoder is used to encode input information whose information modality is the first modality, obtaining an input feature sequence of the input information.

[0144] In one implementation, the input information includes at least two input sub-information pieces; the first modal encoder encodes each input sub-information piece sequentially to obtain the input feature representation corresponding to the input sub-information. The input feature sequence includes at least two input feature representations corresponding to the at least two input sub-information pieces. For example, the encoding process of the first modal encoder on the input sub-information pieces is independent of each other; the first modal encoder encodes the first input feature representation corresponding to the first input sub-information piece based solely on the first input sub-information piece.

[0145] For example, the input information includes at least two input sub-information; the input information is composed of at least two input sub-information. For example: the input information is text information, which is composed of at least two phrases or at least two characters; the input information is video information, which is composed of at least two video frames or at least two video segments; the input information is audio information, which is composed of at least two audio frames or at least two audio segments; the input information is image information, which is composed of at least two pixel blocks or at least two image blocks.

[0146] In one example, we will illustrate this by using text information as the input; text information t i By m t Composed of phrases, the text information is represented as:

[0147]

[0148] in, Represents the m-th element in the text information. t A phrase.

[0149] Furthermore, a special classification embedding character [CLS] is added at the beginning of the text information, and a sequence end identifier [EOS] is added at the end of the text information. The special classification embedding character [CLS] and m in the text information are then processed by the first modal encoder. t Each word group and the sequence end identifier [EOS] are encoded to obtain the corresponding input feature representation, which together form the input feature sequence of the input information.

[0150] In one example, the input feature sequence is represented as:

[0151]

[0152] in, Indicates the input feature sequence. The input feature representation represents the special classification embedding [CLS]. Indicates the m-th t Input feature representation of each word group The input feature representation represents the sequence end identifier [EOS]. In an optional implementation, the input feature representation of phrases in the text information is a 512-dimensional vector.

[0153] In one example, when the input information is text information, the first modality encoder is at least one of a convolutional neural network (CNN), a translation encoder (Transformer), or a deep averaging network (DAN).

[0154] Step 522b: Call the first event generator to perform prediction processing on the input feature sequence to obtain the input event representation of the input information;

[0155] For example, the input event representation is used to indicate event information in the input information; in one implementation, the input event representation includes representation information of at least one first modal event of the input information; in another implementation, the input event representation includes representation information of at least two first modal events of the input information; for example, the first modal event and at least one input sub-information of the input information are related, the first modal event is used to represent the semantic information implied by at least one input sub-information, and the first modal event extracts at least one of the information such as time, location, participating role, and participating action from at least one input sub-information to indicate the aforementioned semantic information. In one example, the first event generator includes a fully connected layer.

[0156] In one alternative implementation, when there are multiple first modal events, the representation information of the i-th first modal event is generated based on the representation information of the (i-1)-th first modal event. It is understood that referencing the representation information of the (i-1)-th first modal event is used to avoid the representation information of the i-th and (i-1)-th first modal events being identical, thus preventing the generation of duplicate representation information. In one implementation, this is used to avoid coupling between the representation information of the i-th and (i-1)-th first modal events, preventing mutual interference.

[0157] In summary, the method provided in this embodiment, in which the first prediction network includes a first modality encoder and a first event generator, encodes each input sub-information, fully extracts the depth information in each input sub-information, and compares the semantic information in information of different information modalities by comparing the differences between input and retrieved event representations, thereby avoiding the differences between input and retrieved information caused by different information modalities and improving the retrieval effect between cross-modal information.

[0158] Next, we will introduce the first event generator in more detail:

[0159] In one implementation of this application, Figure 6 Step 522b in the illustrated embodiment can be implemented as sub-step 1 and sub-step 2:

[0160] Sub-step 1: Call the first event generator to perform prediction processing on the input feature sequence to obtain the input weight information of the input information;

[0161] For example, input weight information is used to describe the weights of at least two input sub-information in at least one first modal event.

[0162] In one example, the input information includes first sub-information and second sub-information, and the input event representation includes the representation information of the two first modal events of the input information. Accordingly, the input weight information includes the representation information of the first sub-information in the two first modal events and the representation information of the second sub-information in the two first modal events.

[0163] In one example, we will use text information as the input information for illustration;

[0164]

[0165]

[0166]

[0167] Among them, W q , W p W pq and W pt This represents the learnable parameter matrix in the first event generator; for example, the learnable parameter matrix is ​​determined through the training process of the event representation prediction model.

[0168] Where ReLU represents the activation function, tanh represents the hyperbolic tangent function, and softmax represents the normalization method. This represents the representational information of the (n-1)th first modal event. In the case of n=1, It is the representation information of the 0th first modal event; in an optional implementation, It is represented as a 512-dimensional zero vector. The input feature representation of the sequence end identifier [EOS] in the text information is used as a global semantic representation of the text information. This represents the input feature representation of the j-th word group. This represents intermediate variables obtained during the calculation process.

[0169] in, The weight representing the text information to the representation information of the nth first modal event is, in one implementation, It is an m t A weight vector of +2 dimensions. The text information includes m... t a phrase, This represents the weight of the special classification embedding [CLS] on the representation information of the nth first modal event. This represents the weight of the sequence end identifier [EOS] in the representation information of the nth first modal event.

[0170] In one alternative implementation, the dimension of the learnable parameter matrix is ​​as follows; R represents a real number.

[0171] W q ∈R 512×1024 ;

[0172]

[0173] W p ∈R 1×512 ;

[0174] W pq ∈R 512×512 ;

[0175] W pt ∈R 512×512 ;

[0176] Sub-step 2: Determine the input event representation of the input information based on the input weight information and the input feature sequence.

[0177] In one example, we will use text information as the input information for illustration;

[0178]

[0179] in, Representational information of the nth first modal event of the text information; This represents the weight of the j-th word group in relation to the representation information of the n-th first modal event. This represents the input feature representation of the j-th word group.

[0180] For example, the representation information of k first modal events of the text information is calculated sequentially:

[0181]

[0182] in, The input event representation of the text information includes the representation information of k first modal events. This represents the representation information of the k-th first modal event.

[0183] In summary, the method provided in this embodiment obtains the input weight information of the input information through the first event generator, fully extracts the semantic information contained in the input information, and expands the way of describing semantic information. By comparing the differences between the input event representation and the retrieval event representation, the semantic information in the information of different information modalities is compared, avoiding the differences between the input information and the retrieval information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0184] The second prediction network in the event representation prediction model is introduced below:

[0185] Figure 7 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 5 Based on the illustrated embodiment, step 525 is also included:

[0186] Step 525: When the information mode of the retrieved information is the second mode, the second prediction network is invoked to perform prediction processing on the retrieved information to obtain the retrieval event representation of the retrieved information;

[0187] For example, the event representation prediction model also includes a second prediction network corresponding to the second modality.

[0188] For example, the second prediction network is used to perform prediction processing on the retrieved information whose information mode is the second mode, to obtain the retrieval event representation of the retrieved information. Taking the retrieved information as video information as an example, the first prediction network can be implemented as a video prediction network, used to extract the semantic information contained in the video information and predict the video event representation of the video information.

[0189] It should be noted that this embodiment only introduces the second prediction network and uses video information as an example for illustration. It can be understood that the retrieved information can be implemented as information in other information modalities.

[0190] In summary, the method provided in this embodiment uses a second prediction network to predict and process the retrieved information to obtain a retrieval event representation, which fully extracts the semantic information contained in the retrieval information and expands the ways of describing semantic information. By comparing the differences between the input event representation and the retrieval event representation, the semantic information in information of different information modalities is compared, avoiding the differences between input information and retrieval information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0191] Figure 8 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 7 In the illustrated embodiment, step 525 can be implemented as steps 525a and 525b:

[0192] Step 525a: When the information mode of the retrieved information is the second mode, call the second mode encoder to encode at least two retrieval sub-information one by one to obtain the retrieval feature sequence of the retrieval information;

[0193] This embodiment further describes the second prediction network, which includes a second modality encoder and a second event generator. The second modality encoder is used to encode retrieval information whose information modality is the second modality, obtaining a retrieval feature sequence of the retrieval information.

[0194] In one implementation, the retrieval information includes at least two retrieval sub-information pieces; the second modal encoder encodes each retrieval sub-information piece sequentially to obtain the retrieval feature representation corresponding to each retrieval sub-information piece. The retrieval feature sequence includes at least two retrieval feature representations corresponding to at least two retrieval sub-information pieces. For example, the encoding process of the second modal encoder for the retrieval sub-information pieces is independent of each other; the second modal encoder encodes only the first retrieval sub-information piece to obtain the first retrieval feature representation corresponding to the first retrieval sub-information piece.

[0195] For example, the retrieval information includes at least two retrieval sub-information; the retrieval information is composed of at least two retrieval sub-information.

[0196] In one example, we will illustrate this by retrieving video information; video information v i By m v It consists of several video frames, and the video information is represented as:

[0197]

[0198] in, Indicates the m-th element in the video information v One video frame.

[0199] Furthermore, the second modal encoder is used to process the m in the video information. v Each video frame is encoded to obtain a corresponding retrieval feature representation, which together form the retrieval feature sequence of the retrieval information.

[0200] In one example, the retrieved feature sequence is represented as:

[0201]

[0202] in, Indicates the retrieval of feature sequences. Indicates the m-th v The retrieval feature representation of each video frame. In one optional implementation, the retrieval feature representation of each video frame in the video information is a 512-dimensional vector.

[0203] In one example, when the retrieved information is video information, the second modality encoder is at least one of a convolutional neural network (CNN), a vision transformer (ViT), a long short-term memory (LSTM), or a multilayer perceptron (MLP).

[0204] Step 525b: Call the second event generator to perform prediction processing on the retrieval feature sequence to obtain the retrieval event representation of the retrieval information;

[0205] For example, a retrieval event representation is used to indicate event information in the retrieval information; in one implementation, the retrieval event representation includes representation information of at least one second modal event of the retrieval information; in another implementation, the retrieval event representation includes representation information of at least two second modal events of the retrieval information; for example, the second modal event and at least one retrieval sub-information of the retrieval information are related, and the second modal event is used to represent the semantic information implied by at least one retrieval sub-information. The second modal event extracts at least one of the information such as time, location, participating role, and participating action from at least one retrieval sub-information to indicate the aforementioned semantic information. In one example, the second event generator includes a fully connected layer.

[0206] In one alternative implementation, when there are multiple second modal events, the representation information of the i-th second modal event is generated based on the representation information of the (i-1)-th second modal event. It is understood that referencing the representation information of the (i-1)-th second modal event is used to avoid the representation information of the i-th and (i-1)-th second modal events being identical, thus preventing the generation of duplicate representation information. In one implementation, this is used to avoid coupling between the representation information of the i-th and (i-1)-th second modal events, preventing mutual interference.

[0207] In summary, the method provided in this embodiment, in which the second prediction network includes a second modality encoder and a second event generator, encodes each retrieval sub-information, fully extracts the deep information in each retrieval sub-information, and compares the semantic information in information of different information modalities by comparing the differences between input event representations and retrieval event representations, thereby avoiding the differences between input information and retrieval information caused by different information modalities and improving the retrieval effect between cross-modal information.

[0208] Next, we will introduce the second event generator in more detail:

[0209] In one implementation of this application, Figure 8 Step 525b in the illustrated embodiment can be implemented as sub-steps 3 and 4:

[0210] Sub-step 3: Call the second event generator to perform prediction processing on the retrieval feature sequence to obtain the retrieval weight information of the retrieval information;

[0211] For example, retrieval weight information is used to describe the weights of at least two retrieval sub-informations in at least one second modal event.

[0212] In one example, the retrieval information includes first sub-information and second sub-information, and the retrieval event representation includes the representation information of two second modal events of the retrieval information. Accordingly, the retrieval weight information includes the representation information of the first sub-information in the two second modal events and the representation information of the second sub-information in the two second modal events.

[0213] In one example, we will illustrate this by taking the retrieval of video information as an example;

[0214]

[0215]

[0216]

[0217] Among them, W q , W p W pq and W ptThis represents the learnable parameter matrix in the second event generator; exemplarily, the learnable parameter matrix is ​​determined through the training process of the event representation prediction model. It should be noted that the learnable parameter matrix in this embodiment is usually different from the learnable parameter matrix of the first event generator described above, but the possibility of the learnable parameter matrix being the same does not exist. Further, during the training process of the event representation prediction model, the first prediction network is trained using sample information in the form of a first modality to obtain the learnable parameter matrix of the first prediction network. During the training process of the event representation prediction model, the second prediction network is trained using sample information in the form of a second modality to obtain the learnable parameter matrix of the second prediction network.

[0218] Where ReLU represents the activation function, tanh represents the hyperbolic tangent function, and softmax represents the normalization method. This represents the representational information of the (n-1)th second modal event, in the case of n=1. It is the representation information of the 0th second modal event; in an optional implementation, It is represented as a 512-dimensional zero vector. This represents the input feature representation of the j-th word group. This represents the global feature representation of the video information obtained through the average pooling operation. This represents intermediate variables obtained during the calculation process.

[0219] in, The weight of the video information to the representation information of the nth second modal event is given in one implementation. It is an m v A weight vector of dimension m. The video information includes m... v One video frame.

[0220] In one alternative implementation, the dimension of the learnable parameter matrix is ​​as follows; R represents a real number.

[0221] W q ∈R 512×1024 ;

[0222]

[0223] W p ∈R 1×512 ;

[0224] W pq ∈R 512×512 ;

[0225] W pt ∈R 512×512 ;

[0226] Sub-step 4: Determine the retrieval event representation of the retrieval information based on the retrieval weight information and retrieval feature sequence;

[0227] In one example, we will illustrate this by taking the retrieval of video information as an example;

[0228]

[0229] in, Representational information of the nth second modal event in the video information; This represents the weight of the representation information of the j-th video frame to the n-th second modal event. This represents the input feature representation of the j-th video frame.

[0230] For example, the representation information of k second-modal events of the video information is calculated sequentially:

[0231]

[0232] in, The retrieval event representation of the text information representing video information includes the representation information of k second-modal events. This represents the characterization information of the k-th second modal event. It should be noted that the number of second modal events in this embodiment may be the same as or different from the number of first modal events mentioned above; this embodiment does not impose any limiting provisions in this regard.

[0233] In summary, the method provided in this embodiment obtains the retrieval weight information of the retrieval information through the second event generator, fully extracts the semantic information contained in the retrieval information, and expands the ways of describing semantic information. By comparing the differences between the input event representation and the retrieval event representation, the semantic information in the information of different information modalities is compared, avoiding the differences between the input information and the retrieval information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0234] Figure 9 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 4 In the illustrated embodiment, step 530 can be implemented as steps 532 and 534:

[0235] Step 532: Calculate the correlation score between the representation information of the first modal event and the representation information of the second modal event;

[0236] For example, a relevance score is calculated using a similarity algorithm. Specifically, the relevance score is expressed as:

[0237]

[0238] in, Indicates input information t j The l-th first modal event and the retrieved information v i The correlation score between z second-modal events; This represents the representation information of the l-th first modal event. This represents the representation information of the z-th second modal event. || ||2 represents the L2-norm operation.

[0239] For example, Represents the correlation matrix ES ij The element in the z-th row and l-th column; the correlation matrix ES ij The similarity between k first-modal events and k second-modal events is expressed as follows:

[0240] ES ij ∈[-1,1] k×k ;

[0241] Correlation Matrix ES ij It is a k-dimensional multiplied by k-dimensional matrix.

[0242] In one example, the information retrieved is video information, and the input information is text information.

[0243] Step 534: Construct the event similarity based on the relevance score;

[0244] For example, the event similarity between the input event representation and the retrieved event representation includes at least one of a first similarity from the input information to the retrieved information and a second similarity from the retrieved information to the input information.

[0245] Let's take the example of event similarity, which includes first similarity and second similarity, to illustrate this.

[0246] First, calculate the maximum relevance score between the representation information of the l-th first modal event of the input information and the k second modal events of the retrieved information:

[0247]

[0248] in, Represents the correlation matrix ES ij The maximum value in the l-th column.

[0249] For example, the first similarity is:

[0250]

[0251] in, This represents the first similarity score, where k represents the number of first-modal events in the input information. Represents the correlation matrix ES ij Find the maximum value in the l-th column of the correlation matrix ES. ij The average of the maximum values ​​in each column is used as the first similarity.

[0252] Similarly, the calculation process for the second similarity is as follows:

[0253] First, calculate the maximum relevance score between the representation information of the z-th second modal event of the retrieved information and the k first modal events of the input information:

[0254]

[0255] in, Represents the correlation matrix ES ij The maximum value in the z-th row.

[0256] For example, the second similarity is:

[0257]

[0258] in, This represents the second similarity, where k represents the number of second modal events in the retrieved information. Represents the correlation matrix ES ij Find the maximum value in the z-th row of the correlation matrix ES. ij The average of the maximum values ​​in each row is used as the second similarity.

[0259] When event similarity includes first similarity and second similarity, the event similarity is:

[0260]

[0261] Wherein, s(v i ,t j () indicates the similarity between events. Indicates the first similarity. The second similarity is represented by the average of the first and second similarities.

[0262] In summary, the method provided in this embodiment describes the differences in semantic information contained in different information modalities by calculating the correlation score between the representation information of the first modal event and the representation information of the second modal event. By comparing the semantic information in the information of different information modalities, the method avoids the differences between input information and retrieval information caused by different information modalities and improves the retrieval effect between cross-modal information.

[0263] Figure 10 A flowchart illustrating an exemplary embodiment of the information retrieval method provided in this application is shown. This method can be executed by a computer device. That is, in Figure 4 In the illustrated embodiment, step 510 can be implemented as step 510a; step 520 can be implemented as step 520a; step 530 can be implemented as step 530a; and step 540 can be implemented as step 540a.

[0264] Step 510a: Obtain text information;

[0265] For example, in this embodiment, the information modality of the input information is the text modality, and the input information is text information.

[0266] Step 520a: Call the event representation prediction model to perform prediction processing on the text information to obtain the text event representation of the text information;

[0267] For example, an event representation prediction model is used to predict text information to obtain text event representations, which are used to represent the semantic information contained in the text information. In an optional implementation, the text event representation includes representation information of at least two text modal events, which are used to represent the semantic information contained in the text information.

[0268] Step 530a: Calculate the event similarity between the text event representation and the video event representation;

[0269] For example, a video event representation is an event representation corresponding to video information; the video event representation uses the semantic information contained in the video information. Event similarity is used to indicate the difference between text event representations and video event representations; event similarity is obtained by comparing text event representations and video event representations.

[0270] Step 540a: If the event similarity exceeds the similarity threshold, determine the video information corresponding to the video event representation as the retrieval result of the text information.

[0271] For example, if the similarity between video information and text information exceeds a similarity threshold, and there is a correlation between them, the video information will be used as the retrieval result for the text information.

[0272] It should be noted that this embodiment only illustrates the case where the first modality is text and the second modality is video. Those skilled in the art will understand that in another alternative implementation, the first modality is video and the second modality is text. In yet another alternative implementation, the first and second modalities are different.

[0273] In summary, the method provided in this embodiment fully extracts the semantic information contained in text information and video information by acquiring text event representations of text information and video event representations of video information, thus expanding the ways to describe semantic information. By comparing the differences between text event representations and video event representations, the semantic information in information of different information modalities is compared, avoiding the differences between text information and video information caused by different information modalities, and improving the video effect between cross-modal information.

[0274] Figure 11 A flowchart illustrating a training method for an event representation prediction model provided in an exemplary embodiment of this application is shown. The method can be performed by a computer device. The method includes:

[0275] Step 610: Obtain the sample information group;

[0276] For example, the sample information group includes a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information corresponds one-to-one with the n second modal information, where n is an integer greater than 1.

[0277] For example, an information modality is used to indicate the information form of information and / or the information source of information. For example, a first modality and a corresponding second modality are used to imply the same semantic information and are different expressions of the same information. In an optional implementation, the first modality is obtained by manually annotating the second modality.

[0278] In one preferred implementation, n takes the value 128.

[0279] Step 620: Perform prediction processing on the first information group to obtain the first event group of the first information group, and perform prediction processing on the second information group to obtain the second event group of the first information group;

[0280] For example, a first event group is used to indicate event information in a first information group. In one implementation, by performing prediction processing on each of the n first modal information pieces in the first information group, a first event representation of the n first modal information pieces corresponding to the n first modal information pieces is obtained; the n first event representations are used to construct the first information group. Similarly, a second event group is used to indicate event information in a second information group. In one implementation, by performing prediction processing on each of the n second modal information pieces in the second information group, a second event representation of the n second modal information pieces corresponding to the n second modal information pieces is obtained; the n second event representations are used to construct the second information group.

[0281] Step 630: Based on the prediction error between the first event group and the second event group, train the event representation prediction model to obtain the trained event representation prediction model.

[0282] For example, the prediction error is used to describe the difference between the first event group and the second event group, and is obtained by comparing the difference between the first event group and the second event group. The event representation prediction model is trained using the prediction error to obtain the event representation prediction model in any of the embodiments above.

[0283] For example, the parameters of the event representation prediction model are updated using a backpropagation algorithm based on the prediction error. Multiple sample information groups are used to compare the prediction error multiple times and update the parameters of the event representation prediction model, thereby improving the prediction accuracy of the event representation prediction model.

[0284] In summary, the method provided in this embodiment trains the event representation prediction model by acquiring sample information groups, calculating the prediction error between the first event group and the second event group, constructing positive and negative samples using multiple first and second pieces of information in the sample information group, comparing the differences between the first and second event representations, and comparing the semantic information in information of different information modalities. This avoids the differences between the first and second information caused by different information modalities and improves the retrieval effect between cross-modal information.

[0285] Figure 12 A flowchart illustrating a training method for an event representation prediction model provided in an exemplary embodiment of this application is shown. This method can be executed by a computer device. That is, in Figure 11 In the illustrated embodiment, step 630 can be implemented as steps 632 and 634:

[0286] Step 632: Calculate the first prediction error between the first event representation and the second event group; and calculate the second prediction error between the second event representation and the first event group;

[0287] For example, the first information group includes n first modal information items, and the first information group is represented as follows: t j This represents the j-th first modal information in the first information group. The second information group includes n second modal information items, and is represented as... v j This represents the j-th second modal information in the second information group.

[0288] For example, a first prediction error is calculated between the first event representation and the second event group.

[0289] For example, the second event group includes n second event representations corresponding to n second modal information; the first event group includes n first event representations corresponding to n first modal information.

[0290] For example, the first prediction error is:

[0291]

[0292] Among them, L t2v s(v) represents the first prediction error. i ,t i ) represents the event similarity between the first event representation and the second event representation; there is a correspondence between the first modality information to which the first event representation belongs and the second modality information to which the second event representation belongs. n represents the number of first modality information items in the first information group, or the number of second modality information items in the second information group. s(v j ,t i Let represent the event similarity between the i-th first modal information in the first information group and the j-th second modal information in the second information group. exp represents an exponential function with the natural constant e as the base.

[0293] For example, please refer to the above text for how event similarity is calculated. Figure 9 The content of the illustrated embodiment will not be repeated in this embodiment.

[0294] For example, a second prediction error is calculated between the second event representation and the first event group.

[0295] For example, the first event group includes n first event representations corresponding to n first modal information; the second event group includes n second event representations corresponding to n second modal information.

[0296] For example, the second prediction error is:

[0297]

[0298] Among them, L v2t This represents the second prediction error, s(v i ,t i ) represents the event similarity between the first event representation and the second event representation; there is a correspondence between the first modality information to which the first event representation belongs and the second modality information to which the second event representation belongs. n represents the number of first modality information items in the first information group, or the number of second modality information items in the second information group. s(v i ,t j Let represent the event similarity between the j-th first modal information in the first information group and the i-th second modal information in the second information group. exp represents an exponential function with the natural constant e as the base.

[0299] For example, please refer to the above text for how event similarity is calculated. Figure 9 The content of the illustrated embodiment will not be repeated in this embodiment.

[0300] Step 634: Train the event representation prediction model based on the first prediction error and the second prediction error to obtain the trained event representation prediction model;

[0301] In one implementation, the average of the first and second prediction errors is used as the prediction error to train the event representation prediction model.

[0302]

[0303] Where L represents the prediction error, L t2v L represents the first prediction error. v2t This represents the second prediction error.

[0304] In summary, the method provided in this embodiment trains the event representation prediction model by calculating the first prediction error and the second prediction error. It makes full use of multiple first and second pieces of information in the sample information group to construct positive and negative samples, avoiding the differences between the first and second information caused by different information modalities, and improving the retrieval effect between cross-modal information.

[0305] Figure 13 A flowchart illustrating a training method for an event representation prediction model provided in an exemplary embodiment of this application is shown. This method can be executed by a computer device. That is, in Figure 11 Based on the illustrated embodiment, steps 642, 644, and 646 are also included:

[0306] Step 642: Obtain verification sample pairs;

[0307] The verification sample pair includes first verification information and second verification information, and the information modalities of the first verification information and the second verification information are different.

[0308] Step 644: Call the trained event representation prediction model to predict the first verification information to obtain the first verification representation, and call the trained event representation prediction model to predict the second verification information to obtain the second verification representation.

[0309] The first verification representation includes representation information of at least two first verification events, and the second verification representation includes representation information of at least two second verification events.

[0310] Step 646: Based on the prediction error between the first verification representation and the second verification representation, verify the trained event representation prediction model, and update the first number of the first verification events in the predicted first verification representation and the second number of the second verification events in the second verification representation.

[0311] For example, the validation samples are used to update the hyperparameters in the event representation prediction model; specifically, the hyperparameters include at least one of a first number of first validation events and a second number of second validation events. It should be noted that the first number and the second number can be the same or different, and this embodiment does not impose any limiting provisions in this regard.

[0312] In summary, the method provided in this embodiment verifies the trained event representation prediction model by obtaining verification sample pairs, updates the hyperparameters in the trained event representation prediction model, and updates the first and second quantities, thereby ensuring the prediction effect of the trained event representation prediction model.

[0313] In a specific example, the training method for an event representation prediction model is introduced:

[0314] Get Video Information Group Text message group The number of iterations t, the learning rate η, and the batch size n.

[0315] In this design, the batch size n is the same as the number of video messages in the video message group, and the batch size n is the same as the number of text messages in the text message group. For example, the number of iterations t and the learning rate η are typically preset.

[0316] The event representation prediction model is trained t times, for example, For i = 1: t.

[0317] For each of the n video and n text data in a batch, prediction processing is performed sequentially, for example, For j = 1: N / n. For example, N represents the number of video data points used to train the event representation prediction model.

[0318] The video encoder and the text encoder are used to encode n video information and n text information respectively, so as to obtain the corresponding video feature sequence of n video information and the text feature sequence of n text information.

[0319] The function calls the video event generator to generate multiple video event representations, and also calls the text event generator to generate multiple video event representations.

[0320] Based on the similarity relationships between multiple video event representations, the similarity relationship between video information and text information is calculated. By minimizing the prediction error obtained based on the similarity relationship, the event representation prediction model is trained to obtain the trained event representation prediction model.

[0321] For example, the performance of the trained event representation prediction model was validated using a public dataset, and the experimental results are shown in Table 1.

[0322] Table 1

[0323]

[0324] The publicly available dataset LSMDC includes 118,081 video-text pairs. R@K represents the proportion of query points in the top K returned search results that contain results related to the query sample; a higher value indicates better search performance. MdR represents the median ranking of results related to the query point in the returned results; a lower value indicates better results. MnR represents the mean ranking of results related to the query point in the returned results; a lower value indicates better search performance. Table 1 shows the prediction results of Collaborative Experts, Multimodal Transformer for Video Retrieval (MMT), Multidomain Multimodal Transformer for Video Retrieval (MDMMT), An Empirical Study of CLIP for End to End Video Clip Retrieval (CLIP4Clip), and the event representation prediction model (CLIPMERG) in this application.

[0325] Those skilled in the art will understand that the above embodiments can be implemented independently, or the above embodiments can be freely combined to create new embodiments to implement the information retrieval method and / or the training method of the event representation prediction model of this application.

[0326] Figure 14 A block diagram of an information retrieval apparatus provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0327] The acquisition module 810 is used to acquire input information, wherein the information mode of the input information is a first mode;

[0328] Prediction module 820 is used to call the event representation prediction model to perform prediction processing on the input information to obtain the input event representation of the input information, wherein the input event representation is used to indicate the event information in the input information;

[0329] The calculation module 830 is used to calculate the event similarity between the input event representation and the retrieved event representation, wherein the retrieved event representation is the event representation corresponding to the retrieved information, and the information mode of the retrieved information is the second mode;

[0330] The determination module 840 is used to determine the retrieval information corresponding to the retrieval event representation as the retrieval result of the input information when the event similarity exceeds a similarity threshold.

[0331] In an optional design of the application, the event representation prediction model includes a first prediction network corresponding to the first modality;

[0332] The prediction module 820 is also used for:

[0333] When the information mode of the input information is the first mode, the first prediction network is invoked to perform prediction processing on the input information to obtain the input event representation of the input information.

[0334] In an alternative design of the application, the first prediction network includes a first modality encoder and a first event generator, and the input information includes at least two input sub-information;

[0335] The prediction module 820 is also used for:

[0336] When the information mode of the input information is the first mode, the first mode encoder is invoked to encode at least two of the input sub-information information one by one to obtain the input feature sequence of the input information, wherein the input feature sequence includes at least two input feature representations corresponding to at least two of the input sub-information information;

[0337] The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the input event representation of the input information, wherein the input event representation includes the representation information of at least one first modality event of the input information.

[0338] In an alternative design of the application, the prediction module 820 is further configured to:

[0339] The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the input weight information of the input information. The input weight information is used to describe the weights of at least two input sub-information in the input information in at least one first modal event.

[0340] The input event representation of the input information is determined based on the input weight information and the input feature sequence.

[0341] In an optional design of the application, the event representation prediction model further includes a second prediction network corresponding to the second modality;

[0342] The prediction module 820 is also used for:

[0343] When the information modality of the retrieved information is the second modality, the second prediction network is invoked to perform prediction processing on the retrieved information to obtain the retrieval event representation of the retrieved information.

[0344] In an alternative design of the application, the second prediction network includes a second modality encoder and a second event generator, and the retrieval information includes at least two retrieval sub-information;

[0345] The prediction module 820 is also used for:

[0346] When the information mode of the input information is the second mode, the second mode encoder is invoked to encode at least two of the retrieval sub-information one by one to obtain the retrieval feature sequence of the retrieval information, wherein the retrieval feature sequence includes at least two retrieval feature representations corresponding to at least two of the retrieval sub-information;

[0347] The second event generator is invoked to perform prediction processing on the retrieval feature sequence to obtain the retrieval event representation of the retrieval information. The retrieval event representation includes the representation information of at least one second modality event of the retrieval information.

[0348] In an optional design of the application, the prediction module 820 is further configured to: call the second event generator to perform prediction processing on the retrieval feature sequence to obtain retrieval weight information of the retrieval information, wherein the retrieval weight information is used to describe the weight of at least two retrieval sub-information in the retrieval information in at least one second modal event;

[0349] Based on the retrieval weight information and the retrieval feature sequence, the retrieval event representation of the retrieval information is determined.

[0350] In an alternative design of the application, the input event representation includes representation information of at least one first modal event of the input information; the retrieval event representation includes representation information of at least one second modal event of the retrieval information;

[0351] The computing module 830 is also used for:

[0352] Calculate the correlation score between the representation information of the first modal event and the representation information of the second modal event;

[0353] The event similarity is constructed based on the relevance score.

[0354] In one alternative design of the application, the first modality is a text modality and the second modality is a video modality;

[0355] The acquisition module 810 is further configured to: acquire text information;

[0356] The prediction module 820 is also used for:

[0357] The event representation prediction model is invoked to perform prediction processing on the text information to obtain the text event representation of the text information.

[0358] The calculation module 830 is further configured to: calculate the event similarity between the text event representation and the video event representation, wherein the video event representation is the event representation corresponding to the video information;

[0359] The determining module 840 is further configured to:

[0360] If the similarity of the events exceeds a similarity threshold, the video information corresponding to the video event representation will be determined as the retrieval result of the text information.

[0361] Figure 15 A block diagram of a training apparatus for an event representation prediction model provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0362] The acquisition module 850 is used to acquire a sample information group, which includes a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information correspond one-to-one with the n second modal information, where n is an integer greater than 1.

[0363] The prediction module 860 is used to perform prediction processing on the first information group to obtain a first event group of the first information group, and to perform prediction processing on the second information group to obtain a second event group of the first information group. The first event group is used to indicate the event information in the first information group, and the second event group is used to indicate the event information in the second information group.

[0364] The training module 870 is used to train the event representation prediction model based on the prediction error between the first event group and the second event group to obtain the trained event representation prediction model.

[0365] In an alternative design of the application, the first event group includes n first event representations corresponding to n first modal information, and the second event group includes n second event representations corresponding to n second modal information;

[0366] The training module 870 is also used for:

[0367] Calculate a first prediction error between the first event representation and the second event group; and calculate a second prediction error between the second event representation and the first event group;

[0368] The event representation prediction model is trained based on the first prediction error and the second prediction error to obtain the trained event representation prediction model.

[0369] In an optional design of the application, the acquisition module 850 is further configured to acquire a verification sample pair, the verification sample pair including first verification information and second verification information, wherein the first verification information and the second verification information have different information modalities;

[0370] The prediction module 860 is further configured to call the trained event representation prediction model to perform prediction processing on the first verification information to obtain a first verification representation, and to call the trained event representation prediction model to perform prediction processing on the second verification information to obtain a second verification representation. The first verification representation includes representation information of at least two first verification events, and the second verification representation includes representation information of at least two second verification events.

[0371] The update module 880 is further configured to verify the trained event representation prediction model based on the prediction error between the first verification representation and the second verification representation, and update the first number of the first verification events in the predicted first verification representation and the second number of the second verification events in the second verification representation.

[0372] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0373] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0374] This application also provides a computer device, which includes: a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the information retrieval method and / or the training method of the event representation prediction model provided in the above-described method embodiments.

[0375] Alternatively, the computer device is a server. For example, Figure 16 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0376] Typically, server 2300 includes a processor 2301 and memory 2302.

[0377] Processor 2301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 2301 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 2301 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 2301 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 2301 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0378] The memory 2302 may include one or more computer-readable storage media, which may be non-transitory. The memory 2302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 2302 is used to store at least one instruction, which is executed by the processor 2301 to implement the information retrieval method and / or the event representation prediction model training method provided in the method embodiments of this application.

[0379] In some embodiments, the server 2300 may optionally include an input interface 2303 and an output interface 2304. The processor 2301, memory 2302, and input interfaces 2303 and 2304 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 2303 and 2304 via a bus, signal lines, or a circuit board. The input interfaces 2303 and 2304 can be used to connect at least one input / output (I / O) related peripheral device to the processor 2301 and memory 2302. In some embodiments, the processor 2301, memory 2302, and input interfaces 2303 and 2304 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2301, memory 2302, and input interfaces 2303 and 2304 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0380] Those skilled in the art will understand that the structure shown above does not constitute a limitation on server 2300, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0381] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is run on a computer device, are used to implement the information retrieval method and / or the training method for an event representation prediction model as described above.

[0382] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the information retrieval method and / or the event representation prediction model training method provided in the above-described method embodiments.

[0383] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program loaded and executed by a processor to implement the information retrieval method and / or the training method for the event representation prediction model provided in the above-described method embodiments.

[0384] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0385] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An information retrieval method, characterized in that, The method includes: Acquire input information, wherein the information mode of the input information is a first mode; The input information is encoded using an event representation prediction model to obtain an input feature sequence. The input feature sequence is then predicted to obtain representation information for multiple first modal events of the input information. The representation information of the i-th first modal event is generated based on the representation information of the (i-1)-th first modal event, where i is a positive integer. The input feature sequence includes at least two input feature representations corresponding to at least two input sub-information in the input information. The first modal event representation is used to indicate event information in at least one input sub-information. The event information is obtained by extracting at least one of time, location, participating role, and participating action. Calculate the correlation matrix between the representation information of the plurality of first modal events and the representation information of at least one second modal event in the retrieval information. Take the average of the maximum values ​​in each column of the correlation matrix as the first similarity, take the average of the maximum values ​​in each row of the correlation matrix as the second similarity, and take the average of the first similarity and the second similarity as the event similarity. The second modal event representation is used to indicate the event information in at least one retrieval sub-information in the retrieval information, and the information modality of the retrieval information is the second modality. If the similarity of the events exceeds a similarity threshold, the retrieval information corresponding to the representation information of the at least one second modality event is determined as the retrieval result of the input information.

2. The method according to claim 1, characterized in that, The event representation prediction model includes a first prediction network corresponding to the first modality; The event representation prediction model performs encoding processing on the input information to obtain an input feature sequence, and performs prediction processing on the input feature sequence to obtain representation information of multiple first modal events of the input information, including: When the information mode of the input information is the first mode, the first prediction network is invoked to perform encoding processing on the input information to obtain an input feature sequence, and the input feature sequence is subjected to prediction processing to obtain the representation information of the multiple first mode events of the input information.

3. The method according to claim 2, characterized in that, The first prediction network includes a first modality encoder and a first event generator; When the information modality of the input information is the first modality, the first prediction network is invoked to perform encoding processing on the input information to obtain an input feature sequence. The input feature sequence is then subjected to prediction processing to obtain representation information of the plurality of first modality events of the input information, including: When the information mode of the input information is the first mode, the first mode encoder is invoked to encode at least two of the input sub-information information one by one to obtain the input feature sequence of the input information; The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the representation information of the multiple first modal events of the input information.

4. The method according to claim 3, characterized in that, The step of calling the first event generator to perform prediction processing on the input feature sequence to obtain the representation information of the multiple first modal events of the input information includes: The first event generator is invoked to perform prediction processing on the input feature sequence to obtain the input weight information of the input information. The input weight information is used to describe the weights of at least two input sub-information in the input information in multiple first modal events. Based on the input weight information and the input feature sequence, the representation information of the plurality of first modal events of the input information is determined.

5. The method according to claim 2, characterized in that, The event representation prediction model also includes a second prediction network corresponding to the second modality; The method further includes: When the information modality of the retrieved information is the second modality, the second prediction network is invoked to perform prediction processing on the retrieved information to obtain the representation information of the at least one second modality event of the retrieved information.

6. The method according to claim 5, characterized in that, The second prediction network includes a second modality encoder and a second event generator, and the retrieval information includes at least two retrieval sub-information; When the information modality of the retrieved information is the second modality, the second prediction network is invoked to perform prediction processing on the retrieved information to obtain representation information of at least one second modality event of the retrieved information, including: When the information mode of the retrieved information is the second mode, the second mode encoder is invoked to encode at least two of the retrieved sub-information one by one to obtain the retrieval feature sequence of the retrieved information. The retrieval feature sequence includes at least two retrieval feature representations corresponding to at least two of the retrieved sub-information. The second event generator is invoked to perform prediction processing on the retrieval feature sequence to obtain the representation information of at least one second modality event of the retrieval information.

7. The method according to claim 6, characterized in that, The step of calling the second event generator to perform prediction processing on the retrieved feature sequence to obtain the representation information of at least one second modality event of the retrieved information includes: The second event generator is invoked to perform prediction processing on the retrieval feature sequence to obtain retrieval weight information of the retrieval information. The retrieval weight information is used to describe the weight of at least two retrieval sub-information in the retrieval information in at least one second modality event. Based on the retrieval weight information and the retrieval feature sequence, the characterization information of the at least one second modality event of the retrieval information is determined.

8. The method according to claim 1, characterized in that, The first modality is a text modality, and the second modality is a video modality; The acquisition of input information includes: Retrieve text information; The event representation prediction model performs encoding processing on the input information to obtain an input feature sequence, and performs prediction processing on the input feature sequence to obtain representation information of multiple first modal events of the input information, including: The event representation prediction model is invoked to encode the text information to obtain a text feature sequence. The text feature sequence is then used for prediction to obtain representation information of multiple text modal events of the text information. The step of calculating a correlation matrix between the representation information of the plurality of first modal events and the representation information of at least one second modal event of the retrieval information, using the average of the maximum values ​​in each column of the correlation matrix as a first similarity, the average of the maximum values ​​in each row of the correlation matrix as a second similarity, and the average of the first similarity and the second similarity as the event similarity, includes: Calculate the correlation matrix between the representation information of the plurality of text modal events and the representation information of at least one video modal event. Take the average of the maximum values ​​in each column of the correlation matrix as the first similarity, take the average of the maximum values ​​in each row of the correlation matrix as the second similarity, and take the average of the first similarity and the second similarity as the event similarity. The representation information of the video modal event is the event representation corresponding to the video information. When the event similarity exceeds a similarity threshold, determining the retrieval information corresponding to the representation information of the at least one second modality event as the retrieval result of the input information includes: If the similarity of the events exceeds a similarity threshold, the video information corresponding to the representation information of the at least one video modal event is determined as the retrieval result of the text information.

9. A training method for an event representation prediction model, characterized in that, The method is used to train the event representation prediction model in any one of claims 1 to 8, the method comprising: Obtain a sample information group, which includes a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information correspond one-to-one with the n second modal information, where n is an integer greater than 1. The first information group is subjected to prediction processing to obtain a first event group of the first information group, and the second information group is subjected to prediction processing to obtain a second event group of the first information group. The first event group is used to indicate the event information in the first information group, and the second event group is used to indicate the event information in the second information group. Based on the prediction error between the first event group and the second event group, the event representation prediction model is trained to obtain the trained event representation prediction model.

10. The method according to claim 9, characterized in that, The first event group includes n first event representations corresponding to n first modal information, and the second event group includes n second event representations corresponding to n second modal information; The step of training the event representation prediction model based on the prediction error between the first event group and the second event group to obtain the trained event representation prediction model includes: Calculate a first prediction error between the first event representation and the second event group; and calculate a second prediction error between the second event representation and the first event group; The event representation prediction model is trained based on the first prediction error and the second prediction error to obtain the trained event representation prediction model.

11. The method according to claim 9, characterized in that, The method further includes: Obtain a verification sample pair, the verification sample pair including first verification information and second verification information, the first verification information and the second verification information having different information modalities; The trained event representation prediction model is invoked to predict the first verification information to obtain a first verification representation, and the trained event representation prediction model is invoked to predict the second verification information to obtain a second verification representation. The first verification representation includes representation information of at least two first verification events, and the second verification representation includes representation information of at least two second verification events. Based on the prediction error between the first verification representation and the second verification representation, the trained event representation prediction model is validated, and the first number of the first verification events in the predicted first verification representation and the second number of the second verification events in the predicted second verification representation are updated.

12. An information retrieval device, characterized in that, The device includes: The acquisition module is used to acquire input information, wherein the information mode of the input information is a first mode; The prediction module is used to call the event representation prediction model to perform encoding processing on the input information to obtain an input feature sequence, and to perform prediction processing on the input feature sequence to obtain representation information of multiple first modal events of the input information. The representation information of the i-th first modal event is generated based on the representation information of the (i-1)-th first modal event, where i is a positive integer. The input feature sequence includes at least two input feature representations corresponding to at least two input sub-information in the input information. The first modal event representation is used to indicate event information in at least one input sub-information. The event information is obtained by extracting at least one of time, location, participating role, and participating action. The calculation module is used to calculate a correlation matrix between the representation information of the plurality of first modal events and the representation information of at least one second modal event of the retrieval information. The average of the maximum values ​​in each column of the correlation matrix is ​​used as the first similarity, the average of the maximum values ​​in each row of the correlation matrix is ​​used as the second similarity, and the average of the first similarity and the second similarity is used as the event similarity. The second modal event representation is used to indicate the event information in at least one retrieval sub-information in the retrieval information, and the information modality of the retrieval information is the second modality. The determination module is used to determine the retrieval information corresponding to the representation information of the at least one second modality event as the retrieval result of the input information when the event similarity exceeds a similarity threshold.

13. The apparatus according to claim 12, characterized in that, The event representation prediction model includes a first prediction network corresponding to the first modality; The prediction module is further configured to: when the information mode of the input information is the first mode, call the first prediction network to perform encoding processing on the input information to obtain an input feature sequence, perform prediction processing on the input feature sequence to obtain the representation information of the plurality of first mode events of the input information.

14. The apparatus according to claim 13, characterized in that, The first prediction network includes a first modality encoder and a first event generator; The prediction module is further configured to: when the information mode of the input information is the first mode, call the first mode encoder to encode at least two of the input sub-information one by one to obtain the input feature sequence of the input information; and call the first event generator to perform prediction processing on the input feature sequence to obtain the representation information of the plurality of first mode events of the input information.

15. The apparatus according to claim 14, characterized in that, The prediction module is further configured to: call the first event generator to perform prediction processing on the input feature sequence to obtain input weight information of the input information, wherein the input weight information is used to describe the weights of at least two input sub-information in the input information in multiple first modal events; and determine the representation information of the multiple first modal events of the input information based on the input weight information and the input feature sequence.

16. The apparatus according to claim 13, characterized in that, The event representation prediction model also includes a second prediction network corresponding to the second modality; The prediction module is further configured to: when the information mode of the retrieved information is the second mode, invoke the second prediction network to perform prediction processing on the retrieved information to obtain the representation information of the at least one second mode event of the retrieved information.

17. The apparatus according to claim 16, characterized in that, The second prediction network includes a second modality encoder and a second event generator, and the retrieval information includes at least two retrieval sub-information; The prediction module is further configured to: when the information modality of the retrieved information is the second modality, call the second modality encoder to encode at least two of the retrieved sub-information one by one to obtain the retrieval feature sequence of the retrieval information, the retrieval feature sequence including at least two retrieval feature representations corresponding to at least two of the retrieved sub-information; call the second event generator to perform prediction processing on the retrieval feature sequence to obtain the representation information of the at least one second modality event of the retrieval information.

18. The apparatus according to claim 17, characterized in that, The prediction module is further configured to: call the second event generator to perform prediction processing on the retrieval feature sequence to obtain retrieval weight information of the retrieval information, wherein the retrieval weight information is used to describe the weight of at least two retrieval sub-information in the retrieval information in at least one second modal event; and determine the representation information of the at least one second modal event of the retrieval information based on the retrieval weight information and the retrieval feature sequence.

19. The apparatus according to claim 12, characterized in that, The first modality is a text modality, and the second modality is a video modality; The acquisition module is also used to: acquire text information; The prediction module is further configured to: call the event representation prediction model to perform encoding processing on the text information to obtain a text feature sequence, perform prediction processing on the text feature sequence to obtain representation information of multiple text modal events of the text information; The calculation module is further configured to: calculate a correlation matrix between the representation information of the plurality of text modal events and the representation information of at least one video modal event, take the average of the maximum values ​​of each column in the correlation matrix as the first similarity, take the average of the maximum values ​​of each row in the correlation matrix as the second similarity, and take the average of the first similarity and the second similarity as the event similarity, wherein the representation information of the video modal event is the event representation corresponding to the video information; The determining module is further configured to: when the event similarity exceeds a similarity threshold, determine the video information corresponding to the representation information of the at least one video modal event as the retrieval result of the text information.

20. A training device for an event representation prediction model, characterized in that, The apparatus is used to train the event representation prediction model in any one of claims 1 to 8, and the apparatus comprises: The acquisition module is used to acquire sample information groups, which include a first information group and a second information group. The first information group includes n first modal information and the second information group includes n second modal information. The information modalities of the first modal information and the second modal information are different. The n first modal information correspond one-to-one with the n second modal information, where n is an integer greater than 1. The prediction module is used to perform prediction processing on the first information group to obtain a first event group of the first information group, and to perform prediction processing on the second information group to obtain a second event group of the first information group. The first event group is used to indicate the event information in the first information group, and the second event group is used to indicate the event information in the second information group. The training module is used to train the event representation prediction model based on the prediction error between the first event group and the second event group, so as to obtain the trained event representation prediction model.

21. The apparatus according to claim 20, characterized in that, The first event group includes n first event representations corresponding to n first modal information, and the second event group includes n second event representations corresponding to n second modal information; The training module is also used to: calculate a first prediction error between the first event representation and the second event group; And calculate the second prediction error between the second event representation and the first event group; The event representation prediction model is trained based on the first prediction error and the second prediction error to obtain the trained event representation prediction model.

22. The apparatus according to claim 20, characterized in that, The acquisition module is further configured to acquire a verification sample pair, the verification sample pair including first verification information and second verification information, wherein the first verification information and the second verification information have different information modalities; The prediction module is further configured to call the trained event representation prediction model to perform prediction processing on the first verification information to obtain a first verification representation, and to call the trained event representation prediction model to perform prediction processing on the second verification information to obtain a second verification representation. The first verification representation includes representation information of at least two first verification events, and the second verification representation includes representation information of at least two second verification events. The update module is further configured to verify the trained event representation prediction model based on the prediction error between the first verification representation and the second verification representation, and update the first number of the first verification events in the predicted first verification representation and the second number of the second verification events in the second verification representation.

23. A computer device, characterized in that, The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is configured to execute the at least one program in the memory to implement the information retrieval method as described in any one of claims 1 to 8, and / or the training method for the event representation prediction model as described in any one of claims 9 to 11.

24. A computer-readable storage medium, characterized in that, The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the information retrieval method as described in any one of claims 1 to 8, and / or the training method for the event representation prediction model as described in any one of claims 9 to 11.

25. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the information retrieval method as described in any one of claims 1 to 8, and / or the training method for the event representation prediction model as described in any one of claims 9 to 11.

Images