Information retrieval method, device, storage medium, and program product

Abstract

This invention provides an information retrieval method, device, storage medium, and program product. The method includes: acquiring a representation model trained on a training dataset, wherein different retrieval tasks correspond to different data modality combinations in the training data pairs, and the text data used to describe image data in the image-text pairs includes visual words that meet a set density condition; acquiring retrieval input data and multiple retrieved data corresponding to the target retrieval task; determining the embedding representation vectors corresponding to the retrieval input data and the multiple retrieved data respectively through the representation model; and selecting the target retrieved data from the multiple retrieved data based on the similarity between the two. The training dataset uses dense multimodal knowledge and training data from various retrieval tasks, improving the multimodal alignment capability of the representation model, enabling it to adapt to different types of retrieval tasks, and improving the accuracy of retrieval results.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an information retrieval method, device, storage medium, and program product. Background Technology

[0002] Since the introduction of Contrastive Language-Image Pre-training (CLIP) models based on contrastive learning, multimodal representation tasks have received widespread attention. A well-trained CLIP model can take a piece of text (or an image) as input and output a vector representation of the text (or image). Therefore, CLIP models can be used to perform cross-modal retrieval tasks, such as taking a piece of text as input and retrieving images that match that text from a large pool of images. Here, "modality" (also called data modality) refers to the data type, which typically includes text, images, and videos.

[0003] While CLIP series models perform well on cross-modal retrieval tasks, they perform poorly on combined modal retrieval tasks. Combined modal retrieval tasks, for example, involve combining images and text to retrieve other images or text.

[0004] Therefore, there is an urgent need to provide a unified multimodal representation model that can be universally applied to retrieval tasks with various combinations of data modalities, so that various retrieval tasks can be accurately completed based on this representation model. Summary of the Invention

[0005] This invention provides an information retrieval method, device, storage medium, and program product that accurately completes retrieval tasks under various modal combinations through a unified representation model.

[0006] In a first aspect, embodiments of the present invention provide an information retrieval method, the method comprising:

[0007] Obtain a representation model trained on a training dataset, wherein the training dataset corresponds to training data pairs for various retrieval tasks, and the training data pairs for different retrieval tasks correspond to different combinations of data modalities. The training data pairs include image-text pairs, and the text data used to describe image data in the image-text pairs includes visual words that meet a set density condition.

[0008] Obtain retrieval data corresponding to the target retrieval task, wherein the target retrieval task is any one of the multiple retrieval tasks, and the retrieval data includes retrieval input data and multiple retrieved data;

[0009] The embedding representation vectors corresponding to the retrieval input data and the plurality of retrieved data are determined through the representation model;

[0010] Based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data, the target retrieved data is selected from the plurality of retrieved data.

[0011] Secondly, embodiments of the present invention provide an information retrieval device, the device comprising:

[0012] The first acquisition module is used to acquire the representation model trained based on the training dataset. The training dataset corresponds to training data pairs for various retrieval tasks. The training data pairs for different retrieval tasks correspond to different combinations of data modes. The training data pairs include image-text pairs. The text data used to describe the image data in the image-text pairs includes visual words that meet the set density conditions.

[0013] The second acquisition module is used to acquire retrieval data corresponding to the target retrieval task, wherein the target retrieval task is any one of the multiple retrieval tasks, and the retrieval data includes retrieval input data and multiple retrieved data.

[0014] An embedding processing module is used to determine the embedding representation vectors corresponding to the retrieval input data and the plurality of retrieved data respectively through the representation model;

[0015] The data retrieval module is used to filter out target retrieved data from the plurality of retrieved data based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor performs the information retrieval method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, wherein when the executable code is executed by a processor of an electronic device, the processor is able to at least implement the information retrieval method as described in the first aspect.

[0018] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, can implement the information retrieval method as described in the first aspect.

[0019] In the information retrieval scheme provided by this invention, a training dataset corresponding to multiple retrieval tasks is obtained. This training dataset includes training data pairs corresponding to various retrieval tasks. Since these retrieval tasks include single-modal retrieval tasks, cross-modal retrieval tasks, and combined modal retrieval tasks, the combination of data modalities corresponding to the training data pairs for each retrieval task is different. Then, a representation model is trained based on this training dataset. The training dataset includes image-text pairs (e.g., corresponding to cross-modal retrieval tasks). To improve the multimodal alignment capability of the representation model, the text data describing image data in the image-text pairs includes visual words that meet a set density condition. In other words, the text data used to train the representation model contains dense multimodal knowledge describing visual information, thereby improving the visual-text alignment capability of the representation model.

[0020] Subsequently, when performing any target retrieval task using this representation model, the corresponding retrieval data is obtained. The retrieval data includes retrieval input data and multiple retrieved data. The embedding representation vectors corresponding to the retrieval input data and multiple retrieved data are determined through the representation model. Based on the similarity between the embedding representation vectors corresponding to the retrieval input data and the embedding representation vectors corresponding to the multiple retrieved data, the target retrieved data is selected from the multiple retrieved data.

[0021] In the above scheme, the training dataset uses dense multimodal knowledge to enable better alignment of image and text information. By training the representation model based on this training dataset, the multimodal alignment capability of the representation model can be improved, enabling it to adapt to different types of retrieval tasks such as cross-modal and combined modal retrieval, thereby improving the accuracy of retrieval results. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the error situation in negative samples under a cross-modal retrieval task provided in an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of the composition structure of the characterization model provided in the embodiments of the present invention;

[0025] Figure 3 A flowchart of an information retrieval method provided in an embodiment of the present invention;

[0026] Figure 4 A flowchart for filtering retrieved data based on a ranking model is provided in this embodiment of the invention;

[0027] Figure 5 A flowchart illustrating a training method for a representation model provided in an embodiment of the present invention;

[0028] Figure 6 A schematic diagram illustrating the training process of the representation model provided in an embodiment of the present invention;

[0029] Figure 7 A flowchart of another training method for a representation model provided in an embodiment of the present invention;

[0030] Figure 8 A schematic diagram illustrating the training process of the representation model provided in an embodiment of the present invention;

[0031] Figure 9 A flowchart of another training method for a representation model provided in an embodiment of the present invention;

[0032] Figure 10 This is a schematic diagram of the structure of an information retrieval device provided in an embodiment of the present invention;

[0033] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the timing of the steps in the following method embodiments is only an example and not a strict limitation.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0036] The following is a brief introduction to some concepts involved in the embodiments of the present invention.

[0037] Representation Model: A model that can compress input information into a vector, which can be used for subsequent retrieval tasks.

[0038] Large Language Model (LLM): A model that is pre-trained on a large amount of data and has a great understanding of language structure and common sense knowledge.

[0039] Multimodal Large Language Model (MLLM): Building upon the large language model, this model extends the understanding of input modalities beyond text. Multimodal typically refers to data modalities such as text, images, and video.

[0040] Multimodal knowledge refers to information that helps to better connect visual and textual information, and usually includes some visual words in the text.

[0041] Dense knowledge: This refers to dense multimodal knowledge, meaning that a text contains a denser amount of multimodal knowledge.

[0042] Cross-modal retrieval refers to the retrieval of textual and visual information, with visual information mainly consisting of images and videos.

[0043] Combined modal retrieval: also known as multimodal combined retrieval, refers to the retrieval of other visual or textual information by combining visual and textual information.

[0044] In practical applications, a core scenario is that users want to search for content in their self-built multimodal databases. However, the performance of existing multimodal representation models often fails to meet the requirements in cross-modal retrieval and combined modal retrieval tasks. Therefore, it is necessary to optimize the multimodal representation capabilities of existing representation models.

[0045] Combined modality retrieval has always been a key component in search systems and retrieval enhancement applications. Due to the good cross-modal retrieval performance of CLIP models, models from the CLIP series are widely used. However, CLIP models perform poorly on combined modality retrieval tasks. In the text domain, LLM has achieved remarkable results in text representation, but in the multimodal domain, due to performance and training resource issues, CLIP models still dominate. There is little exploration of MLLM-based multimodal representation (the task of vector representation of combined data from multiple modalities), especially regarding high-quality training data. Previous research on vector representation of multimodal data based on MLLM has mainly focused on mapping data from different modalities to a unified embedding space, neglecting the importance of aligning multimodal information. As a result, the cross-modal retrieval performance of traditional MLLM-based representation models lags significantly behind CLIP models. Therefore, research on a unified representation model that balances performance across different types of retrieval tasks, such as cross-modal and combined modality retrieval, has become a key research direction.

[0046] Therefore, in this embodiment of the invention, firstly, a training dataset for contrastive learning is constructed, rich in dense multimodal knowledge, which can effectively enhance the multimodal alignment capability in representation tasks. Secondly, based on this training dataset, a unified multimodal representation model is trained, which is trained based on MLLM. By comparing with other types of representation models (such as CLIP series models), it is found that the representation model provided by this embodiment of the invention not only aligns text, images, videos, and text-image combinations within a unified embedding space (i.e., the representation model can simultaneously support vector representation processing of plain text, images, videos, and data combinations of the above-mentioned modalities), but also achieves better performance in combined modal retrieval scenarios and achieves cross-modal retrieval performance comparable to CLIP series models.

[0047] The inventors' research revealed that previous studies on multimodal representation based on MLLM had a common problem: their cross-modal performance was significantly inferior to that of CLIP series models. This gap was due to the fact that their training data did not activate the multimodal alignment capability in MLLM.

[0048] Previously, MLLM-based representation models exhibited weak cross-modal retrieval performance, reflecting a deficiency in their multimodal alignment capabilities. Referring to the CLIP series models, achieving strong multimodal alignment in a representation model typically requires billion-level training data pairs. However, the model parameter data volume of MLLM is often many times larger than that of CLIP series models, resulting in unacceptable training overhead. Furthermore, since the training data volume for current combined modality retrieval tasks is generally only at the millillion level, blindly introducing additional billion-level training data from other retrieval tasks can easily lead to knowledge forgetting. Therefore, improving the efficiency of multimodal alignment is crucial for MLLM-based representation models.

[0049] In view of this, such as Figure 1 As shown, analysis of training data for cross-modal retrieval tasks reveals three common errors in negative samples with higher confidence: action errors, object errors, and description errors (or appearance description errors). Action errors include, for example, the act of "riding" a motorcycle; object errors include the state of an orange; and description errors include the color of a car. These errors are concentrated on visually relevant words, indicating that these visual words are crucial for solving cross-modal retrieval problems. In this embodiment, words that align image (visual) and text modalities are referred to as multimodal knowledge. Generally, assuming text descriptions correspond well to image content, this knowledge can be mapped to words of different parts of speech: for example, action knowledge maps to verbs, object knowledge maps to nouns, and description knowledge maps to adjectives. Furthermore, comparing dense and sparse knowledge data reveals that dense knowledge data contains far more multimodal knowledge than sparse knowledge data, providing a more comprehensive and detailed description of almost all content in the image. Sparse knowledge data, compared to dense knowledge data, contains fewer words describing the visual features of an image.

[0050] Therefore, in creating a training dataset adaptable to various retrieval tasks, this embodiment of the invention collects a large number of image-text pairs as part of the training data pairs. The text data used to describe the image data in the image-text pairs includes visual words that meet a set density condition, such as a text data containing at least 50 knowledge points (i.e., words describing vision). Such dense visual description data can significantly improve the efficiency of alignment between images and text, enabling the multimodal alignment capability of the representation model to be effectively improved without relying on a larger training dataset.

[0051] In this context, image-text pairs can correspond to cross-modal retrieval tasks (e.g., retrieving text from an image, or retrieving an image from text) or combined modal retrieval tasks (e.g., retrieving an image and / or text using a combined modality of image and text). In summary, the training dataset includes training data pairs corresponding to various retrieval tasks. Different training data pairs for different retrieval tasks correspond to different combinations of data modalities. For example, in a cross-modal retrieval task, the data modality used as input may be one type, while the data modality used as output may be another. Similarly, in a cross-modal retrieval task, there may be multiple data modalities used as input, while the data modality used as output may be one or more types. Furthermore, in a single-modal retrieval task, there may be one type of data modality used as input, while the data modality used as output may be the same type.

[0052] Therefore, for different types of retrieval tasks, multiple training data pairs can be collected from known open-source datasets and added to the training dataset provided in this embodiment of the invention. For example, cross-modal video-text data pairs can be collected from known "video-text" datasets; single-modal data pairs of text and images can be collected from "text-text" and "image-image" datasets; cross-modal image-text data pairs can be collected from "image-text" datasets; and multi-modal combined data pairs can be collected from multi-modal combined datasets, such as data pairs consisting of a combined modality of image and text and text, or data pairs consisting of a combined modality of image and text and images.

[0053] To facilitate understanding of combined modal retrieval task scenarios, examples are provided below. For instance, using image + text data as retrieval input, the task might be a visual question-answering task, such as image a + text data 1: "How many birds are in this picture?", retrieving text data 2: "There are 3 yellow birds in the image." Another example is using image + text data as retrieval input, the task might be an image editing task, such as image b + text data: "Change the clothes in the picture to yellow", retrieving the edited image c.

[0054] As described above, in this embodiment of the invention, to improve the multimodal alignment efficiency of the representation model, the main focus is on optimizing the image-text pair data. Specifically, this may include the following steps:

[0055] Obtain the image and text dataset corresponding to the image and text retrieval task, which is included in various retrieval tasks corresponding to the training dataset;

[0056] For the text data in this image-text dataset, visual words with multiple parts of speech are labeled.

[0057] Based on the annotation results of visual words of various parts of speech, text data in the image and text dataset that meet the set density conditions are selected from the annotated visual words.

[0058] The selected text data and its corresponding image data are added to the training dataset.

[0059] In practical applications, this image-text retrieval task can be a cross-modal retrieval task, such as retrieving text from an image or retrieving an image from text. The corresponding image-text dataset includes several image-text pairs, where the text data in each pair describes the content contained in the corresponding image data. In reality, some text data may be shorter, meaning the description is coarser, while other text data may be longer, providing a more detailed description.

[0060] As analyzed above, the richer the visual descriptive words contained in the text data, the more positive the impact on the accuracy of the retrieval task. Therefore, in this embodiment of the invention, for each image-text pair in the image-text dataset, the text data can first be labeled with visual words of various parts of speech, such as names for describing objects, verbs for describing actions, and adjectives for describing appearance, as mentioned above. Then, based on the labeling results of these visual words, text data that meets a set density condition is selected. This density condition is, for example, that the number of labeled visual words is greater than or equal to a preset value. Finally, the selected text data and its corresponding image data from the image-text dataset are added as the selected image-text pairs to the training dataset used to train the representation model.

[0061] In practical applications, the selected image-text pairs can undergo further data cleaning. For image data, this includes filtering by image resolution (removing lower-resolution images) and deduplicating images. For text data, this includes deduplicating different text sets, filtering out incoherent sentences, and removing special symbols or sensitive words.

[0062] Similarly, other training data pairs added to the training dataset can also undergo corresponding data cleaning processes. For example, for multimodal training data pairs, existing multimodal representation models such as the CLIP model can be used to calculate alignment scores and filter out irrelevant training data pairs with low alignment scores.

[0063] After obtaining the training dataset, the training dataset can be used to train the unified representation model in the embodiments of the present invention, and to perform various retrieval tasks based on the trained representation model.

[0064] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features thereof can be combined with each other.

[0065] First, combine Figure 2 This illustrates an optional component structure of the characterization model provided in an embodiment of the present invention, such as... Figure 2 As shown, the representation model includes a first MLLM, a second MLLM, a first embedding transformation model connected to the MLLM (illustrated as the first embed head in the figure), and a second embedding transformation model connected to the second MLLM (illustrated as the second embed head in the figure). Optionally, as... Figure 2 As shown, the representation model also includes a ranking model (shown as rankhead in the figure) connected to the first MLLM.

[0066] The two MLLMs mentioned above have the same structure, such as Figure 2 As shown, the data may include a vision encoder, a projector, and an LLM. In this embodiment of the invention, it is assumed that, apart from text data, the main modalities of the multimodal data are visual, such as image data and video data. Therefore, the aforementioned vision encoder may be included. However, in practical applications, in addition to visual data, data such as audio data may also be included. In such cases, the vision encoder can be replaced with a "multimodal encoder" for feature vector encoding of image, video, and audio data.

[0067] A visual encoder encodes input visual data (such as image data or video frame data) into a vector space. A mapper maps the feature vectors output by the visual encoder to the text token space required by the LLM, enabling the LLM to better understand these features. Pre-trained LLMs, containing extensive knowledge, possess more fine-grained token-level understanding capabilities.

[0068] The first and second embed heads, for example, can be linear layers composed of multilayer perceptrons, used to map the output vectors of their respective LLMs to a specified embedding dimension. Similarly, the rank head can also be a linear layer composed of multilayer perceptrons, whose output dimension can be considered to involve only two dimensions: whether it matches (or the probability of matching and the probability of not matching), which will be explained in detail below.

[0069] The embodiments of the present invention do not specifically limit the implementation structure of the above-mentioned constituent units in the representation model.

[0070] The training process of the above representation model will not be introduced for the time being. Instead, the process of performing information retrieval based on the trained representation model will be introduced first.

[0071] Figure 3 A flowchart of an information retrieval method provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method includes the following steps:

[0072] 301. Obtain the representation model trained based on the training dataset. The training dataset corresponds to training data pairs for various retrieval tasks. The training data pairs for different retrieval tasks correspond to different combinations of data modalities. Among them, the training data pairs include image-text pairs. The text data used to describe the image data in the image-text pairs includes visual words that meet the set density conditions.

[0073] 302. Obtain the retrieval data corresponding to the target retrieval task. The target retrieval task can be any one of multiple retrieval tasks. The retrieval data includes retrieval input data and multiple retrieved data.

[0074] 303. Determine the embedding representation vectors corresponding to the retrieval input data and multiple retrieved data through the representation model.

[0075] 304. Based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data, the target retrieved data is selected from the multiple retrieved data.

[0076] After obtaining the representation model trained by the training dataset described above, the representation model can be used to perform retrieval tasks. Taking any of the above retrieval tasks as the target retrieval task in this embodiment as an example, firstly, the retrieval data corresponding to the target retrieval task is obtained. The retrieval data includes retrieval input data and multiple retrieved data.

[0077] Then, the embedding representation vectors corresponding to the retrieval input data and each of the multiple retrieved data are determined through a representation model. Based on Figure 2 The representation model shown specifically involves inputting the retrieval input data into a first MLLM to obtain the embedding representation vector corresponding to the retrieval input data output by the first embedding head. Multiple retrieved data are then input into a second MLLM to obtain the embedding representation vectors corresponding to each of the retrieved data output by the second embedding head.

[0078] In practical applications, if the target retrieval task is a cross-modal retrieval task, then the aforementioned retrieval input data is data of the first modality, and the multiple retrieved data are data of the second modality. In this case, the inputs of both the first MLLM and the second MLLM are only data of a single modality. For example, in a scenario where image data is used to retrieve text data, the retrieval input data is a certain image data, and the multiple retrieved data are multiple text data. Similarly, in a scenario where text data is used to retrieve image data, the retrieval input data is a certain text data, and the multiple retrieved data are multiple image data.

[0079] Taking text data as the input data and multiple image data as the retrieved data as an example, the text data is split into multiple tokens according to a defined token definition and then vector-encoded (the vector before input to the LLM is referred to as the token vector in this paper). This encoded token vector is then input into the LLM of the first MLLM, and the token vector output by the LLM is input into the first embedding head to obtain the embedding vector representation corresponding to the text data. For any image data, the text is input into the visual encoder of the second MLLM for vector encoding. The encoded feature vector is then input into the mapper of the second MLLM, mapped into multiple token vectors, which are then input into the LLM of the second MLLM. The token vector output by the LLM is then input into the second embedding head to obtain the embedding vector representation corresponding to the image data. The same processing is performed on other image data to obtain the embedding vector representation for each image data.

[0080] It's important to clarify two points here. First, for text data, it's not input into the visual encoder and mapper of the corresponding MLLM; instead, it's directly input into the corresponding LLM. Second, the LLM's input and output consist of an equal number of token vectors, which can be represented as y = MLLM(x), where x represents the token sequence input to the LLM, and y represents the token sequence output by the LLM, both having the same number of token vectors. Optionally, the last token vector in the token sequence y can be used as the input to the embed head of the corresponding MLLM. Of course, this is not a limitation; other implementations, such as the average of multiple token vectors in a token sequence, are also possible.

[0081] In other words, optionally, the retrieval input data can be input into a first MLLM to obtain the last token vector output by the first MLLM, and this last token vector can be input into a first embedding head to obtain the embedding representation vector corresponding to the retrieval input data. Multiple retrieved data can be input into a second MLLM to obtain the last token vectors corresponding to each of the multiple retrieved data outputs by the second MLLM, and these last token vectors can be sequentially input into a second embedding head to obtain the embedding representation vectors corresponding to each of the multiple retrieved data.

[0082] Taking the target retrieval task as a cross-modal retrieval task as an example, this paper illustrates the process of determining the corresponding retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data.

[0083] The following example, using a combined modal retrieval task as an example, illustrates the process of determining the corresponding retrieval input data and the embedding representation vectors for each of the multiple retrieved data. Assume the target retrieval task is to retrieve image data using a combination of image and text data. For ease of description, assume the retrieval input data includes image data 1 and text data 1, and the multiple retrieved data include image data 2 through image data 9. In this case, the input data for the first MLLM involves both image and text modal data, while the input data for the second MLLM involves only the single modal data of image data.

[0084] The process of inputting image data 2 to image data 9 into the second MLLM to output their respective embedded representation vectors through the second embed head can be referred to the image data processing process in the cross-modal retrieval task described above, and will not be repeated here.

[0085] This section only describes the process of processing image data 1 and text data 1 into corresponding embedding representation vectors using the first MLLM and the first embedding head. Specifically, similar to the processing of text data in the cross-modal retrieval task described above, text data 1 is encoded into multiple token vectors, which serve as part of the input to the LLM in the first MLLM. In addition, for image data 1, the visual encoder in the first MLLM is input, and the feature vector output by the visual encoder is mapped through a corresponding mapper to obtain multiple token vectors corresponding to image data 1, which serve as another part of the input to the LLM in the first MLLM. That is, the input to the LLM in the first MLLM is the concatenation result of multiple token vectors corresponding to text data 1 and multiple token vectors corresponding to image data 1 (for example, if text data 1 corresponds to 4 token vectors and image data 1 corresponds to 2 token vectors, the concatenation result is 6 token vectors). Then, the last of the multiple token vectors output by the LLM in the first MLLM is input to the first embedding head to obtain its output embedding representation vector, which is the embedding representation vector corresponding to the retrieval input data composed of image data 1 and text data 1.

[0086] After obtaining the retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data through the above process, the target retrieved data is selected from the multiple retrieved data based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data. The calculation of similarity includes, but is not limited to, the calculation of cosine similarity.

[0087] In practical applications, optionally, the target data to be retrieved can be the data with the highest similarity among multiple data to be retrieved.

[0088] In the information retrieval scheme provided by this invention, a training dataset corresponding to multiple retrieval tasks is obtained. This training dataset includes training data pairs corresponding to various retrieval tasks. Since these retrieval tasks include single-modal retrieval tasks, cross-modal retrieval tasks, and combined modal retrieval tasks, the combination of data modalities corresponding to the training data pairs for each retrieval task is different. Then, a representation model is trained based on this training dataset. The training dataset includes image-text pairs (e.g., corresponding to cross-modal retrieval tasks). To improve the multimodal alignment capability of the representation model, the text data describing image data in the image-text pairs includes visual words that meet a set density condition. In other words, the text data used to train the representation model contains dense multimodal knowledge describing visual information, thereby improving the visual-text alignment capability of the representation model.

[0089] Subsequently, when performing any target retrieval task using this representation model, the corresponding retrieval data is obtained. The retrieval data includes retrieval input data and multiple retrieved data. The embedding representation vectors corresponding to the retrieval input data and multiple retrieved data are determined through the representation model. Based on the similarity between the embedding representation vectors corresponding to the retrieval input data and the embedding representation vectors corresponding to the multiple retrieved data, the target retrieved data is selected from the multiple retrieved data.

[0090] In the above scheme, the training dataset uses dense multimodal knowledge to enable better alignment of image and text information. By training the representation model based on this training dataset, the multimodal alignment capability of the representation model can be improved, enabling it to adapt to different types of retrieval tasks such as cross-modal and combined modal retrieval, thereby improving the accuracy of retrieval results.

[0091] In another alternative embodiment, the representation model includes Figure 2 Based on the sorting model shown, such as Figure 4 As shown, the process of determining the target data to be retrieved can also be achieved through the following steps:

[0092] 401. Based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data, a set number of candidate retrieved data are selected from the multiple retrieved data.

[0093] 402. Generate multiple sorted data combinations, wherein each sorted data combination includes retrieval input data and a candidate retrieval data.

[0094] 403. Input multiple sorted data combinations into the first multimodal large language model to obtain the matching scores corresponding to each sorted data combination output by the sorting model.

[0095] 404. Based on the matching scores corresponding to each of the multiple sorted data combinations, the target data to be retrieved is selected from the set number of candidate retrieval data.

[0096] In this embodiment, it is assumed that the set quantity in step 401 is 10, that is, according to the ranking of the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vectors corresponding to each of the multiple retrieved data, the 10 candidate retrieved data with the highest similarity are selected from the multiple retrieved data.

[0097] Then, the retrieval input data is combined with these 10 candidate retrieved data to obtain 10 ranked data combinations. Each ranked data combination includes the retrieval input data and one of the candidate retrieved data.

[0098] Since this embodiment of the invention assumes that the sorting model is connected to the first MLLM side, the 10 sorting data are combined and input into the first MLLM respectively. In fact, the same applies if the sorting model is connected to the second MLLM side.

[0099] It is understandable that, regardless of whether the target retrieval task is a cross-modal retrieval task, a combined modal retrieval task, or a single modal retrieval task, each ranked data combination includes at least two data points. These at least two data points may be of the same modality (e.g., in a single modal retrieval task) or of different modalities (e.g., in cross-modal and combined modal retrieval tasks). For each ranked data combination containing at least two data points, the processing procedure is similar to that for the retrieval input data in the combined modal retrieval task described above. Multiple token vectors corresponding to this ranked data combination need to be output through the LLM in the first MLLM. The specific process will not be elaborated further.

[0100] Optionally, for each ranked data combination output by the LLM, the last token vector can be selected and input into the rank head to obtain the matching score corresponding to the ranked data combination. For example, if a ranked data combination includes text data t and image data i, the rank head output might be: Match - 0.85, No Match - 0.15, indicating that the matching probability between text data t and image data i is 85%, and the no match probability is 15%. For each ranked data combination, the ranked data combination with the highest matching probability (i.e., matching score) is selected. The candidate retrieved data contained in this ranked data combination is the target retrieved data that best matches the retrieval input data.

[0101] In this embodiment, the similarity between the retrieval input data and each retrieved data is calculated based on the input of the embed heads corresponding to the two MLLMs. The retrieved data with higher similarity is selected as candidates. The matching score between the retrieval input data and the candidate retrieved data is then calculated using a ranking model. Compared with the method of directly selecting target retrieved data based on similarity, the retrieval accuracy is higher due to the further analysis of matching.

[0102] The training process of the above representation model will be described below with reference to some examples.

[0103] Figure 5 A flowchart illustrating a training method for a representation model provided in an embodiment of the present invention is shown below. Figure 5 As shown, the training method includes the following steps:

[0104] 501. Obtain the target training data pair from the training dataset, the target training data pair including first training data used as retrieval input and second training data used as retrieval output.

[0105] 502. Input the first training data into the first MLLM to obtain the embedding representation vector corresponding to the first training data output by the first embed head.

[0106] 503. Input the second training data into the second MLLM to obtain the embedding representation vector corresponding to the second training data output by the second embed head, wherein the first MLLM and the second MLLM share parameters before training.

[0107] 504. Determine the similarity between the embedding representation vector corresponding to the first training data and the embedding representation vector corresponding to the second training data, and use it as the similarity between the target training data pairs.

[0108] 505. Based on the similarity of the target training data pairs, determine the contrastive learning loss function value, and train the first MLLM, the second MLLM, the first embedded head, and the second embedded head using the contrastive learning loss function value.

[0109] When the representation model includes the first embed head, second embed head, and ranking head mentioned above, the training process can be divided into two stages: The first stage primarily trains the first and second embed heads. Assuming the rank head is connected to the first MLLM, the parameters of the first embed head, second MLLM, and second embed head are fixed after the first stage of training. The second stage trains the rank head. At this point, the parameters of the first MLLM trained in the first stage are further adjusted. In other words, the second stage of training only involves the first MLLM and the rank head. The first stage of training uses a contrastive learning approach, i.e., the loss function is the contrastive learning loss function. The second stage of training uses a matching score ranking approach, i.e., the loss function is the ranking loss function.

[0110] It should be noted that the first MLLM and the second MLLM share parameters before training. That is, when training the representation model begins, the weight parameters of each component unit in the two MLLMs are consistent in the initial state. Subsequently, during the training process, the parameters of the two MLLMs change differently. However, in an optional embodiment, the parameters of the visual encoder in the two MLLMs remain frozen, i.e., unchanged, regardless of whether it is the first training phase or the second training phase of the representation model.

[0111] Since the rank head is optional, this embodiment will first introduce the first stage of the training process.

[0112] In the first stage of training, the training of the representation model is actually carried out in batches (or iterations). In one iteration of training, a batch of training data pairs is sampled from the training dataset. The number of training data pairs in each batch can be preset.

[0113] Since the training dataset includes training data pairs corresponding to different types of retrieval tasks, for a given target training batch, the multiple training data pairs contained therein may correspond to the same or different retrieval task types. Let's take any one of these training data pairs as an example to illustrate the processing procedure.

[0114] Regardless of the retrieval task, the target training data pair will include first training data used as retrieval input and second training data used as retrieval output.

[0115] In a unimodal retrieval task, the first training data is data of a certain modality, such as text data 1, and the second training data is another data of the same modality, such as text data 2. In practical applications, text data 1 might be an article, and text data 2 might be the summary of that article.

[0116] In a cross-modal retrieval task, the first training data can be data from one modality, such as text data 1, and the second training data can be data from another modality, such as image data 1. In practical applications, text data 1 might be the text describing the content contained in an image, and image data 1 might be the image itself.

[0117] For the combined modality retrieval task, the first training data can be a combination of data from multiple modalities, such as a combination of text data 1 and image data 1. The second training data can be data from one or more modalities, such as image data 2. In practical applications, image data 1 is an image, text data 1 is text describing a certain editing operation performed on image data 1, and image data 2 is the image obtained after performing this editing operation on image data 1.

[0118] For a target training data pair containing first training data used as retrieval input and second training data used as retrieval output, the first training data can be input into a first MLLM to obtain the embedding representation vector corresponding to the first training data output by the first embedding head, and the second training data can be input into a second MLLM to obtain the embedding representation vector corresponding to the second training data output by the second embedding head. The process of obtaining the embedding representation vectors of these two training data can be referred to the relevant description in the process of performing the target retrieval task through the trained representation model in the foregoing embodiments, and will not be elaborated here.

[0119] Then, the similarity between the embedding representation vector corresponding to the first training data and the embedding representation vector corresponding to the second training data is determined and used as the similarity between the target training data pairs.

[0120] Similar to the process of determining the similarity between target training data pairs, the same process is performed for each training data pair contained in the current target training batch to obtain the similarity between each training data pair.

[0121] Understandably, since each training data pair in the target training batch contains matching data pairs used as retrieval input and output, each training data pair is a positive sample. However, in the target training batch, training data from different training data pairs can constitute negative samples. For example, if a training data pair a (i.e., positive sample a) includes matching text data 1 and image data 1, and another training data pair b (i.e., positive sample b) includes matching text data 2 and image data 2, then text data 1 and image data 2 constitute a negative sample c, and text data 2 and image data 1 constitute another negative sample d.

[0122] Through the above training process, we can obtain the embedding representation vectors corresponding to text data 1, text data 2, image data 1, and image data 2 respectively. The similarity between the embedding representation vectors of text data 1 and image data 1 is taken as the similarity corresponding to positive sample a, the similarity between the embedding representation vectors of text data 2 and image data 2 is taken as the similarity corresponding to positive sample b, the similarity between the embedding representation vectors of text data 1 and image data 2 is taken as the similarity corresponding to negative sample c, and the similarity between the embedding representation vectors of text data 2 and image data 1 is taken as the similarity corresponding to negative sample d.

[0123] Therefore, based on the positive and negative samples that can be formed by the target training batch, a first similarity matrix corresponding to the target training batch can be obtained. Each row of this matrix corresponds to the training data used as retrieval input sequentially in the target training batch, and each column corresponds to the training data used as retrieval output sequentially in the target training batch. The elements in this matrix represent the similarity between the corresponding row and column training data. Thus, the diagonal elements of the first similarity matrix are the similarity between each training data pair (i.e., positive samples) in the target training batch, and the off-diagonal elements are the similarity between training data (i.e., negative samples) in different training data pairs in the target training batch. Therefore, the contrastive learning loss function value can be determined according to the first similarity matrix, and the parameters of the first MLLM, the first embedding head, the second MLLM, and the second embedding head can be adjusted during backpropagation. Through this training process of several training batches, the training of the first-stage representation model is finally completed.

[0124] It should be noted that, optionally, during the training process described above, for the first MLLM and the second MLLM, the parameters of the visual encoder in the first MLLM and the second MLLM can be kept constant, and only the parameters of the mapper and the LLM can be adjusted.

[0125] To more clearly understand the training process of the representation model based on contrastive learning, combined with Figure 6 Let's illustrate with an example. Suppose the current target training batch contains 6 training data pairs, then... Figure 6 As shown, the final first similarity matrix is ​​a 6*6 matrix. The 6 elements on the diagonal of this matrix (represented by black squares) correspond to the similarity calculation results of the 6 training data pairs respectively. The other elements off the diagonal (represented by white squares) are the similarities corresponding to the various negative samples formed by the training data in this training batch.

[0126] exist Figure 6In this study, these six training data pairs are represented as: text data 1 to text data 6, and their corresponding image data 1 to image data 6. Figure 6 The diagram illustrates a cross-modal retrieval task scenario where text data is used to retrieve image data. Thus, text data 1 to text data 6 are sequentially input into the first MLLM, and image data 1 to image data 6 are correspondingly input into the second MLLM.

[0127] based on Figure 6 The first similarity matrix shown in the figure, according to the definition of contrastive learning, aims to maximize the elements on the diagonal and minimize the elements off the diagonal.

[0128] Figure 7 A flowchart of another training method for a representation model provided in an embodiment of the present invention is shown below. Figure 7 As shown, the training method includes the following steps:

[0129] 701. Obtain multiple training data pairs corresponding to the target training batch from the training dataset. Each training data pair includes training data used as retrieval input and training data used as retrieval output.

[0130] 702. Obtain the first negative sample set and the second negative sample set corresponding to the target training batch from the training dataset. The multiple training data in the first negative sample set are negative samples of the multiple training data used as retrieval input in the target training batch. The multiple training data in the second negative sample set are negative samples of the multiple training data used as retrieval output in the target training batch. The first negative sample set and the second negative sample set do not contain the training data in the target training batch.

[0131] 703. Input the multiple training data used as retrieval inputs from the multiple training data pairs of the target training batch into the first MLLM to obtain the embedding representation vectors corresponding to the multiple training data output by the first embed head. Input the multiple training data used as retrieval outputs from the multiple training data pairs of the target training batch into the second MLLM to obtain the embedding representation vectors corresponding to the multiple training data output by the second embed head. The first MLLM and the second MLLM share parameters before training.

[0132] 704. Obtain the embedding representation vectors corresponding to multiple training data in the first negative sample set through the second MLLM and the second embed head, and obtain the embedding representation vectors corresponding to multiple training data in the second negative sample set through the first MLLM and the first embed head.

[0133] 705. Based on the embedding representation vectors corresponding to the multiple training data pairs used as retrieval inputs and the embedding representation vectors corresponding to the multiple training data pairs used as retrieval outputs in the target training batch, determine the first similarity matrix corresponding to the multiple training data pairs in the target training batch.

[0134] 706. Determine the second similarity matrix based on the embedding representation vectors corresponding to the multiple training data used as retrieval inputs in the target training batch and the embedding representation vectors corresponding to the multiple training data in the first negative sample set. Determine the third similarity matrix based on the embedding representation vectors corresponding to the multiple training data used as retrieval outputs in the target training batch and the embedding representation vectors corresponding to the multiple training data in the second negative sample set.

[0135] 707. Based on the first similarity matrix, the second similarity matrix, and the third similarity matrix, determine the contrastive learning loss function value, and train the first MLLM, the second MLLM, the first embedded head, and the second embedded head using the contrastive learning loss function value.

[0136] In the first stage of training the representation model based on contrastive learning provided in this embodiment, the training process for the representation model involves... Figure 5 The contrastive learning method provided in the illustrated embodiment has been optimized, primarily by introducing the concept of "hard negative samples." For the target training batch, Figure 5 In the illustrated embodiment, for a training data pair within the target training batch, training data from other training data pairs within the same batch will be used as negative samples for that training data pair. For example, for a training data pair consisting of text data 1 and image data 1, image data 2 from another training data pair can be used as a negative sample for text data 1. Due to the randomness of sampling training data pairs within the same training batch, "soft negative samples" may frequently occur. A soft negative sample is a sample with low distinguishability from the positive samples. In the example above, image data 1 and image data 2 may have low distinguishability; for instance, image data 1 might contain an image of a cat of a certain breed, while image data 2 might contain an image of the same breed or a different breed of cat. This will affect the performance of the representation model.

[0137] In this embodiment, a hard negative sample refers to a negative sample that has higher discriminative power compared to a positive sample. Using the example above, image data 3 can be resampled from the training dataset as a hard negative sample for text data 1. This image data 3 could be, for example, a landscape painting, unrelated to an image of a cat. This embodiment does not restrict the process of sampling and judging hard negative samples; any method that meets the set hard negative sample conditions is applicable.

[0138] As can be seen from step 702, since each training data pair in the target training batch includes training data used as retrieval input and training data used as retrieval output, corresponding negative samples are sampled from the training dataset for each of these two purposes.

[0139] However, it should be noted that, taking any training data pair in the target training batch as an example, which includes training data 'a' used as retrieval input and training data 'b' used as retrieval output, the data modality of the hard negative sample 'c' corresponding to training data 'a' is consistent with the data modality of training data 'b'. Correspondingly, the data modality of the hard negative sample 'd' corresponding to training data 'b' is consistent with the data modality of training data 'a'. In other words, training data 'a' and hard negative sample 'c' can actually form a negative example training data pair, and training data 'b' and hard negative sample 'd' can actually form a negative example training data pair, which is different from the naturally formed positive example training data pairs in the target training batch. In addition, since training data 'a' is used as retrieval input, its corresponding hard negative sample 'c' will be used as retrieval output, and training data 'b' is used as retrieval output, its corresponding hard negative sample 'd' will be used as retrieval input. Thus, based on the correspondence between the retrieval input / output and the first MLLM and the second MLLM, the corresponding MLLM is input, and the corresponding embedding representation vector is obtained through the corresponding embedding head, as shown in steps 703-704.

[0140] Afterwards, refer to Figure 5 and Figure 6 In the illustrated embodiment, a first similarity matrix can be formed for the embedding representation vectors corresponding to each training data point in the target training batch, which will not be elaborated further here. Furthermore, a second similarity matrix is ​​determined based on the embedding representation vectors corresponding to the multiple training data points used as retrieval inputs in the target training batch and the embedding representation vectors corresponding to the multiple training data points in the first negative sample set. A third similarity matrix is ​​determined based on the embedding representation vectors corresponding to the multiple training data points used as retrieval outputs in the target training batch and the embedding representation vectors corresponding to the multiple training data points in the second negative sample set.

[0141] For ease of understanding, the following is a continuation. Figure 6 In the example, assuming the hard negative samples corresponding to text data 1-6 are image data 7-image data 12, and the hard negative samples corresponding to image data 1-image data 6 are text data 7-text data 12, then the final result can be formed as follows: Figure 8 The second and third similarity matrices are shown in the diagram.

[0142] like Figure 8As shown, the diagonal elements (represented by black squares) in the first similarity matrix correspond to the training data pairs of positive examples in the target training batch. The off-diagonal elements (represented by white squares) in the first similarity matrix, as well as all elements in the second and third similarity matrices, are all training data pairs of positive examples corresponding to the target training batch. Therefore, according to the definition of contrastive learning, the training objective is to maximize the elements represented by black squares and minimize the elements represented by white squares.

[0143] In this embodiment, by introducing hard negative samples to improve contrastive learning, the representation model trained based on the improved contrastive learning loss function can more fully understand the alignment semantics of text-image (visual) and improve its multimodal alignment capability, thereby enabling the retrieval task performed based on the representation model to obtain more accurate retrieval results.

[0144] Figure 9 A flowchart of another training method for a representation model provided in an embodiment of the present invention is shown below. Figure 9 As shown, the training method includes the following steps:

[0145] 901. Obtain third training data and multiple fourth training data from the training dataset, including training data that are used as training data pairs with the third training data.

[0146] 902. Obtain the embedding representation vector corresponding to the third training data through the first MLLM and the first embedding head, and obtain the embedding representation vectors corresponding to multiple fourth training data through the second MLLM and the second embedding head respectively.

[0147] 903. Based on the similarity between the embedding representation vector corresponding to the third training data and the embedding representation vectors corresponding to multiple fourth training data, select a set number of fourth training data from the multiple fourth training data.

[0148] 904. Generate multiple ranked data combinations. Using the first MLLM and rank head, obtain the matching scores corresponding to each of the multiple ranked data combinations. Each ranked data combination includes the third training data and a selected fourth training data.

[0149] 905. Determine the ranking loss function value based on the matching scores corresponding to each of the multiple ranked data combinations, and train the first MLLM and rank head based on the ranking loss function value.

[0150] The foregoing Figures 5-8The examples shown illustrate the first stage of training the representation model. When the representation model includes a ranking head, a second training stage for the ranking head is required after the first stage of training is completed.

[0151] It should be noted that in this embodiment, the first MLLM, the first embedded head, the second MLLM, and the second embedded head used in step 902 are models that have been trained through the first training phase.

[0152] To conduct the second phase of training, the training data for this phase must first be generated. Specifically, third training data used as retrieval input and multiple fourth training data sets used as retrieval data can be obtained from the training dataset. These fourth training data sets will include training data as supervision information; that is, the supervision information training data is actually training data that exists in pairs with the third training data. Here, only one third training data set is used as an example. In reality, the second phase of training requires the collection of a large amount of training data based on this principle.

[0153] Then, using the first MLLM and the first embedding head trained in the first training phase, the embedding representation vectors corresponding to the third training data are extracted. Then, using the second MLLM and the second embedding head, multiple embedding representation vectors corresponding to the fourth training data are obtained respectively.

[0154] Next, based on the similarity between the embedding representation vectors corresponding to the third training data and the embedding representation vectors corresponding to multiple fourth training data, a predetermined number of fourth training data are selected from the multiple fourth training data. For example, the top 10 fourth training data are selected by sorting them from largest to smallest similarity. Then, the third training data and each selected fourth training data form a sorted data combination, thus obtaining multiple (10 in the example above) sorted data combinations.

[0155] Next, the third and fourth training data in each ranked data combination are treated as a multimodal combination and input together into the first MLLM. Optionally, the last token vector output by the LLM in the first MLLM is input into the rank head to obtain the matching score corresponding to the current ranked data combination. This yields multiple matching scores for multiple ranked data combinations. Finally, based on the matching scores of each of the multiple ranked data combinations, the ranking loss function value is determined, and the first MLLM and rank head are trained based on the ranking loss function value. Among the multiple ranked data combinations, the ranked data combination c1 corresponding to the fourth training data (used as supervision information) should ideally have the highest matching score. Optionally, the ranked data combination c2 with the highest actual matching score can be selected, and the ranking loss function value can be determined using c1 and c2 (the definition of this loss function is not specifically limited).

[0156] Understandably, in the second training phase, only the second MLLM, the second embedded head, and the first embedded head are used, but their parameters are not adjusted. Instead, the parameters of the first MLLM and the rank head are adjusted only based on the ranking loss function value.

[0157] The information retrieval apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that these apparatuses can be configured using commercially available hardware components through the steps taught in this solution.

[0158] Figure 10 This is a schematic diagram of the structure of an information retrieval device provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the device includes: a first acquisition module 11, a second acquisition module 12, an embedding processing module 13, and a data retrieval module 14.

[0159] The first acquisition module 11 is used to acquire the representation model trained based on the training dataset. The training dataset corresponds to training data pairs for various retrieval tasks. The training data pairs for different retrieval tasks correspond to different combinations of data modes. The training data pairs include image-text pairs. The text data used to describe the image data in the image-text pairs includes visual words that meet the set density conditions.

[0160] The second acquisition module 12 is used to acquire retrieval data corresponding to the target retrieval task, wherein the target retrieval task is any one of the multiple retrieval tasks, and the retrieval data includes retrieval input data and multiple retrieved data.

[0161] Embedding processing module 13 is used to determine the embedding representation vectors corresponding to the retrieval input data and the plurality of retrieved data respectively through the representation model.

[0162] The data retrieval module 14 is used to filter out target retrieved data from the plurality of retrieved data based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data.

[0163] Optionally, the representation model includes a first multimodal large language model, a second multimodal large language model, a first embedding transformation model connected to the first multimodal large language model, and a second embedding transformation model connected to the second multimodal large language model. Therefore, the embedding processing module 13 is specifically used to: input the retrieval input data into the first multimodal large language model to obtain the embedding representation vector corresponding to the retrieval input data output by the first embedding transformation model; and input the plurality of retrieved data into the second multimodal large language model respectively to obtain the embedding representation vector corresponding to each of the plurality of retrieved data output by the second embedding transformation model.

[0164] Specifically, the embedding processing module 13 is used to: input the retrieval input data into the first multimodal large language model to obtain the last token vector output by the first multimodal large language model; input the last token vector into the first embedding transformation model to obtain the embedding representation vector corresponding to the retrieval input data; input the plurality of retrieved data into the second multimodal large language model respectively to obtain the last token vectors corresponding to the plurality of retrieved data output by the second multimodal large language model respectively; and input the last token vectors corresponding to the plurality of retrieved data respectively into the second embedding transformation model sequentially to obtain the embedding representation vectors corresponding to the plurality of retrieved data respectively.

[0165] Optionally, the representation model further includes a ranking model connected to the first multimodal large language model. Therefore, the data retrieval module 14 is specifically configured to: filter a predetermined number of candidate retrieved data from the plurality of retrieved data based on the similarity between the embedding representation vectors corresponding to the retrieval input data and the embedding representation vectors corresponding to the plurality of retrieved data; generate a plurality of ranked data combinations, wherein each ranked data combination includes the retrieval input data and one candidate retrieved data; input the plurality of ranked data combinations into the first multimodal large language model respectively to obtain the matching scores corresponding to each of the plurality of ranked data combinations output by the ranking model; and filter the target retrieved data from the predetermined number of candidate retrieved data based on the matching scores corresponding to each of the plurality of ranked data combinations.

[0166] Specifically, the data retrieval module 14 is used to: input the multiple sorted data combinations into the first multimodal large language model to obtain the last token vector corresponding to each of the multiple sorted data combinations output by the first multimodal large language model; and input the last token vector corresponding to each of the multiple sorted data combinations into the sorting model in sequence to obtain the matching score corresponding to each of the multiple sorted data combinations.

[0167] Optionally, the apparatus further includes: a training module, configured to obtain target training data pairs from the training dataset, wherein the target training data pairs include first training data used as retrieval input and second training data used as retrieval output, and the target training data pairs are any one of the training data pairs in the training dataset; input the first training data into the first multimodal large language model to obtain the embedding representation vector corresponding to the first training data output by the first embedding conversion model; input the second training data into the second multimodal large language model to obtain the embedding representation vector corresponding to the second training data output by the second embedding conversion model; wherein the first multimodal large language model and the second multimodal large language model share parameters before training; determine the similarity between the embedding representation vector corresponding to the first training data and the embedding representation vector corresponding to the second training data as the similarity corresponding to the target training data pairs; determine the contrastive learning loss function value based on the similarity corresponding to the target training data pairs, and train the first multimodal large language model, the second multimodal large language model, the first embedding conversion model, and the second embedding conversion model through the contrastive learning loss function value.

[0168] Optionally, the training module is further configured to: determine a first similarity matrix corresponding to a target training batch, wherein the target training batch includes the target training data pair and other training data pairs, the diagonal elements of the first similarity matrix are the similarities corresponding to each training data pair in the target training batch, and the off-diagonal elements of the first similarity matrix are the similarities between training data in different training data pairs in the target training batch; and determine the contrastive learning loss function value based on the first similarity matrix.

[0169] Optionally, the training module is further configured to: obtain a first negative sample set and a second negative sample set corresponding to the target training batch, wherein the plurality of training data in the first negative sample set are negative samples of the plurality of training data used as retrieval inputs in the target training batch, and the plurality of training data in the second negative sample set are negative samples of the plurality of training data used as retrieval outputs in the target training batch, and the first negative sample set and the second negative sample set do not contain the training data in the target training batch; obtain the embedding representation vectors corresponding to the plurality of training data in the first negative sample set through the second multimodal large language model and the second embedding transformation model, and obtain the embedding representation vectors through the first... The multimodal large language model and the first embedding transformation model are used to obtain the embedding representation vectors corresponding to multiple training data in the second negative sample set; a second similarity matrix is ​​determined based on the embedding representation vectors corresponding to multiple training data used as retrieval input in the target training batch and the embedding representation vectors corresponding to multiple training data in the first negative sample set; a third similarity matrix is ​​determined based on the embedding representation vectors corresponding to multiple training data used as retrieval output in the target training batch and the embedding representation vectors corresponding to multiple training data in the second negative sample set; and the contrastive learning loss function value is determined based on the first similarity matrix, the second similarity matrix, and the third similarity matrix.

[0170] Optionally, the training module is further configured to: obtain third training data and multiple fourth training data from the training dataset, wherein the multiple fourth training data includes training data that is used as a training data pair with the third training data; obtain the embedding representation vector corresponding to the third training data through the first multimodal large language model and the first embedding transformation model; obtain the embedding representation vector corresponding to the multiple fourth training data respectively through the second multimodal large language model and the second embedding transformation model; select a set number of fourth training data from the multiple fourth training data based on the similarity between the embedding representation vector corresponding to the third training data and the embedding representation vector corresponding to the multiple fourth training data; generate multiple ranked data combinations, wherein each ranked data combination includes the third training data and one selected fourth training data; obtain the matching score corresponding to each of the multiple ranked data combinations through the first multimodal large language model and the ranking model; determine the ranking loss function value based on the matching score corresponding to each of the multiple ranked data combinations, so as to train the first multimodal large language model and the ranking model based on the ranking loss function value.

[0171] Optionally, the device further includes: a training data acquisition module, used to acquire an image and text dataset corresponding to the image and text retrieval task, wherein the image and text retrieval task is included in the multiple retrieval tasks; to annotate the text data in the image and text dataset with visual words of various parts of speech; to filter out the text data in the image and text dataset whose annotated visual words satisfy the set density condition based on the annotation results of the visual words of various parts of speech; and to add the filtered text data and its corresponding image data to the training dataset.

[0172] Figure 10 The device shown can perform the steps in the information retrieval method in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0173] This invention also provides an electronic device, such as... Figure 11 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to implement the information retrieval method as described in the preceding embodiments.

[0174] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the information retrieval method provided in the foregoing embodiments.

[0175] The invention also provides a computer program product, which includes a computer program that, when executed by a processor, enables the processor to at least implement the information retrieval method provided in the foregoing embodiments.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An information retrieval method, characterized in that, include: Obtain a representation model trained on a training dataset, wherein the training dataset corresponds to training data pairs for various retrieval tasks, and the training data pairs for different retrieval tasks correspond to different combinations of data modalities. The training data pairs include image-text pairs, and the text data used to describe image data in the image-text pairs includes visual words that meet a set density condition. Obtain retrieval data corresponding to the target retrieval task, wherein the target retrieval task is any one of the multiple retrieval tasks, and the retrieval data includes retrieval input data and multiple retrieved data; The embedding representation vectors corresponding to the retrieval input data and the plurality of retrieved data are determined through the representation model; Based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data, the target retrieved data is selected from the plurality of retrieved data.

2. The method according to claim 1, characterized in that, The representation model includes a first multimodal large language model, a second multimodal large language model, a first embedding transformation model connected to the first multimodal large language model, and a second embedding transformation model connected to the second multimodal large language model; The step of determining the embedding representation vectors corresponding to the retrieval input data and the plurality of retrieved data through the representation model includes: The retrieval input data is input into the first multimodal large language model to obtain the embedding representation vector corresponding to the retrieval input data output by the first embedding transformation model; The multiple retrieved data are respectively input into the second multimodal large language model to obtain the embedding representation vectors corresponding to each of the multiple retrieved data output by the second embedding transformation model.

3. The method according to claim 2, characterized in that, The step of inputting the retrieval input data into the first multimodal large language model to obtain the embedding representation vector corresponding to the retrieval input data output by the first embedding transformation model includes: The retrieval input data is input into the first multimodal large language model to obtain the last token vector output by the first multimodal large language model. The last token vector is then input into the first embedding transformation model to obtain the embedding representation vector corresponding to the retrieval input data. The step of inputting the plurality of retrieved data into the second multimodal large language model to obtain the embedding representation vectors corresponding to each of the plurality of retrieved data output by the second embedding transformation model includes: The multiple retrieved data are respectively input into the second multimodal large language model to obtain the last token vector corresponding to each of the multiple retrieved data output by the second multimodal large language model. The last token vector corresponding to each of the multiple retrieved data is then sequentially input into the second embedding transformation model to obtain the embedding representation vector corresponding to each of the multiple retrieved data.

4. The method according to claim 2, characterized in that, The representation model also includes a ranking model connected to the first multimodal large language model; The step of filtering target retrieved data from the plurality of retrieved data based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data includes: Based on the similarity between the embedding representation vector corresponding to the retrieval input data and the embedding representation vector corresponding to each of the plurality of retrieved data, a set number of candidate retrieved data are selected from the plurality of retrieved data. Multiple sorted data combinations are generated, wherein each sorted data combination includes the retrieval input data and one candidate retrieval data; The multiple sorted data combinations are respectively input into the first multimodal large language model to obtain the matching scores corresponding to each of the multiple sorted data combinations output by the sorting model; Based on the matching scores corresponding to the multiple sorted data combinations, the target data to be retrieved is selected from the set number of candidate retrieval data.

5. The method according to claim 4, characterized in that, The step of inputting the multiple ranked data combinations into the first multimodal large language model to obtain the matching scores corresponding to each of the multiple ranked data combinations output by the ranking model includes: The multiple sorted data combinations are respectively input into the first multimodal large language model to obtain the last token vector corresponding to each of the multiple sorted data combinations output by the first multimodal large language model. The last token vectors corresponding to each of the multiple sorted data combinations are sequentially input into the sorting model to obtain the matching score corresponding to each of the multiple sorted data combinations.

6. The method according to claim 2, characterized in that, The training process of the representation model includes: Obtain target training data pairs from the training dataset, wherein the target training data pairs include first training data used as retrieval input and second training data used as retrieval output, and the target training data pairs are any training data pairs in the training dataset; The first training data is input into the first multimodal large language model to obtain the embedding representation vector corresponding to the first training data output by the first embedding conversion model. The second training data is input into the second multimodal large language model to obtain the embedding representation vector corresponding to the second training data output by the second embedding conversion model; wherein the first multimodal large language model and the second multimodal large language model share parameters before training; The similarity between the embedding representation vector corresponding to the first training data and the embedding representation vector corresponding to the second training data is determined as the similarity of the target training data pair. Based on the similarity of the target training data pairs, a contrastive learning loss function value is determined, and the first multimodal large language model, the second multimodal large language model, the first embedding transformation model, and the second embedding transformation model are trained using the contrastive learning loss function value.

7. The method according to claim 6, characterized in that, The step of determining the contrastive learning loss function value based on the similarity of the target training data pairs includes: A first similarity matrix is ​​determined corresponding to the target training batch, wherein the target training batch includes the target training data pair and other training data pairs. The diagonal elements of the first similarity matrix are the similarity scores of each training data pair in the target training batch, and the off-diagonal elements of the first similarity matrix are the similarity scores between training data in different training data pairs in the target training batch. Based on the first similarity matrix, determine the value of the contrastive learning loss function.

8. The method according to claim 7, characterized in that, The method further includes: Obtain a first negative sample set and a second negative sample set corresponding to the target training batch, wherein the multiple training data in the first negative sample set are negative samples of multiple training data used as retrieval input in the target training batch, and the multiple training data in the second negative sample set are negative samples of multiple training data used as retrieval output in the target training batch, and the first negative sample set and the second negative sample set do not contain the training data in the target training batch. By using the second multimodal large language model and the second embedding transformation model, embedding representation vectors corresponding to multiple training data in the first negative sample set are obtained; and by using the first multimodal large language model and the first embedding transformation model, embedding representation vectors corresponding to multiple training data in the second negative sample set are obtained. The second similarity matrix is ​​determined based on the embedding representation vectors corresponding to multiple training data used as retrieval inputs in the target training batch and the embedding representation vectors corresponding to multiple training data in the first negative sample set; The third similarity matrix is ​​determined based on the embedding representation vectors corresponding to multiple training data used as retrieval output in the target training batch and the embedding representation vectors corresponding to multiple training data in the second negative sample set; The contrastive learning loss function value is determined based on the first similarity matrix, the second similarity matrix, and the third similarity matrix.

9. The method according to any one of claims 6-8, characterized in that, The training process of the representation model also includes: Obtain third training data and multiple fourth training data from the training dataset, wherein the multiple fourth training data include training data that is used as a training data pair with the third training data; The embedding representation vector corresponding to the third training data is obtained by using the first multimodal large language model and the first embedding transformation model; The embedding representation vectors corresponding to the multiple fourth training data are obtained by using the second multimodal large language model and the second embedding transformation model, respectively. Based on the similarity between the embedding representation vector corresponding to the third training data and the embedding representation vector corresponding to the plurality of fourth training data, a set number of fourth training data are selected from the plurality of fourth training data; Multiple sorted data combinations are generated, wherein each sorted data combination includes the third training data and a selected fourth training data; Using the first multimodal large language model and the ranking model, obtain the matching scores corresponding to each of the multiple ranked data combinations; Based on the matching scores corresponding to each of the multiple sorted data combinations, a sorting loss function value is determined, and the first multimodal large language model and the sorting model are trained based on the sorting loss function value.

10. The method according to claim 1, characterized in that, The method further includes: Obtain the image and text dataset corresponding to the image and text retrieval task, wherein the image and text retrieval task is included in the various retrieval tasks; For the text data in the aforementioned image and text dataset, visual words with multiple parts of speech are labeled. Based on the annotation results of visual words with various parts of speech, text data in the image and text dataset that have been annotated with visual words that meet the set density conditions are selected; The selected text data and its corresponding image data are added to the training dataset.

11. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the information retrieval method as described in any one of claims 1 to 10.

12. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the information retrieval method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the information retrieval method as described in any one of claims 1 to 10.

Images