Training Method of Multimodal Retrieval Model, Multimodal Retrieval Method and Device

Abstract

The present application discloses a training method for a multimodal retrieval model, a multimodal retrieval method, and a device, relating to the field of multimodal retrieval technology. The method comprises constructing a vector database; constructing a hard negative sample training set based on the vector database; the hard negative sample training set comprising: multiple query texts and multiple hard negative sample document images corresponding to the query texts; and inputting the hard negative sample training set into an initial multimodal retrieval model for training to obtain a target multimodal retrieval model. Because the vector database stores multi-vector representations of multiple documents, negative samples (i.e., hard negative samples) that are similar but unrelated to the query vector can be found through the vector database. Mining these samples through the vector database to construct a hard negative sample training set allows the model to learn more subtle distinguishing features, thereby improving the model's ability to distinguish between positive and negative samples and, during the retrieval process, enhancing retrieval accuracy.

Description

Technical Field

[0001] The present application relates to the field of multimodal retrieval technology, and in particular to a training method for a multimodal retrieval model, a multimodal retrieval method, and a multimodal retrieval device. Background Art

[0002] Document retrieval is used to match user queries with relevant documents in a given corpus by similarity, either as a ranking system in a search engine or as part of more complex information extraction or retrieval-enhanced generation applications.

[0003] Related technologies use a VLM-based multimodal retrieval model (VLM2Vec, Training Vision-Language Models for Massive Multimodal Embedding Tasks) to extract document text content for multimodal retrieval tasks. However, when processing PDFs, VLM2Vec relies on external tools (such as OCR or PDF parsing libraries) to extract text content. These tools can suffer from inaccurate text extraction when working with complex layouts, scanned documents, or low-quality documents, leading to biased retrieval results.

[0004] Therefore, how to improve the multimodal retrieval accuracy of documents is an urgent problem that needs to be solved. Summary of the Invention

[0005] The present application provides a multimodal retrieval model training method, a multimodal retrieval method and a device to at least solve the problem of how to improve the multimodal retrieval accuracy of documents in the related art.

[0006] In a first aspect, the present application provides a method for training a multimodal retrieval model, comprising:

[0007] Constructing a vector database; the vector database includes multi-vector representations of multiple document images;

[0008] Based on the vector database, a hard negative sample training set is constructed; the hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts;

[0009] The hard negative sample training set is input into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0010] In a second aspect, the present application provides a multimodal retrieval method, comprising:

[0011] Obtaining the data to be retrieved input by the user; the data to be retrieved includes: target query text and target document image;

[0012] The data to be retrieved is input into the target multimodal retrieval model to obtain a retrieval result corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in the first aspect.

[0013] In a third aspect, the present application further provides a training device for a multimodal retrieval model, comprising:

[0014] A construction module for constructing a vector database; the vector database includes multi-vector representations of multiple document images;

[0015] A screening module is used to construct a hard negative sample training set based on the vector database; the hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts;

[0016] The training module is used to input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0017] In a fourth aspect, the present application further provides a multimodal retrieval device, comprising:

[0018] An input module is used to obtain the data to be retrieved input by the user; the data to be retrieved includes: target query text and target document image;

[0019] A retrieval module is used to input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in the first aspect.

[0020] In a fifth aspect, the present application also provides an electronic device comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned multimodal retrieval model training methods or any of the above-mentioned multimodal retrieval methods when executing the computer program.

[0021] In a sixth aspect, the present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the training method of any of the above-mentioned multimodal retrieval models or the steps of any of the above-mentioned multimodal retrieval methods are implemented.

[0022] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned multimodal retrieval model training methods or any of the above-mentioned multimodal retrieval methods.

[0023] Through this application, a vector database is constructed, and based on the vector database, a hard negative sample training set is constructed, wherein the hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts; the hard negative sample training set is input into the initial multimodal retrieval model for training to obtain the target multimodal retrieval model. Since the vector database stores multi-vector representations of multiple documents, these multi-vector representations capture the semantic information of the document images. Through the vector database, negative samples (i.e., hard negative samples) that are similar but unrelated to the query vector can be found. That is, hard negative samples refer to document image samples that are similar to the query text in the feature space but are actually unrelated to the query text. By mining these samples through the vector database and constructing a hard negative sample training set, the model can learn more subtle distinguishing features, thereby improving the model's ability to distinguish between positive and negative samples, and improving the accuracy of the retrieval during the retrieval process. At the same time, matching the query (to be retrieved) vector through the vector database helps to quickly locate and retrieve the vector representation with the highest relevance to the query, thereby improving retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A flowchart of a method for training a multimodal retrieval model provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a multimodal search method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the structure of a multimodal retrieval model training device provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of a multimodal retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0032] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] Explanation of terms:

[0034] Multimodal retrieval is a technology that uses multiple data modalities (such as text, images, video, audio, etc.) to retrieve information. It integrates data in different forms to provide more comprehensive, accurate and rich retrieval results to meet users' diverse query needs.

[0035] In some embodiments, as Figure 1 As shown, a training method for a multimodal retrieval model is provided, comprising the following steps S11-S13:

[0036] S11. Build a vector database.

[0037] Among them, the vector database includes multi-vector representations of multiple document images.

[0038] Specifically, a vector database is constructed by multi-vector representation of multiple document images. The multi-vector representation of multiple document images is obtained by training an initial multimodal retrieval model with an initial training sample set.

[0039] In some embodiments, before executing the above step S11 (building a vector database), the following steps are further executed:

[0040] (1) Obtain the initial training sample set.

[0041] The initial training sample set includes multiple query texts, positive sample document images corresponding to the query texts, and negative sample document images. Positive sample document images are document images associated with the query texts, while negative sample document images are document images not associated with the query texts. Each positive and negative sample document image is labeled.

[0042] Specifically, a data set containing document images and related query texts is obtained as an initial training sample set.

[0043] (2) Inputting the initial training sample set into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images.

[0044] The initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on the initial training sample set. This article describes the model architecture, which is used to train the multimodal retrieval model. The preset multimodal retrieval model includes a visual language model (VLM), a large language model (LLM), and a mapping layer.

[0045] Specifically, the initial training sample set is input into the preset multimodal retrieval model, the visual encoder of the visual language model is used to extract document visual features from the document image, the large language model is used to extract query features from the query text, and the document visual features and query features are vectorized and encoded respectively using the mapping layer to generate multi-vector representations of the query text and multi-vector representations of multiple document images.

[0046] Optionally, the initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on the initial training sample set, specifically comprising the following steps:

[0047] Randomly sampling based on the initial training sample set to obtain an in-batch negative sample training set;

[0048] The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

[0049] Specifically, a certain number of samples are randomly selected from the set of non-positive samples in the initial training sample set as negative samples. These negative samples, along with the positive samples for each query, are then input into the pre-set multimodal retrieval model as the in-batch negative sample training set for training, thereby obtaining the initial multimodal retrieval model. For example, a stratified random sampling method can be used to obtain the in-batch negative sample training set. This ensures that each negative class is appropriately represented in the negative samples, which helps the model learn the characteristics of negative samples from different categories.

[0050] In some embodiments, the above step S11 (building a vector database) can be implemented as follows:

[0051] 1) Segment multiple document images using a preset window size and preset resolution to obtain multiple document image data.

[0052] The preset window size and the preset resolution may be set according to actual application scenarios. For example, the preset window size may be 256×256 pixels, and the preset resolution may be 1024×1024 pixels.

[0053] Specifically, since each document includes at least one document image, each document image is segmented according to a preset window size and resolution to obtain multiple sliced images, i.e., multiple document image data. Segmenting the document image according to the predetermined window size and resolution helps extract more fine-grained information from the document, improving retrieval accuracy.

[0054] 2) Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector.

[0055] Specifically, visual feature encoding is the process of converting visual information in an image or video into low-dimensional, structured feature vectors. These feature vectors can represent information such as objects, scenes, textures, and spatial relationships within the image. The segmented document image data is fed into the VLM's visual encoder for visual feature encoding, generating visual feature vectors.

[0056] 3) Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector.

[0057] Specifically, the visual feature vector is input into the mapping layer of the initial multimodal retrieval model for embedding encoding to obtain a visual embedding vector. The mapping layer converts the visual feature vector into an embedding vector with the same dimension as the query text features, facilitating subsequent multimodal fusion. For example, a linear layer maps the visual feature vector to the target embedding dimension to obtain the visual embedding vector.

[0058] 4) Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features.

[0059] Specifically, the visual embedding vector is input into the large language model of the initial multimodal retrieval model. For example, the large language model executes the instruction: "Briefly describe the image information" and generates semantic features.

[0060] 5) Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document.

[0061] Specifically, the semantic features are input into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document.

[0062] 6) Storing the multi-vector representation of each document in a vector database.

[0063] Specifically, the multi-vector representation of each document is stored in a vector database. It should be noted that if new document images are added to the dataset in the vector database, the vector representations of these new images must be recalculated and added to the vector database. If certain document images in the dataset are deleted or modified, the corresponding vectors must also be deleted from the database or updated. If the dataset or model has not changed and the performance of the vector database meets current requirements, there is no need to re-acquire the vector database.

[0064] S12. Construct a hard negative sample training set based on the vector database.

[0065] The hard negative training set includes multiple query texts and multiple hard negative document images corresponding to the query texts. Hard negative document images are document image samples that are similar to the query text in feature space but are actually irrelevant to the query text.

[0066] In some embodiments, the above step S12 (building a hard negative sample training set based on the vector database) can be implemented as follows:

[0067] a. Determine N multi-vector representations having the highest similarity to the multi-vector representations of each query text from the vector database to obtain a first candidate sample set.

[0068] The first candidate sample set includes each query text and a candidate sample set for each query text. N is an integer greater than 0.

[0069] Optionally, the above step a (determining N multi-vector representations having the highest similarity to the multi-vector representations of each query text from the vector database to obtain the first candidate sample set) can be implemented as follows:

[0070] Calculating the delayed interaction similarity between each query text and each document in the vector database;

[0071] The delayed interaction similarities are used for sorting, and N multi-vector representations having the highest similarity to the multi-vector representations of the query texts are determined from the vector database to obtain a first candidate sample set.

[0072] Specifically, the delayed interaction similarity between each query text and all documents in the vector database is calculated. The documents are sorted in descending order based on the delayed interaction similarity scores, and the N documents with the highest delayed interaction similarity are selected from the sorted results as the first candidate sample set. Since the core concept of delayed interaction similarity is to maintain independence between the query and document encoding processes, document encoding can be completed offline, while the computational complexity of the query phase is low. Therefore, it is suitable for processing large datasets, such as in document retrieval scenarios.

[0073] Furthermore, optionally, based on the delayed interaction similarity calculation formula, the delayed interaction similarity between each query text and each document in the vector database is calculated:

[0074] .

[0075] in, Represents the query text and documentation The similarity score between represents the multi-vector representation of the query vector, A multi-vector representation representing the document vector, and are the number of query vectors and document vectors respectively. represents the i-th query vector, represents the j-th document vector, express and The dot product of two vectors.

[0076] For each vector in the query , find the document vector with the largest dot product . Calculate the values of these maximum dot products, take their maximum value, and sum all these maximum dot product values to get the final similarity score Specifically, the model encodes the query and document separately, generating their own vector representations, and then evaluates the correlation between the query multi-vector representation and the document multi-vector representation using a maximum similarity function. This approach efficiently calculates the similarity between the query vector and the vectors in the vector database, thereby finding the most relevant documents. This approach maintains independence in the query and document encoding process, while performing fine-grained interaction when calculating similarity, thereby reducing computational complexity.

[0077] b. Eliminate the target candidate sample set from the first candidate sample set to obtain a second candidate sample set.

[0078] Among them, the target candidate sample set does not include positive samples of the target query text.

[0079] c. Based on the second candidate sample set, construct a hard negative sample training set.

[0080] Specifically, candidate samples are first retrieved. For each query text, the top 100 candidate samples with the highest similarity scores are retrieved as the first candidate sample set. Invalid samples are then removed, i.e., candidate sample sets and corresponding queries that do not contain positive samples from the first candidate sample set are removed, as these samples are not helpful for model training. This results in a second candidate sample set. Finally, hard negative samples are selected. Negative samples that are similar but unrelated to each query text are selected from the second candidate sample set as the hard negative sample set. These samples help the model better distinguish between similar but unrelated samples.

[0081] S13: Input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0082] In some embodiments, the above step S13 (inputting the hard negative sample training set into the initial multimodal retrieval model for training to obtain the target multimodal retrieval model) can be implemented as follows:

[0083] Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text;

[0084] Determine a preset loss function based on the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text;

[0085] The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

[0086] Specifically, multiple query texts are input into the initial multimodal retrieval model to obtain the document retrieval results corresponding to each query. At this stage, the initial multimodal retrieval model will generate a list of documents related to the query text, and attach the multimodal information of the document (such as pictures). That is, multiple query texts are input into the initial multimodal retrieval model to obtain the document retrieval pictures corresponding to each query text.

[0087] For each query text, in addition to the retrieved positive samples (relevant documents), it is also necessary to introduce hard negative samples (irrelevant but easily confused documents). Based on these positive and negative sample pairs, a preset loss function is determined. That is, the preset loss function is determined based on the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text. The preset loss function is used to make similar samples closer in the feature space and dissimilar samples farther apart. That is, the goal of the preset loss function is to make the similarity between the query text and the positive sample higher than the similarity between the query text and the negative sample.

[0088] By minimizing a preset loss function and adjusting the weight parameters of the initial multimodal retrieval model, the initial multimodal retrieval model learns from the aforementioned training data to optimize the similarity calculation between query text and documents, thereby improving retrieval accuracy and discrimination capabilities. After optimized training, the resulting target multimodal retrieval model can more accurately process queries and distinguish between relevant and irrelevant documents, thereby providing higher-quality retrieval results in actual retrieval.

[0089] Optionally, during the model training process, a preset loss function is used to train the initial multimodal retrieval model, wherein the preset loss function formula is as follows:

[0090] .

[0091] in, represents the loss value, represents the batch size, that is, the number of samples used in each training iteration, represents the sum of all samples in the batch, Indicates the The query multi-vector and the Similarity scores between multiple vectors of positive sample documents; Indicates the The query multi-vector and the The similarity score between multiple vectors of negative sample documents. The function is a smooth activation function that ensures that the gradient of the loss function is continuous and non-negative.

[0092] This preset contrastive loss function works by optimizing the initial multimodal retrieval model by maximizing the similarity score of positive sample pairs and minimizing the similarity score of negative sample pairs. In this way, the model is trained to bring similar samples closer together and push dissimilar samples further apart, thereby improving the discriminative power of feature representation and further improving retrieval accuracy.

[0093] The present disclosure provides a training method for a multimodal retrieval model. Since the vector database stores multi-vector representations of multiple documents, these multi-vector representations capture the semantic information of the document images. The vector database can be used to find negative samples (i.e., hard negative samples) that are similar to but not related to the query vector. That is, hard negative samples refer to document image samples that are similar to the query text in the feature space but are actually irrelevant to the query text. By mining these samples through the vector database and constructing a hard negative sample training set, the model can learn more subtle distinguishing features, thereby improving the model's ability to distinguish between positive and negative samples, and improving the accuracy of retrieval during the retrieval process. At the same time, matching the query (to be retrieved) vector through the vector database helps to quickly locate and retrieve the vector representation with the highest relevance to the query, thereby improving retrieval efficiency.

[0094] Figure 2 is a flow chart of a multimodal retrieval method provided by an embodiment of the present disclosure, such as Figure 2 As shown, a multimodal retrieval method provided by an embodiment of the present disclosure includes:

[0095] S21. Obtain the data to be retrieved input by the user.

[0096] The data to be retrieved includes: target query text and target document image.

[0097] Specifically, the data to be retrieved input by the user is obtained, and the data to be retrieved includes the target query text and the target document image.

[0098] S22: Input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved.

[0099] The target multimodal retrieval model is obtained based on any one of the multimodal retrieval model training methods provided in the above method embodiments.

[0100] In some embodiments, the above step S22 (inputting the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved) can be implemented as follows:

[0101] A. Based on the target multimodal retrieval model, perform word segmentation and vectorization encoding on the target query text to obtain a multi-vector representation of the target query text.

[0102] Specifically, based on the target multimodal retrieval model, the user's target query text is preprocessed, including word segmentation, stop word removal, stemming, etc., to convert it into a format that the model can understand.

[0103] Optionally, the above step A (performing word segmentation and vectorization encoding on the target query text based on the target multimodal retrieval model to obtain a multi-vector representation of the target query text) can be implemented as follows:

[0104] Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words;

[0105] Encoding the segmented words to obtain multiple word codes;

[0106] Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word;

[0107] The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

[0108] Specifically, the preprocessed target query text is input into the language model (LLM) part of the model, and the text features are converted into vector representations through the model's mapping layer.

[0109] B. Determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved, and output them.

[0110] Wherein, M is an integer greater than 0.

[0111] Specifically, a similarity search is performed against a database of document images using a query vector. A similarity score is calculated between the query vector and the document image vectors stored in the database. The results are sorted by similarity score, typically selecting the top N results with the highest scores as candidate matches.

[0112] Furthermore, the retrieved candidate matches are post-processed, which may include removing duplicates, re-ranking, etc. If needed, the retrieved document images can be further analyzed, such as extracting key information or generating summaries. The retrieval results are presented to the user, typically including a thumbnail of the document image, a title, a preview of some of the contents, etc. A user interaction interface is provided, allowing the user to view detailed information, obtain the complete document, etc. In some systems, users can also provide feedback on the retrieval results, such as marking them as relevant or irrelevant. User feedback can be used to further optimize the model and vector database. Based on user feedback and new data, the model and vector database are regularly updated to improve retrieval performance and accuracy.

[0113] Through these steps, the trained multimodal retrieval model can effectively process user queries and retrieve the most relevant results from a large number of document images.

[0114] The multimodal retrieval method provided herein uses a vector database to store multi-vector representations of multiple documents. These multi-vector representations capture the semantic information of document images. Matching query (to-be-retrieved) vectors against the vector database facilitates rapid location and retrieval of the vector representation most relevant to the query, thereby improving retrieval efficiency.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0116] In some embodiments, reference Figure 3As shown, an embodiment of the present application further provides a multimodal retrieval model training device 300, comprising:

[0117] A construction module 310 is used to construct a vector database; the vector database includes multi-vector representations of multiple document images;

[0118] A screening module 320 is configured to construct a hard negative sample training set based on the vector database; the hard negative sample training set includes: a plurality of query texts and a plurality of hard negative sample document images corresponding to the query texts;

[0119] The training module 330 is configured to input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0120] As an optional implementation of the embodiment of the present disclosure, the device further includes:

[0121] An acquisition module is used to acquire an initial training sample set; the initial training sample set includes: multiple query texts, multiple positive sample document images of the query texts, and negative sample document images;

[0122] A generation module is used to input the initial training sample set into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images; the initial multimodal retrieval model is obtained by training the preset multimodal retrieval model based on the initial training sample set.

[0123] As an optional implementation of the embodiment of the present disclosure, the construction module 310 is specifically configured to:

[0124] Segment multiple document images using a preset window size and a preset resolution to obtain multiple document image data;

[0125] Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector;

[0126] Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector;

[0127] Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features;

[0128] Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document;

[0129] The multi-vector representation of each document is stored in a vector database.

[0130] As an optional implementation of the embodiment of the present disclosure, the screening module 320 includes:

[0131] a matching unit, configured to determine, from the vector database, N multi-vector representations having the highest similarity to the multi-vector representations of each query text, and obtain a first candidate sample set; the first candidate sample set includes each query text and a candidate sample set for each query text; wherein N is an integer greater than 0;

[0132] a culling unit, configured to cull a target candidate sample set from the first candidate sample set to obtain a second candidate sample set; the target candidate sample set does not include positive samples of the target query text;

[0133] A construction unit is used to construct a hard negative sample training set based on the second candidate sample set.

[0134] As an optional implementation of the embodiment of the present disclosure, the matching unit is specifically configured to:

[0135] Calculating the delayed interaction similarity between each query text and each document in the vector database;

[0136] The delayed interaction similarities are used for sorting, and N multi-vector representations with the highest similarity to the multi-vector representations of each query text are determined from the vector database to obtain a first candidate sample set; wherein N is an integer greater than 0.

[0137] As an optional implementation of the embodiment of the present disclosure, the training module 330 is specifically configured to:

[0138] Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text;

[0139] Determine a preset loss function based on the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text;

[0140] The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

[0141] As an optional implementation of the embodiment of the present disclosure, the initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on an initial training sample set, including:

[0142] Randomly sampling based on the initial training sample set to obtain an in-batch negative sample training set;

[0143] The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

[0144] For the description of the features in the embodiment corresponding to the training device of the multimodal retrieval model, please refer to the relevant description of the embodiment corresponding to the training method of the multimodal retrieval model, and will not be repeated here.

[0145] In some embodiments, reference Figure 4 As shown, an embodiment of the present disclosure provides a multimodal search device 400, which includes:

[0146] The input module 410 is used to obtain the data to be retrieved input by the user; the data to be retrieved includes: target query text and target document image;

[0147] The retrieval module 420 is used to input the data to be retrieved into the target multimodal retrieval model to obtain the retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on any embodiment of the multimodal retrieval model training method provided in the above method embodiment.

[0148] As an optional implementation of the embodiment of the present disclosure, the retrieval module 420 includes:

[0149] A processing unit, configured to perform word segmentation and vectorization encoding on the target query text based on the target multimodal retrieval model to obtain a multi-vector representation of the target query text;

[0150] A retrieval unit is used to determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved and output them; wherein M is an integer greater than 0.

[0151] As an optional implementation of the embodiment of the present disclosure, the processing unit is specifically configured to:

[0152] Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words;

[0153] Encoding the segmented words to obtain multiple word codes;

[0154] Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word;

[0155] The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

[0156] For the description of the features in the embodiment corresponding to the multimodal retrieval device, please refer to the relevant description of the embodiment corresponding to the multimodal retrieval method, and will not be repeated here.

[0157] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned multimodal retrieval model training method embodiments.

[0158] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned multimodal retrieval model training method embodiments or the steps of any of the above-mentioned multimodal retrieval methods when run.

[0159] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0160] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned multimodal retrieval model training method embodiments or the steps in any of the above-mentioned multimodal retrieval methods.

[0161] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned multimodal retrieval model training method embodiments or any of the above-mentioned multimodal retrieval methods.

[0162] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0163] The above is a detailed introduction to a training method for a multimodal retrieval model and a multimodal retrieval method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A training method for a multimodal retrieval model, characterized in that: include: Build vector database; The vector database includes multiple vector representations of multiple document images; Based on the vector database, construct a hard negative sample training set; The hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts; Inputting the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model; The step of constructing a hard negative sample training set based on the vector database includes: Based on the delayed interaction similarity calculation formula, the delayed interaction similarity between each query text and each document in the vector database is calculated: in, represents the similarity score between the query text q and the document d, represents the multi-vector representation of the query vector, A multi-vector representation representing the document vector, and are the number of query vectors and document vectors respectively, represents the i-th query vector, represents the j-th document vector, express and The dot product of two vectors; Sorting by using the delayed interaction similarities, determining N multi-vector representations with the highest similarity to the multi-vector representations of each query text from the vector database, and obtaining a first candidate sample set; the first candidate sample set includes each query text and a candidate sample set for each query text; wherein N is an integer greater than 0; Eliminate the target candidate sample set from the first candidate sample set to obtain a second candidate sample set; the target candidate sample set does not include positive samples of the target query text; Based on the second candidate sample set, a hard negative sample training set is constructed.

2. The method for training a multimodal retrieval model according to claim 1, wherein: Before building the vector database, the method further includes: Obtaining an initial training sample set; the initial training sample set includes: multiple query texts, multiple positive sample document images of the query texts, and negative sample document images; The initial training sample set is input into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images; the initial multimodal retrieval model is obtained by training the preset multimodal retrieval model based on the initial training sample set.

3. The method for training a multimodal retrieval model according to claim 1, wherein: The constructing of the vector database includes: Segment multiple document images using a preset window size and a preset resolution to obtain multiple document image data; Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector; Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector; Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features; Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document; The multi-vector representation of each document is stored in a vector database.

4. The method for training a multimodal retrieval model according to claim 1, wherein: Inputting the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model includes: Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text; Determine a preset loss function based on the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text; The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

5. The method for training a multimodal retrieval model according to claim 2, wherein: The initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on an initial training sample set, including: Randomly sampling based on the initial training sample set to obtain an in-batch negative sample training set; The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

6. A multimodal retrieval method, characterized in that: include: Get the data to be retrieved entered by the user; The data to be retrieved includes: target query text and target document image; The data to be retrieved is input into the target multimodal retrieval model to obtain a retrieval result corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in any one of claims 1-5.

7. The multimodal retrieval method according to claim 6, characterized in that: Inputting the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved includes: Based on the target multimodal retrieval model, the target query text is segmented and vectorized to obtain a multi-vector representation of the target query text; Determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved and output them; wherein M is an integer greater than 0.

8. The multimodal retrieval method according to claim 7, characterized in that: The step of performing word segmentation and vectorization encoding on the target query text based on the target multimodal retrieval model to obtain a multi-vector representation of the target query text includes: Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words; Encoding the segmented words to obtain multiple word codes; Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word; The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

9. A training device for a multimodal retrieval model, characterized in that: include: Construction module, used to build vector database; The vector database includes multiple vector representations of multiple document images; A screening module, configured to construct a hard negative sample training set based on the vector database; The hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts; A training module, configured to input the hard negative sample training set into an initial multimodal retrieval model for training to obtain a target multimodal retrieval model; The screening module is specifically used for: Based on the delayed interaction similarity calculation formula, the delayed interaction similarity between each query text and each document in the vector database is calculated: in, represents the similarity score between the query text q and the document d, represents the multi-vector representation of the query vector, A multi-vector representation representing the document vector, and are the number of query vectors and document vectors respectively, represents the i-th query vector, represents the j-th document vector, express and The dot product of two vectors; Sorting by using the delayed interaction similarities, determining N multi-vector representations with the highest similarity to the multi-vector representations of each query text from the vector database, and obtaining a first candidate sample set; the first candidate sample set includes each query text and a candidate sample set for each query text; wherein N is an integer greater than 0; Eliminate the target candidate sample set from the first candidate sample set to obtain a second candidate sample set; the target candidate sample set does not include positive samples of the target query text; Based on the second candidate sample set, a hard negative sample training set is constructed.

10. A multimodal search device, characterized in that: include: Input module, used to obtain the data to be retrieved input by the user; The data to be retrieved includes: target query text and target document image; A retrieval module, used to input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in any one of claims 1-5.

11. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the training method for a multimodal retrieval model according to any one of claims 1 to 5, or the multimodal retrieval method according to any one of claims 6 to 8, when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein, when the computer program is executed by a processor, it implements the training method of the multimodal retrieval model according to any one of claims 1 to 5, or the multimodal retrieval method according to any one of claims 6 to 8.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the multimodal retrieval model according to any one of claims 1 to 5, or the multimodal retrieval method according to any one of claims 6 to 8.

Images