Training method of multi-modal retrieval model and multi-modal retrieval method and device

Abstract

The invention discloses a multi-modal retrieval model training method and a multi-modal retrieval method and device, and relates to the technical field of multi-modal retrieval, and the method comprises the following steps: constructing a vector database; based on the vector database, constructing a difficult-to-negative sample training set; the difficult negative sample training set comprises a plurality of query texts and difficult negative sample document pictures corresponding to the query texts; and inputting the difficult-to-negative sample training set into the initial multi-modal retrieval model for training to obtain a target multi-modal retrieval model. Due to the fact that the vector database stores multi-vector representation of multiple documents, negative samples (namely, difficult-to-negative samples) similar to but not related to query vectors can be found through the vector database, the samples are mined through the vector database, a difficult-to-negative sample training set is constructed, the model can learn more subtle distinguishing features, and the classification accuracy is improved. Therefore, the positive and negative sample distinguishing capability of the model is improved, and the retrieval accuracy is improved in the retrieval process.

Description

Technical Field

[0001] The present application relates to the field of multimodal retrieval technology, and in particular to a training method for a multimodal retrieval model, a multimodal retrieval method and a multimodal retrieval device. Background Art

[0002] Document retrieval is used to match user queries with relevant documents in a given corpus by similarity, either as a ranking system in a search engine or as part of more complex information extraction or retrieval-enhanced generation applications.

[0003] In related technologies, for multimodal retrieval tasks, a VLM-based multimodal retrieval model (VLM2Vec, Training Vision-Language Models for Massive Multimodal Embedding Tasks) is used to extract the text content of documents. However, when processing PDFs, VLM2Vec relies on external tools (such as OCR or PDF parsing libraries) to extract text content. When these tools process complex layouts, scanned documents, or low-quality documents, inaccurate text extraction may occur, resulting in biased retrieval results.

[0004] Therefore, how to improve the multimodal retrieval accuracy of documents is an urgent problem that needs to be solved. Summary of the invention

[0005] The present application provides a multimodal retrieval model training method, a multimodal retrieval method and a device to at least solve the problem of how to improve the multimodal retrieval accuracy of documents in the related art.

[0006] In a first aspect, the present application provides a method for training a multimodal retrieval model, comprising: Constructing a vector database; the vector database includes multi-vector representations of multiple document images; Based on the vector database, a difficult negative sample training set is constructed; the difficult negative sample training set includes: a plurality of query texts and a plurality of difficult negative sample document images corresponding to the query texts; The hard negative sample training set is input into an initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0007] In a second aspect, the present application provides a multimodal retrieval method, comprising: Obtaining the data to be retrieved input by the user; the data to be retrieved includes: target query text and target document image; The data to be retrieved is input into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in the first aspect.

[0008] In a third aspect, the present application also provides a training device for a multimodal retrieval model, comprising: A construction module, used to construct a vector database; the vector database includes multi-vector representations of multiple document images; A screening module is used to construct a difficult negative sample training set based on the vector database; the difficult negative sample training set includes: a plurality of query texts and a plurality of difficult negative sample document images corresponding to the query texts; The training module is used to input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0009] In a fourth aspect, the present application also provides a multimodal retrieval device, comprising: An input module is used to obtain the data to be retrieved input by the user; the data to be retrieved includes: a target query text and a target document image; A retrieval module is used to input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in the first aspect.

[0010] In a fifth aspect, the present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned multimodal retrieval model training methods or any of the above-mentioned multimodal retrieval methods when executing the computer program.

[0011] In a sixth aspect, the present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the training method of any of the above-mentioned multimodal retrieval models or the steps of any of the above-mentioned multimodal retrieval methods are implemented.

[0012] In a seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned multimodal retrieval model training methods or any of the above-mentioned multimodal retrieval methods.

[0013] Through this application, a vector database is constructed, and based on the vector database, a difficult negative sample training set is constructed, wherein the difficult negative sample training set includes: multiple query texts and multiple difficult negative sample document images corresponding to the query texts; the difficult negative sample training set is input into the initial multimodal retrieval model for training to obtain the target multimodal retrieval model. Since the vector database stores multi-vector representations of multiple documents, these multi-vector representations capture the semantic information of the document images. Through the vector database, negative samples (i.e., difficult negative samples) that are similar to but not related to the query vector can be found. That is, difficult negative samples refer to document image samples that are similar to the query text in the feature space, but are actually irrelevant to the query text. By mining these samples through the vector database and constructing a difficult negative sample training set, the model can learn more subtle distinguishing features, thereby improving the model's ability to distinguish between positive and negative samples, and improving the accuracy of retrieval during the retrieval process. At the same time, matching the query (to be retrieved) vector through the vector database helps to quickly locate and retrieve the vector representation with the highest relevance to the query, thereby improving retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 A flowchart of a method for training a multimodal retrieval model provided in an embodiment of the present application; Figure 2 A schematic diagram of a multimodal search method provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a training device for a multimodal retrieval model provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a multimodal retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0020] Terminology explanation: Multimodal retrieval is a technology that uses multiple data modalities (such as text, images, videos, audio, etc.) to retrieve information. It integrates data in different forms to provide more comprehensive, accurate and rich retrieval results to meet users' diverse query needs.

[0021] In some embodiments, Figure 1 As shown, a training method for a multimodal retrieval model is provided, comprising the following steps S11-S13: S11. Build a vector database.

[0022] Among them, the vector database includes multi-vector representations of multiple document images.

[0023] Specifically, a vector database is constructed through multi-vector representations of multiple document images. The multi-vector representations of multiple document images are obtained by training an initial multimodal retrieval model with an initial training sample set.

[0024] In some embodiments, before executing the above step S11 (building a vector database), the following steps are further executed: (1) Obtain the initial training sample set.

[0025] The initial training sample set includes: multiple query texts, multiple positive sample document images of the query texts, and negative sample document images. The positive sample document images are document images associated with the query texts, and the negative sample document images are document images not associated with the query texts. Each positive and negative sample document image has been labeled.

[0026] Specifically, a data set including document images and related query texts is obtained as an initial training sample set.

[0027] (2) Inputting the initial training sample set into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of the multiple document images.

[0028] The initial multimodal retrieval model is obtained by training the preset multimodal retrieval model based on the initial training sample set. In combination with the model architecture on which the execution of the training method of the multimodal retrieval model depends, the model architecture is described here. The preset multimodal retrieval model includes: a visual language model VLM, a large language model LLM, a mapping layer, etc.

[0029] Specifically, the initial training sample set is input into the preset multimodal retrieval model, the visual encoder of the visual language model is used to extract document visual features from the document image, the large language model is used to extract query features from the query text, and the document visual features and query features are respectively vectorized and encoded using a mapping layer to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images.

[0030] Optionally, the initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on an initial training sample set, specifically comprising the following steps: Based on random sampling of the initial training sample set, an in-batch negative sample training set is obtained; The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

[0031] Specifically, a certain number of samples are randomly selected from the set of non-positive samples in the initial training sample set as negative samples, and then these negative samples are input into the preset multimodal retrieval model together with the positive samples of each query as the in-batch negative sample training set for training to obtain the initial multimodal retrieval model. For example, a stratified random sampling method can be used to obtain the in-batch negative sample training set, which can ensure that each negative class can be properly represented in the negative samples, which is conducive to the model's learning of the characteristics of negative samples of different categories.

[0032] In some embodiments, the above step S11 (building a vector database) can be implemented in the following manner: 1) Segment multiple document images with a preset window size and preset resolution to obtain multiple document image data.

[0033] The preset window size and the preset resolution may be set according to actual application scenarios. For example, the preset window size may be 256×256 pixels, and the preset resolution may be 1024×1024 pixels.

[0034] Specifically, since each document includes at least one document image, by segmenting each document image according to a preset window size and a preset resolution, multiple slice images, that is, multiple document image data, are obtained. Segmenting the document image according to a predetermined window size and resolution helps to extract more fine-grained information from the document and improve the accuracy of retrieval.

[0035] 2) Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector.

[0036] Specifically, visual feature encoding refers to the process of converting visual information in an image or video into a low-dimensional, structured feature vector. These feature vectors can represent information such as objects, scenes, textures, and spatial relationships in the image. The segmented document image data is input into the VLM's visual encoder for visual feature encoding to obtain a visual feature vector.

[0037] 3) Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector.

[0038] Specifically, the visual feature vector is input into the mapping layer of the initial multimodal retrieval model for embedding encoding to obtain a visual embedding vector. The mapping layer is used to convert the visual feature vector into an embedding vector consistent with the query text feature dimension for subsequent multimodal fusion. For example, the visual feature vector is mapped to the target embedding dimension through a linear layer to obtain a visual embedding vector.

[0039] 4) Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features.

[0040] Specifically, the visual embedding vector is input into the large language model of the initial multimodal retrieval model. For example, the large language model executes the instruction: "Briefly describe the image information" and generates semantic features.

[0041] 5) Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document.

[0042] Specifically, the semantic features are input into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorized encoding to obtain a multi-vector representation of each document.

[0043] 6) Store the multi-vector representation of each document in a vector database.

[0044] Specifically, the multi-vector representation of each document is stored in the vector database. It should be noted that if new document images are added to the dataset in the vector database, the vector representations of these new images need to be recalculated and added to the vector database. If some document images in the dataset are deleted or modified, the corresponding vectors also need to be deleted or updated from the database. If the dataset or model has not changed and the performance of the vector database meets the current needs, there is no need to re-acquire the vector database.

[0045] S12. Based on the vector database, construct a hard negative sample training set.

[0046] The hard negative sample training set includes: multiple query texts and multiple hard negative sample document images corresponding to the query texts. The hard negative sample document images are document image samples that are similar to the query text in the feature space but are actually irrelevant to the query text.

[0047] In some embodiments, the above step S12 (building a hard negative sample training set based on the vector database) can be implemented as follows: a. Determine N multi-vector representations with the highest similarity to the multi-vector representations of each query text from the vector database, and obtain a first candidate sample set.

[0048] The first candidate sample set includes each query text and a candidate sample set of each query text. N is an integer greater than 0.

[0049] Optionally, the above step a (determining N multi-vector representations having the highest similarity to the multi-vector representations of each query text from the vector database to obtain the first candidate sample set) can be implemented as follows: Calculating the delayed interaction similarity between each query text and each document in the vector database; The delayed interaction similarities are used for sorting, and N multi-vector representations having the highest similarity to the multi-vector representations of the query texts are determined from the vector database to obtain a first candidate sample set.

[0050] Specifically, by calculating the delayed interaction similarity between each query text and all documents in the vector database, the documents are sorted in descending order according to the delayed interaction similarity score, and the N documents with the highest delayed interaction similarity are selected from the sorting results as the first candidate sample set. Since the core idea of ​​delayed interaction similarity is to maintain independence in the encoding process of query and document, the encoding of documents can be completed offline, and the computational complexity of the query stage is low, it is suitable for processing large-scale data sets, such as document retrieval scenarios.

[0051] Further, optionally, based on the delayed interaction similarity calculation formula, the delayed interaction similarity between each query text and each document in the vector database is calculated: .

[0052] in, Represents the query text and Documentation The similarity score between represents the multi-vector representation of the query vector, A multi-vector representation representing the document vector, and are the number of query vectors and document vectors respectively. represents the i-th query vector, represents the jth document vector, express and The dot product of two vectors.

[0053] For each vector in the query , find the document vector with the largest dot product . Calculate the values ​​of these maximum dot products, take their maximum value, and sum all these maximum dot product values ​​to get the final similarity score Specifically, the model encodes the query and document separately, generates their own vector representations, and then evaluates the correlation between the query multi-vector representation and the document multi-vector representation through the maximum similarity function. This method can effectively calculate the similarity between the query vector and the vector in the vector database to find the most relevant documents. This method maintains independence in the encoding process of the query and document, and performs fine-grained interaction when calculating the similarity, thereby reducing the computational complexity.

[0054] b. Eliminate the target candidate sample set from the first candidate sample set to obtain a second candidate sample set.

[0055] Among them, the target candidate sample set does not include positive samples of the target query text.

[0056] c. Based on the second candidate sample set, construct a hard negative sample training set.

[0057] Specifically, first, candidate samples are retrieved. For each query text, the top 100 candidate samples with the highest similarity scores are retrieved as the first candidate sample set. Then invalid samples are removed, and candidate sample sets and corresponding queries that do not contain positive samples in the first candidate sample set are removed, because these samples are not helpful for model training, to obtain the second candidate sample set. Finally, difficult negative samples are selected, and negative samples that are similar but unrelated to each query text are selected from the second candidate sample set as the difficult negative sample set. These samples can help the model better distinguish similar but unrelated samples.

[0058] S13: input the hard negative sample training set into an initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0059] In some embodiments, the above step S13 (inputting the hard negative sample training set into the initial multimodal retrieval model for training to obtain the target multimodal retrieval model) can be implemented as follows: Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text; Determine a preset loss function according to the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text; The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

[0060] Specifically, multiple query texts are input into the initial multimodal retrieval model to obtain the document retrieval results corresponding to each query. At this stage, the initial multimodal retrieval model will generate a list of documents related to the query texts, and attach the multimodal information of the documents (such as pictures). That is, multiple query texts are input into the initial multimodal retrieval model to obtain the document retrieval pictures corresponding to each query text.

[0061] For each query text, in addition to the retrieved positive samples (relevant documents), it is also necessary to introduce hard negative samples (irrelevant but easily confused documents). Based on these positive and negative sample pairs, a preset loss function is determined, that is, according to the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text, the preset loss function is determined. The preset loss function is used to make similar samples closer in the feature space and dissimilar samples farther apart, that is, the goal of the preset loss function is to make the similarity between the query text and the positive sample higher than the similarity between the query text and the negative sample.

[0062] By minimizing the preset loss function and adjusting the weight parameters of the initial multimodal retrieval model, the initial multimodal retrieval model optimizes the similarity calculation between the query text and the document by learning the above training data, thereby improving the accuracy and discrimination ability of the retrieval. After optimization training, the target multimodal retrieval model can process queries more accurately and distinguish between relevant and irrelevant documents, thereby providing higher quality retrieval results in the actual retrieval process.

[0063] Optionally, during the model training process, a preset loss function is used to train the initial multimodal retrieval model, wherein the preset loss function formula is as follows: .

[0064] in, represents the loss value, represents the batch size, that is, the number of samples used in each training iteration, represents the sum of all samples in the batch, Indicates The query multi-vector and Similarity scores between multiple vectors of positive sample documents; Indicates The query multi-vector and The similarity score between multiple vectors of negative sample documents. The function is a smooth activation function that is used to ensure that the gradient of the loss function is continuous and non-negative.

[0065] The preset contrast loss function works by optimizing the initial multimodal retrieval model by maximizing the similarity score of positive sample pairs and minimizing the similarity score of negative sample pairs. In this way, the model is trained to bring similar samples closer and push dissimilar samples further away, thereby improving the discriminative ability of feature representation and further improving retrieval accuracy.

[0066] The training method of the multimodal retrieval model provided by the present disclosure, since the vector database stores multi-vector representations of multiple documents, these multi-vector representations capture the semantic information of the document image. Through the vector database, negative samples (i.e., difficult negative samples) similar to but irrelevant to the query vector can be found, that is, difficult negative samples refer to document image samples that are similar to the query text in the feature space, but are actually irrelevant to the query text. By mining these samples through the vector database and constructing a difficult negative sample training set, the model can learn more subtle distinguishing features, thereby improving the model's ability to distinguish between positive and negative samples, and improving the accuracy of retrieval during the retrieval process. At the same time, matching the query (to be retrieved) vector through the vector database helps to quickly locate and retrieve the vector representation with the highest relevance to the query, thereby improving retrieval efficiency.

[0067] Figure 2 is a flowchart of a multimodal retrieval method provided by an embodiment of the present disclosure, such as Figure 2 As shown, a multimodal retrieval method provided by an embodiment of the present disclosure includes: S21. Obtain the data to be retrieved input by the user.

[0068] The data to be retrieved includes: target query text and target document image.

[0069] Specifically, the data to be retrieved input by the user is obtained, and the data to be retrieved includes the target query text and the target document image.

[0070] S22: input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved.

[0071] Among them, the target multimodal retrieval model is obtained based on any embodiment of the training method of the multimodal retrieval model provided in the above method embodiments.

[0072] In some embodiments, the above step S22 (inputting the data to be searched into the target multimodal search model to obtain the search results corresponding to the data to be searched) can be implemented as follows: A. Based on the target multimodal retrieval model, the target query text is segmented and vectorized to obtain a multi-vector representation of the target query text.

[0073] Specifically, based on the target multimodal retrieval model, the user's target query text is preprocessed, including word segmentation, stop word removal, stemming, etc., to convert it into a format that the model can understand.

[0074] Optionally, the above step A (based on the target multimodal retrieval model, performing word segmentation processing and vectorization encoding on the target query text to obtain a multi-vector representation of the target query text) can be implemented as follows: Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words; Encoding the segmented words to obtain multiple word codes; Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word; The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

[0075] Specifically, the preprocessed target query text is input into the language model (LLM) part of the model, and the text features are converted into vector representations through the mapping layer of the model.

[0076] B. Determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved and output them.

[0077] Wherein, M is an integer greater than 0.

[0078] Specifically, a similarity search is performed in a vector database using a query vector. A similarity score is calculated between the query vector and the document image vectors stored in the database. The results are sorted according to the similarity score, and the top N results with the highest scores are usually selected as candidate matches.

[0079] Further, the retrieved candidate matches are post-processed, which may include removing duplicates, re-ranking, etc. If needed, the retrieved document images can be further analyzed, such as extracting key information or generating summaries. The retrieval results are presented to the user, usually including thumbnails of the document images, titles, partial content previews, etc. A user interaction interface is provided to allow users to view detailed information, obtain complete documents, etc. In some systems, users can also provide feedback on the retrieval results, such as marking relevant or irrelevant. User feedback can be used to further optimize the model and vector database. Based on user feedback and new data, the model and vector database are regularly updated to improve retrieval performance and accuracy.

[0080] Through these steps, the trained multimodal retrieval model can effectively process user queries and retrieve the most relevant results from a large number of document images.

[0081] The multimodal retrieval method provided by the present disclosure stores multi-vector representations of multiple documents in a vector database, and these multi-vector representations capture the semantic information of the document images. Matching the query (to be retrieved) vectors through the vector database helps to quickly locate and retrieve the vector representation with the highest relevance to the query, thereby improving the retrieval efficiency.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0083] In some embodiments, reference Figure 3 As shown, the embodiment of the present application also provides a multimodal retrieval model training device 300, comprising: A construction module 310 is used to construct a vector database; the vector database includes multi-vector representations of multiple document images; A screening module 320 is used to construct a difficult negative sample training set based on the vector database; the difficult negative sample training set includes: a plurality of query texts and a plurality of difficult negative sample document images corresponding to the query texts; The training module 330 is used to input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

[0084] As an optional implementation of the embodiment of the present disclosure, the device further includes: An acquisition module is used to acquire an initial training sample set; the initial training sample set includes: a plurality of query texts, a plurality of positive sample document images of the query texts, and a negative sample document image; A generation module is used to input the initial training sample set into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images; the initial multimodal retrieval model is obtained by training the preset multimodal retrieval model based on the initial training sample set.

[0085] As an optional implementation of the embodiment of the present disclosure, the construction module 310 is specifically used to: Segment multiple document images with a preset window size and a preset resolution to obtain multiple document image data; Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector; Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector; Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features; Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorization encoding to obtain a multi-vector representation of each document; The multi-vector representation of each document is stored in a vector database.

[0086] As an optional implementation of the embodiment of the present disclosure, the screening module 320 includes: A matching unit, configured to determine N multi-vector representations having the highest similarity to the multi-vector representations of each query text from the vector database, and obtain a first candidate sample set; the first candidate sample set includes each query text and a candidate sample set of each query text; wherein N is an integer greater than 0; A elimination unit, configured to eliminate a target candidate sample set from the first candidate sample set to obtain a second candidate sample set; the target candidate sample set does not include positive samples of the target query text; A construction unit is used to construct a hard negative sample training set based on the second candidate sample set.

[0087] As an optional implementation of the embodiment of the present disclosure, the matching unit is specifically used for: Calculating the delayed interaction similarity between each query text and each document in the vector database; The delayed interaction similarities are used for sorting, and N multi-vector representations with the highest similarity to the multi-vector representations of each query text are determined from the vector database to obtain a first candidate sample set; wherein N is an integer greater than 0.

[0088] As an optional implementation of the embodiment of the present disclosure, the training module 330 is specifically used for: Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text; Determine a preset loss function according to the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text; The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

[0089] As an optional implementation of the embodiment of the present disclosure, the initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on an initial training sample set, including: Based on random sampling of the initial training sample set, an in-batch negative sample training set is obtained; The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

[0090] The description of the features in the embodiment corresponding to the training device of the multimodal retrieval model can be found in the relevant description of the embodiment corresponding to the training method of the multimodal retrieval model, and will not be repeated here.

[0091] In some embodiments, reference Figure 4 As shown, the embodiment of the present disclosure provides a multimodal retrieval device 400, the multimodal retrieval device 400 comprising: The input module 410 is used to obtain the data to be retrieved input by the user; the data to be retrieved includes: target query text and target document image; The retrieval module 420 is used to input the data to be retrieved into the target multimodal retrieval model to obtain the retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on any embodiment of the training method of the multimodal retrieval model provided in the above method embodiment.

[0092] As an optional implementation of the embodiment of the present disclosure, the retrieval module 420 includes: A processing unit, configured to perform word segmentation and vectorization encoding on the target query text based on the target multimodal retrieval model to obtain a multi-vector representation of the target query text; A retrieval unit is used to determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved and output them; wherein M is an integer greater than 0.

[0093] As an optional implementation of the embodiment of the present disclosure, the processing unit is specifically configured to: Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words; Encoding the segmented words to obtain multiple word codes; Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word; The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

[0094] For the description of the features in the embodiment corresponding to the multimodal retrieval device, reference can be made to the relevant description of the embodiment corresponding to the multimodal retrieval method, which will not be repeated here.

[0095] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned multimodal retrieval model training method embodiments.

[0096] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above-mentioned multimodal retrieval model training method embodiments or the steps in any of the above-mentioned multimodal retrieval methods when running.

[0097] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0098] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned multimodal retrieval model training method embodiments or the steps in any of the above-mentioned multimodal retrieval methods.

[0099] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned multimodal retrieval model training method embodiments or the steps of any of the above-mentioned multimodal retrieval methods.

[0100] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0101] The above is a detailed introduction to a training method for a multimodal retrieval model and a multimodal retrieval method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for training a multimodal retrieval model, characterized in that: include: Build vector database; The vector database includes multiple vector representations of multiple document images; Based on the vector database, construct a hard negative sample training set; The hard negative sample training set includes: a plurality of query texts and a plurality of hard negative sample document images corresponding to the query texts; The hard negative sample training set is input into an initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

2. The method for training a multimodal retrieval model according to claim 1, characterized in that: Before constructing the vector database, the method further includes: Acquire an initial training sample set; the initial training sample set includes: multiple query texts, multiple positive sample document images of the query texts, and negative sample document images; The initial training sample set is input into a preset multimodal retrieval model to generate a multi-vector representation of the query text and a multi-vector representation of multiple document images; the initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on the initial training sample set.

3. The training method of the multimodal retrieval model according to claim 1, characterized in that: The constructing of the vector database comprises: Segment multiple document images with a preset window size and a preset resolution to obtain multiple document image data; Inputting the plurality of document image data into the visual encoder of the initial multimodal retrieval model for visual feature encoding to obtain a visual feature vector; Inputting the visual feature vector into the mapping layer of the initial multimodal retrieval model for embedding coding to obtain a visual embedding vector; Inputting the visual embedding vector into the large language model of the initial multimodal retrieval model to generate semantic features; Inputting the semantic features into the multi-vector output mapping layer of the initial multimodal retrieval model for vectorization encoding to obtain a multi-vector representation of each document; The multi-vector representation of each document is stored in a vector database.

4. The method for training a multimodal retrieval model according to claim 1, characterized in that: The constructing a hard negative sample training set based on the vector database includes: Determine N multi-vector representations with the highest similarity to the multi-vector representations of each query text from the vector database to obtain a first candidate sample set; the first candidate sample set includes each query text and a candidate sample set of each query text; wherein N is an integer greater than 0; Eliminate the target candidate sample set from the first candidate sample set to obtain a second candidate sample set; the target candidate sample set does not include positive samples of the target query text; Based on the second candidate sample set, a hard negative sample training set is constructed.

5. The method for training a multimodal retrieval model according to claim 4, characterized in that: The step of determining N multi-vector representations having the highest similarity to the multi-vector representations of each query text from the vector database to obtain a first candidate sample set includes: Calculating the delayed interaction similarity between each query text and each document in the vector database; The delayed interaction similarities are used for sorting, and N multi-vector representations with the highest similarity to the multi-vector representations of each query text are determined from the vector database to obtain a first candidate sample set; wherein N is an integer greater than 0.

6. The method for training a multimodal retrieval model according to claim 1, characterized in that: The step of inputting the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model includes: Inputting the multiple query texts into the initial multimodal retrieval model to obtain a document retrieval image corresponding to each query text; Determine a preset loss function according to the document retrieval images corresponding to each query text and the hard negative samples corresponding to each query text; The weight parameters of the initial multimodal retrieval model are adjusted according to the preset loss function to obtain the target multimodal retrieval model.

7. The method for training a multimodal retrieval model according to claim 2, characterized in that: The initial multimodal retrieval model is obtained by training a preset multimodal retrieval model based on an initial training sample set, and includes: Based on random sampling of the initial training sample set, an in-batch negative sample training set is obtained; The negative sample training set in the batch is input into a preset multimodal retrieval model for training to obtain an initial multimodal retrieval model.

8. A multimodal retrieval method, characterized in that: include: Get the data to be retrieved entered by the user; The data to be retrieved includes: target query text and target document image; The data to be retrieved is input into the target multimodal retrieval model to obtain the retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in any one of claims 1-7.

9. The multimodal retrieval method according to claim 8, characterized in that: The step of inputting the data to be retrieved into the target multimodal retrieval model to obtain a retrieval result corresponding to the data to be retrieved includes: Based on the target multimodal retrieval model, the target query text is segmented and vectorized to obtain a multi-vector representation of the target query text; Determine M multi-vector representations with the highest similarity to the multi-vector representation of the target query text from the vector database, determine the document image data corresponding to the M multi-vector representations as the retrieval results of the data to be retrieved and output them; wherein M is an integer greater than 0.

10. The multimodal retrieval method according to claim 9, characterized in that: The step of performing word segmentation and vectorization encoding on the target query text based on the target multimodal retrieval model to obtain a multi-vector representation of the target query text includes: Performing word segmentation processing on the target query text based on the word segmenter of the target multimodal retrieval model to obtain segmented words; Encoding the segmented words to obtain multiple word codes; Vectorizing the plurality of word codes based on the embedding layer of the target multimodal retrieval model to obtain a vector representation of each word; The vector representation of each word is combined to obtain a multi-vector representation of the target query text.

11. A training device for a multimodal retrieval model, characterized in that: include: Construction module, used to construct vector database; The vector database includes multiple vector representations of multiple document images; A screening module, used for constructing a hard negative sample training set based on the vector database; The hard negative sample training set includes: a plurality of query texts and a plurality of hard negative sample document images corresponding to the query texts; The training module is used to input the hard negative sample training set into the initial multimodal retrieval model for training to obtain a target multimodal retrieval model.

12. A multimodal retrieval device, characterized in that: include: An input module, used to obtain the data to be retrieved input by the user; The data to be retrieved includes: target query text and target document image; A retrieval module, used to input the data to be retrieved into the target multimodal retrieval model to obtain retrieval results corresponding to the data to be retrieved; the target multimodal retrieval model is obtained based on the training method of the multimodal retrieval model described in any one of claims 1-7.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the training method for a multimodal retrieval model as described in any one of claims 1 to 7, or the multimodal retrieval method as described in any one of claims 8 to 10 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the training method of the multimodal retrieval model according to any one of claims 1 to 7, or the multimodal retrieval method according to any one of claims 8 to 10.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the multimodal retrieval model as described in any one of claims 1 to 7, or the multimodal retrieval method as described in any one of claims 8 to 10.

Images