Positive and negative sample construction method and full-modal reordering model training method

Abstract

The invention discloses a positive and negative sample construction method and a full-modal reordering model training method, and belongs to the technical field of artificial intelligence, the method comprises the following steps: constructing Query of a text mode and an image-text mixed arrangement mode of a first page based on a first multi-modal large language model, the first page being a page in a first document; according to the modality of the first page, converting the first page into a plurality of Documents in different modalities; and performing permutation and combination on the Query of the text mode and the image-text mixed arrangement mode of the first page and the Document of the plurality of different modes converted from the first page to obtain a full-mode positive sample corresponding to the first page. According to the method, effective and multi-modal positive and negative samples can be constructed.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for constructing positive and negative samples and a method for training a full-modal reordering model. Background Technology

[0002] In the RAG (Retrieve Augmented Generation) system, hard negative samples are used to train the reranker. The reranker is used to re-score the similarity of samples in the recall phase, thereby obtaining a higher recall rate; the reranker also has a corresponding improvement effect on other retrieval tasks.

[0003] In related technologies, when constructing hard-negative samples and training reordering models, only text-modal queries are typically used as samples, and training is also performed using only text-modal queries.

[0004] However, in multimodal scenarios, the types of user input queries can be diverse. Training the reorderer with only text-modal queries and positive and negative samples composed solely of text-modal queries will result in poor performance of the reorderer. Summary of the Invention

[0005] This disclosure provides a method for constructing positive and negative samples and a method for training a full-modal reordering model, capable of constructing effective, multimodal positive and negative samples. The technical solution includes at least the following: Firstly, a method for constructing positive samples is provided, wherein each sample includes a Query and a Document, and the sample type includes positive samples, comprising: constructing a Query of the text modality and a mixed text-image modality of a first page based on a first multimodal large language model, wherein the first page is a page in a first document; converting the first page into multiple Documents of different modalities according to the modality of the first page; and arranging and combining the Query of the text modality and the mixed text-image modality of the first page and the multiple Documents of different modalities converted from the first page to obtain a full-modal positive sample corresponding to the first page.

[0006] Optionally, if the first page does not contain an image, the step of converting the first page into multiple documents with different modalities based on the modality of the first page includes: converting the first page into Markdown text to obtain a text-modal document of the first page; and converting the Markdown text into a Markdown image to obtain an image-modal document of the first page. If the first page contains an image, the step of converting the first page into multiple documents with different modalities based on the modality of the first page further includes: combining the Markdown text and the Markdown image to obtain a text-image mixed-layout document of the first page.

[0007] Optionally, constructing the text modality and image-text mixed-format query of the first page based on the first multimodal large language model includes: obtaining the first query of the text modality corresponding to the first page using the first multimodal large language model based on the Markdown text; taking a screenshot of the Markdown image to obtain a first screenshot; obtaining the text content of the second query of the image-text mixed-format query corresponding to the first page using the first screenshot and the Markdown image based on the first multimodal large language model; and combining the first screenshot and the text content of the second query to obtain the second query of the image-text mixed-format query of the first page.

[0008] Optionally, if the first page does not contain an image, the multiple positive samples of different modalities constructed by the first page include: the first query and the Markdown text, the second query and the Markdown text, the first query and the Markdown image, and the second query and the Markdown image.

[0009] Optionally, when the first page contains images, the multiple positive samples constructed by the first page in different modalities include: the first query and the Markdown text, the second query and the Markdown text, the first query and the Markdown image, the second query and the Markdown image, the first query and the document of the first page in the mixed text and image modality, and the second query and the document of the first page in the mixed text and image modality.

[0010] Secondly, a method for constructing negative samples is provided. Each sample includes a Query and a Document. The sample type includes negative samples. The method includes: obtaining a full-modal positive sample corresponding to a second page and a third page, wherein the second page and the third page are different pages in the first document, and the similarity between the second page and the third page is greater than a similarity threshold. The full-modal positive sample corresponding to the second page is obtained using the method described in the first aspect; inputting the Query corresponding to the second page and the third page into a detection agent, and obtaining the output of the detection agent; if the output of the detection agent indicates that the third page cannot answer the Query corresponding to the second page, then the Query corresponding to the second page and the third page are taken as a difficult negative sample.

[0011] Thirdly, a method for training a full-modal re-ranking model is provided, comprising: encoding multiple positive samples and multiple hard negative samples using a visual encoder, wherein the multiple positive samples are obtained using the method described in the first aspect, and the multiple hard negative samples are obtained using the method described in the second aspect; inputting the encoded multiple positive samples and multiple hard negative samples into a second multimodal large language model; for any sample input into the second multimodal large language model, obtaining the Softmax probability of the second multimodal large language model on the two tokens "yes" and "no", and using the probability of the second multimodal large language model on the token "yes" as the similarity between the Query and Document of the sample; optimizing the samples of the second multimodal large language model using a loss function to obtain a full-modal re-ranking model; wherein the loss function includes Softmax loss and cross-modal loss.

[0012] Optionally, in the loss function, the Softmax loss is calculated using the original value of the last token in the output layer of the second multimodal large language model; when the fourth page does not contain an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query among the multiple samples corresponding to the fourth page is calculated as the cross-modal loss, and the fourth page is a page in a document used to train the second multimodal large language model; when the fourth page contains an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query among the samples corresponding to the fourth page that do not contain Markdown text is calculated as the cross-modal loss.

[0013] Fourthly, a positive sample construction device is also provided, wherein each sample includes a Query and a Document, and the sample type includes positive samples, comprising: a Query construction module, used to construct a Query of the text modality and the image-text mixed modality of the first page based on a first multimodal large language model, wherein the first page is a page in a first document; a Document construction module, used to convert the first page into multiple Documents of different modalities according to the modality of the first page; and a positive sample construction module, used to arrange and combine the Query of the text modality and the image-text mixed modality of the first page and the multiple Documents of different modalities converted from the first page to obtain a full-modality positive sample corresponding to the first page.

[0014] Optionally, if the first page does not contain an image, the Document construction module is further configured to convert the first page into Markdown text to obtain a text-modal Document of the first page; convert the Markdown text into a Markdown image to obtain an image-modal Document of the first page; if the first page contains an image, the Document construction module is further configured to combine the Markdown text and the Markdown image to obtain a text-image mixed-layout Document of the first page.

[0015] Optionally, the Query construction module is further configured to: obtain a first Query of the text modality corresponding to the first page using a first multimodal large language model based on the Markdown text; take a screenshot of the Markdown image to obtain a first screenshot; obtain the text content in a second Query of the mixed text and image modality corresponding to the first page using the first multimodal large language model based on the first screenshot and the Markdown image; and combine the first screenshot and the text content in the second Query to obtain a second Query of the mixed text and image modality of the first page.

[0016] Optionally, if the first page does not contain an image, the positive sample construction module may construct multiple positive samples of different modalities from the first page, including: the first query and the Markdown text, the second query and the Markdown text, the first query and the Markdown image, and the second query and the Markdown image.

[0017] Optionally, when the first page contains images, the positive sample construction module constructs multiple positive samples of different modalities for the first page, including: the first query and the Markdown text, the second query and the Markdown text, the first query and the Markdown image, the second query and the Markdown image, the first query and the document of the first page in the mixed text and image mode, and the second query and the document of the first page in the mixed text and image mode.

[0018] Fifthly, a negative sample construction device is also provided, wherein each sample includes a Query and a Document, and the sample type includes negative samples, comprising: a first acquisition module, configured to acquire a full-modal positive sample corresponding to a second page and a third page, wherein the second page and the third page are different pages in the first document, and the similarity between the second page and the third page is greater than a similarity threshold, and the full-modal positive sample corresponding to the second page is obtained by the method described in the first aspect; a second acquisition module, configured to input the Query corresponding to the second page and the third page into a detection agent, and acquire the output of the detection agent; and a difficult negative sample construction module, configured to, when the output of the detection agent indicates that the third page cannot answer the Query corresponding to the second page, take the Query corresponding to the second page and the third page as a difficult negative sample.

[0019] Sixthly, a full-modal re-ranking model training device is also provided, comprising: an encoding module for encoding multiple positive samples and multiple hard-negative samples using a visual encoder, wherein the multiple positive samples are obtained using the method described in the first aspect, and the multiple hard-negative samples are obtained using the method described in the second aspect; an input module for inputting the encoded multiple positive samples and multiple hard-negative samples into a second multimodal large language model; a similarity output module for obtaining the Softmax probability of the second multimodal large language model on the two tokens "yes" and "no" for any sample input into the second multimodal large language model, and using the probability of the second multimodal large language model on the token "yes" as the similarity between the Query and Document of the sample; and a training optimization module for optimizing the samples of the second multimodal large language model using a loss function to obtain a full-modal re-ranking model; wherein the loss function includes Softmax loss and cross-modal loss.

[0020] Optionally, in the training optimization module, the Softmax loss is calculated using the original value of the last token in the output layer of the second multimodal large language model in the loss function; when the fourth page does not contain an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query among the multiple samples corresponding to the fourth page is calculated as the cross-modal loss, and the fourth page is a page in a document used to train the second multimodal large language model; when the fourth page contains an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query among the samples corresponding to the fourth page that do not contain Markdown text is calculated as the cross-modal loss.

[0021] In a seventh aspect, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the positive and negative sample construction method and the full modality reordering model training method described in the above embodiments.

[0022] Eighthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the positive and negative sample construction method and the full modality reordering model training method described in the above embodiments.

[0023] Ninthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the methods described in the first, second, and third aspects.

[0024] The beneficial effects of the technical solutions provided in this disclosure include at least the following: In this embodiment, based on a first multimodal large language model, queries for the text modality and mixed text / image modality of a first page are constructed. The first page is a page in a first document. According to the modality of the first page, it is transformed into multiple documents of different modalities. The queries for the text modality and mixed text / image modality of the first page, along with the multiple documents of different modalities transformed from the first page, are arranged and combined to obtain a full-modal positive sample corresponding to the first page. Since different modal queries and documents are constructed for the first page, combining these different modal queries and documents is equivalent to constructing a full-modal positive sample for the first page. Subsequent training of the reorderer using this full-modal positive sample can effectively improve the training effect of the reorderer, resulting in a full-modal reorderer. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a positive sample construction method provided in an exemplary embodiment of this disclosure is shown; Figure 2 A flowchart illustrating a negative sample construction method provided in an exemplary embodiment of this disclosure is shown; Figure 3 This is a flowchart of the algorithm for retrieving queries of different modalities from the first page; Figure 4 A flowchart illustrating a full-modal reordering model training method provided in an exemplary embodiment of this disclosure is shown. Figure 5 A schematic diagram of the structure of a positive sample construction apparatus provided in an exemplary embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of a negative sample construction apparatus provided in an exemplary embodiment of this disclosure is shown; Figure 7 This illustration shows a schematic diagram of the structure of a full-modal reordering model training apparatus provided in an exemplary embodiment of the present disclosure; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0027] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0028] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0029] After the user inputs a query, the RAG can retrieve multiple candidate documents based on that query. The reorderer calculates the similarity between each candidate document and the query, and then reorders these candidate documents according to the similarity scores. In multimodal scenarios, the modal types of the user-input query and the candidate documents retrieved from it are diverse. The modal types of the user-input query include text modality and mixed text / image modality. Here, when using the reorderer, users generally do not input only images; otherwise, the large model or the RAG system would not be able to understand the user's purpose. Therefore, the user-input query usually contains text; that is, the user-input query is almost never a purely image modality.

[0030] Candidate documents typically include multiple pages, with page modal types including text modality, image modality, and mixed text and image modality. Based on the combination of these different modalities, the modality of the candidate document may include the following five cases: pure text modality (every page is text modality), pure image modality (every page is image modality), pure mixed text and image modality (every page is mixed text and image modality), text modality plus image modality (some pages are text modality, some pages are image modality, and some pages are mixed text and image modality).

[0031] Figure 1 A flowchart illustrating a positive sample construction method provided in an exemplary embodiment of this disclosure is shown, the method being executable by a computer device. See also Figure 1 The method includes: In step 101, the first page is converted into multiple Documents with different modalities according to the modality of the first page.

[0032] The first page is a page within the first document. Each sample consists of a query and a document, and the sample type includes positive samples.

[0033] In this embodiment of the disclosure, the first document is a document used to construct a sample, and the document contains multiple pages.

[0034] Here, each sample contains a query and a document, which can be denoted as... Positive samples are those where the Document can answer the Query, and negative samples are those where the Document cannot answer the Query. In this embodiment, the samples are constructed from pages; that is, in this embodiment, both the Query and the Document in a sample are obtained from a single page, not from a single document.

[0035] When constructing multiple positive samples with different modalities, considering that both Query and Document may have different modalities, the page needs to be processed in a certain way to obtain the Query and Document with different modalities corresponding to the page.

[0036] Optionally, if the first page does not contain an image, step 101 includes steps a to b as follows.

[0037] Step a: Convert the first page into Markdown text to obtain the text modal document of the first page; Step b: Convert the Markdown text into a Markdown image to obtain the image-modal Document of the first page; In this embodiment, the first document may be a file of type PDF, Word, Excel, Wikipedia, etc., and correspondingly, the first page may also be a file of the above type. The modalities of each file type are different. For example, a page in Word may be in text modality, a mixed text and image modality, or an image modality, while a page in PDF is only in image modality.

[0038] In order to unify different types of documents and obtain different modal queries and documents on each page, in this embodiment of the disclosure, the first page is first converted into Markdown text to obtain the text modal document of the first page; then the Markdown text is converted into an image modal Markdown image to obtain the image modal document of the first page.

[0039] There are many technologies available for retrieving the Markdown text from the first page and converting it into a Markdown image. For example, it can be achieved using a document parser, which will not be detailed here.

[0040] If the first page contains an image, in addition to performing steps a to b, step 101 also includes step c.

[0041] Step c: Combine the Markdown text and Markdown images to obtain the document in the first page's mixed text and image format.

[0042] In step 102, based on the first multimodal large language model, the queries for the text modality and the mixed text and image modality of the first page are constructed.

[0043] Since the query input by the user when using the reorderer usually includes text, and tasks are assigned to the model through text, the modal types of the query in the samples constructed by each page in this embodiment include: text modal and mixed text and image modal.

[0044] Optionally, step 102 includes steps d to g as follows. Figure 2 This is a flowchart of the algorithm for retrieving queries of different modalities from the first page. The following section combines... Figure 2 Steps d through g will be explained.

[0045] Step d: Based on the Markdown text, use the first multimodal large language model to obtain the first query of the text modality corresponding to the first page.

[0046] The first multimodal large language model in this embodiment is an MLLM (Multimodal Large Language Models).

[0047] When implementing step d, the first prompt word and Markdown text can be input into the first multimodal large language model by pre-setting the first prompt word, thereby obtaining the first query of the text modality corresponding to the first page output by the first multimodal large language model.

[0048] The first prompt word is used to indicate how to extract a query from Markdown text.

[0049] Step e: Take a screenshot of the Markdown image to obtain the first screenshot.

[0050] For example, when a user doesn't understand something on a webpage or in a document, they usually take a screenshot, paste the screenshot into the large model dialog box, and enter a question, thus forming a query with mixed text and image input.

[0051] The first screenshot in step e is to simulate the image portion of the query in the user-input mixed text and image modality.

[0052] Step f: Based on the first screenshot and the Markdown image, the text content in the second query of the image-text mixed layout modality corresponding to the first page is obtained using the first multimodal large language model.

[0053] Here, the second query is the query for the mixed text and image layout mode corresponding to the first page.

[0054] Step f is to obtain the text portion of the second query, which, together with the first screenshot itself, constitutes the second query.

[0055] In step f, a second prompt word, a first screenshot, and a Markdown image can be input into the first multimodal large language model to obtain the text content of the second query corresponding to the first page output by the first multimodal large language model. Here, the Markdown image is input into the first multimodal large language model as raw information, which provides global information, while the first screenshot provides local information. Inputting both into the first multimodal large language model improves the reliability of the text content in the generated second query.

[0056] The second prompt word is used to indicate how to extract the query based on the first screenshot and the Markdown image.

[0057] Step g: Combine the screenshot of the first page and the text content in the second query to obtain the second query of the image and text mixed layout modality of the first page.

[0058] In step 103, the queries of the text modality and the mixed text and image modality of the first page, as well as the documents of multiple different modalities transformed from the first page, are arranged and combined to obtain the positive samples of the full modality corresponding to the first page.

[0059] For the case where the first page does not contain images, the samples corresponding to the first page include two modalities for the Query (text modality and mixed text / image modality) and two modalities for the Document (text modality and image modality). Therefore, the permutations and combinations of different modalities of Query and Document yield four positive samples corresponding to the first page. These four positive samples are: the first Query and the Markdown text. Second Query and Markdown text First Query and Markdown Images Second Query and Markdown Images .

[0060] If the first page contains images, then among the samples corresponding to the first page, the Query includes two modalities (text modality and mixed text / image modality), and the Document includes three modalities (text modality, image modality, and mixed text / image modality). Therefore, the permutations and combinations of Query and Document with different modalities can yield six positive samples corresponding to the first page. These six positive samples are: the first Query and the Markdown text. Second Query and Markdown text First Query and Markdown Images Second Query and Markdown Images The first query and the document of the first page with mixed text and image layout. The document of the second query and the first page's mixed text and image layout modality .

[0061] in, Indicates Query, The subscript indicates the modality of the query. Represents Document. The subscript indicates the modality of the Document. Represents text modality, Represents image modality, This indicates a mixed text and image layout mode.

[0062] In some embodiments, multiple screenshots of the page can be taken to obtain multiple different second queries, and the number of positive samples corresponding to each page can be even greater.

[0063] After processing each page in the first document using the methods in steps 101 to 103, we can obtain the full-modal positive samples corresponding to each page in the first document.

[0064] In this embodiment, based on a first multimodal large language model, queries for the text modality and mixed text / image modality of a first page are constructed. The first page is a page in a first document. According to the modality of the first page, it is transformed into multiple documents of different modalities. The queries for the text modality and mixed text / image modality of the first page, along with the multiple documents of different modalities transformed from the first page, are arranged and combined to obtain a full-modal positive sample corresponding to the first page. Since different modal queries and documents are constructed for the first page, combining these different modal queries and documents is equivalent to constructing a full-modal positive sample for the first page. Subsequent training of the reorderer using this full-modal positive sample can effectively improve the training effect of the reorderer, resulting in a full-modal reorderer.

[0065] Figure 3 A flowchart illustrating a negative sample construction method provided in an exemplary embodiment of this disclosure is shown, the method being executable by a computer device. See also Figure 3 The method includes: In step 301, the positive samples of the full modality corresponding to the second page and the third page are obtained.

[0066] The second and third pages are different pages in the first document, and the similarity between the second and third pages is greater than the similarity threshold. The positive samples of the full modality corresponding to the second page are obtained using the positive sample construction method in steps 101 to 103.

[0067] In implementing step 301, a visual encoder can first encode each page in the first document to obtain the embedding features (VD embedding) of each page. Then, multiple page pairs are constructed (any two different pages can be selected to form a page pair). The similarity between the two pages in each page pair is calculated, thus obtaining page pairs with a similarity greater than a similarity threshold. The second page and the third page are two pages in a page pair with a similarity greater than the similarity threshold. If there are multiple page pairs with a similarity greater than the similarity threshold, steps 301 to 303 are executed sequentially for the two pages in each page pair to construct hard-to-bear samples.

[0068] In step 302, the Query corresponding to the second page and the third page are input into the detection agent to obtain the output of the detection agent.

[0069] Here, the query corresponding to the second page is obtained from the positive samples of the full modality of the second page.

[0070] In this embodiment of the disclosure, the detection agent is used to determine whether the third page can answer the query corresponding to the second page.

[0071] When implementing step 302, all queries corresponding to the second page (including the first and second queries) and the third page are input to the detection agent.

[0072] If the third page can answer the query corresponding to the second page, the detection intelligence will output "yes"; if the third page cannot answer the query corresponding to the second page, the detection intelligence will output "no".

[0073] In step 303, if the output of the detection agent indicates that the third page cannot answer the query corresponding to the second page, the query corresponding to the second page and the third page are treated as a hard negative sample.

[0074] The fact that the third page cannot answer the query corresponding to the second page indicates that the third page and the query corresponding to the second page constitute a negative sample. The document in the positive sample of the query corresponding to the second page is from multiple modalities of the second page, and the similarity between the third page and the second page is greater than the similarity threshold, meaning the third page and the second page are relatively similar. This indicates that the document in the positive sample of the query corresponding to the second page is also relatively similar. Therefore, the model has difficulty identifying this negative sample formed by the third page and the query corresponding to the second page, which meets the definition of a "hard negative sample." In other words, the query corresponding to the second page and the third page constitute a hard negative sample.

[0075] Based on a similar principle, the second page and the third page in steps 301 to 303 can also be swapped. That is, multiple positive samples corresponding to the third page and the second page are obtained, and then the query corresponding to the third page and the second page are input into the detection agent to obtain the output of the detection agent. When the output of the detection agent indicates that the second page cannot answer the query corresponding to the third page, the query corresponding to the third page and the second page are taken as a hard negative sample.

[0076] Through steps 301 to 303 above, hard negative samples can be constructed from the positive samples of the full modality of the first document, thereby realizing the construction of hard negative samples.

[0077] In this embodiment, by acquiring the full-modal positive sample corresponding to the second page and the third page (the second and third pages are different pages in the first document, and their similarity is greater than a similarity threshold), the query corresponding to the second page and the third page are input into the detection agent, and the output of the detection agent is obtained. When the output of the detection agent indicates that the third page cannot answer the query corresponding to the second page, the query corresponding to the second page and the third page are used as a hard-negative sample, ensuring the effectiveness and accuracy of the constructed hard-negative sample. Since a multimodal query has been constructed for each page, the query corresponding to the second page is also a full-modal query. Using the query corresponding to the second page and the third page as a hard-negative sample is equivalent to obtaining a full-modal hard-negative sample. Subsequent training of the reorderer using this full-modal hard-negative sample can effectively improve the training effect of the reorderer, resulting in a full-modal reorderer.

[0078] Figure 4 A flowchart illustrating a full-modal reordering model training method provided in an exemplary embodiment of this disclosure is shown, which can be executed by a computer device. See also Figure 4 The method includes: In step 401, a visual encoder is used to encode multiple positive samples and multiple hard negative samples.

[0079] Multiple positive samples are obtained using the positive sample construction method in steps 101 to 103, and multiple difficult negative samples are obtained using the negative sample construction method in steps 301 to 303.

[0080] In this embodiment of the disclosure, multiple positive samples and multiple negative samples are used to train a second multimodal large language model, which serves as the base for the full-modal reordering model. Since the samples obtained by the aforementioned method are full-modal samples, the full-modal reordering model can be obtained after the second multimodal large language model has been trained.

[0081] The second multimodal large language model in this embodiment is an MLLM (Multimodal Large Language Models).

[0082] In step 402, the encoded positive samples and multiple hard negative samples are input into the second multimodal large language model.

[0083] In step 403, for any sample input to the second multimodal large language model, the softmax probability of the second multimodal large language model on the two tokens yes and no is obtained, and the probability of the second multimodal large language model on the token yes is used as the similarity between the Query and Document of the sample.

[0084] For example, when a sample is input into a second multimodal large language model, the softmax probability of the second multimodal large language model on the "yes" token represents the probability that the document in that sample can answer the query; the softmax probability of the second multimodal large language model on the "no" token represents the probability that the document in that sample cannot answer the query. In this embodiment of the disclosure, the probability of the second multimodal large language model on the "yes" token is used as the similarity between the query and document in that sample. This similarity can be used to re-rank the samples, thereby obtaining a re-ranking model.

[0085] In step 404, the loss function is used to optimize the samples of the second multimodal large language model to obtain the full modality reordering model.

[0086] The loss function includes Softmax loss and cross-modal loss, which is expressed by formula (1).

[0087] (1) In formula (1), For loss function, The loss is the Softmax loss, which is the output loss of the second multimodal large language model. For cross-modal loss, and The preset weights, and Plus It equals 1.

[0088] The softmax loss and cross-modal loss are explained below.

[0089] Optionally, the Softmax loss can be calculated using the original value of the last token in the output layer of the second multimodal language model (i.e., the Logit of the last token). Here, the original value of the last token in the output layer of the second multimodal language model is not directly a probability; it needs to be transformed to obtain the Softmax probabilities of the second multimodal language model for the yes and no tokens. Detailed explanations of how to calculate the Softmax loss are available in various related technologies and will not be elaborated upon here.

[0090] Cross-modal loss is used for cross-modal alignment. In this embodiment, the cross-modal loss is calculated for the same query. For example, if two samples have the same query but different document modalities, the cross-modal loss between these two samples can be calculated, thereby achieving cross-modal alignment of the model's output. In this case, since the corresponding positive samples differ depending on whether the page contains images, the objects requiring cross-modal alignment also differ. Therefore, the following discussion uses the fourth page as an example, where the fourth page is a page in the document used to train the second multimodal large language model.

[0091] When the fourth page does not contain images and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence is calculated as the cross-modal loss among different samples with the same query in the multiple samples corresponding to the fourth page.

[0092] That is, calculation and The KL divergence between them, and the calculation and The KL divergence between the two modes is used as the cross-modal loss.

[0093] When the fourth page contains images and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence is calculated as the cross-modal loss among different samples with the same query in the samples corresponding to the fourth page that do not contain Markdown text.

[0094] That is, calculation and The KL divergence between them, and the calculation and The KL divergence between the two modes is used as the cross-modal loss.

[0095] Here, in the case where the fourth page contains an image, in the plain text modal of the fourth page (i.e. In the fourth page, the image will become a link, which is equivalent to the plain text modality of the fourth page at this time (i.e., Information loss exists in the fourth page (because the fourth page has become a link, the information originally carried by the image on the fourth page is lost), which is not conducive to alignment. Therefore, when calculating the cross-modal loss, only the cross-modal loss between samples that do not contain Markdown text is calculated to improve the efficiency of cross-modal alignment.

[0096] By introducing cross-modal loss into the loss function, we can ensure that the training of the second multimodal large language model aligns the modal information, ensuring that the final training effect is unbiased across all modalities, and thus obtaining a full-modal reorderer.

[0097] When calculating the KL divergence between two samples, the embedding of the last hidden layer in the second multimodal large language model is used to calculate the KL divergence. This ultimately achieves the goal of similar embeddings in the last hidden layer, i.e., similar distributions across all dimensions. If this goal can be achieved, for the same query, using different modalities to represent candidate documents can yield relatively consistent outputs, achieving the goal of unbiasedness.

[0098] For example, the calculation is performed using formula (2). and The KL divergence between them.

[0099] (2) In formula (2), if formula (2) is used to calculate and The KL divergence between them, then This indicates that the input is In the case of the second multimodal large language model, the embedding distribution in the last hidden layer is... This indicates that the input is In the case of the second multimodal large language model, the feature of the i-th dimension in the embedding of the last hidden layer; then This indicates that the input is In the case of the second multimodal large language model, the embedding distribution in the last hidden layer is... This indicates that the input is In the case of the second multimodal large language model, the feature of the i-th dimension in the embedding of the last hidden layer.

[0100] In related technologies, when performing multimodal reordering, if the input is a query in the same text modality and both plain text and image modality candidate documents are retrieved, the similarity between the query and each candidate document needs to be calculated. Since both the query and the plain text candidate documents belong to the image modality, a first similarity distribution can be easily obtained. However, the query and the image modality candidate documents belong to different modalities, so calculating their similarity requires mapping the text modality query to the image modality candidate documents to obtain a second similarity distribution. Due to the modal differences, the first and second similarity distributions are different distributions, indicating a modality gap between them.

[0101] In this embodiment of the disclosure, by introducing cross-modal loss when training the second multimodal large language model, the second multimodal large language model can obtain relatively consistent outputs when using different modalities to represent candidate documents on the same query, thus offsetting the information loss caused by modality conversion. This ensures that the conversion between different modalities does not affect the final model output conclusion, thereby solving the problem of the alignment modality gap.

[0102] The training process of the aforementioned full-modal reorderer can align textual meaning to specific image spaces in a supervised manner, and vice versa, complementing each other. Therefore, the training effect of the full-modal reorderer is superior to that of a single-modal reorderer.

[0103] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0104] Figure 5 A schematic diagram of a positive sample construction apparatus provided in an exemplary embodiment of this disclosure is shown. See also Figure 5 Each sample includes a Query and a Document. The sample type includes positive samples. The hard sample construction device 500 includes: a Query construction module 501, a Document construction module 502, and a positive sample construction module 503.

[0105] The Query construction module 501 is used to construct the text modality and image-text mixed mode of the first page based on the first multimodal large language model. The first page is a page in the first document. Document construction module 502 is used to convert the first page into multiple Documents with different modalities based on the modality of the first page; The positive sample construction module 503 is used to arrange and combine the queries of the text modality and the mixed text and image modality of the first page, as well as the documents of multiple different modalities transformed from the first page, to obtain the positive samples of the full modality corresponding to the first page.

[0106] Optionally, if the first page does not contain an image, the Document construction module 502 is further configured to convert the first page into Markdown text to obtain a text-modal Document for the first page; convert the Markdown text into a Markdown image to obtain an image-modal Document for the first page; if the first page contains an image, the Document construction module 502 is further configured to combine the Markdown text and the Markdown image to obtain a text-image mixed-layout Document for the first page.

[0107] Optionally, the Query construction module 501 is further configured to obtain the first Query of the text modality corresponding to the first page based on the Markdown text using the first multimodal large language model; take a screenshot of the Markdown image to obtain a first screenshot; based on the first screenshot and the Markdown image, use the first multimodal large language model to obtain the text content in the second Query of the image-text mixed layout modality corresponding to the first page; and combine the text content in the first screenshot and the second Query to obtain the second Query of the image-text mixed layout modality of the first page.

[0108] Optionally, if the first page does not contain an image, the positive samples constructed in the positive sample construction module 503 include the following multiple modalities of positive samples: first query and Markdown text, second query and Markdown text, first query and Markdown image, and second query and Markdown image.

[0109] Optionally, when the first page contains images, the positive sample construction module 503 constructs multiple positive samples of different modalities for the first page, including: a first query and Markdown text, a second query and Markdown text, a first query and Markdown image, a second query and Markdown image, a document of the first query and the first page's mixed text and image modality, and a document of the second query and the first page's mixed text and image modality.

[0110] Figure 6 A schematic diagram of a negative sample construction apparatus provided in an exemplary embodiment of this disclosure is shown. See also Figure 6 Each sample includes a Query and a Document. The sample type includes negative samples. The hard negative sample construction device 600 includes: a first acquisition module 601, a second acquisition module 602 and a hard negative sample construction module 603.

[0111] The first acquisition module 601 is used to acquire the full-modal positive samples corresponding to the second page and the third page. The second page and the third page are different pages in the first document, and the similarity between the second page and the third page is greater than the similarity threshold. The full-modal positive samples corresponding to the second page are obtained using the positive sample construction method.

[0112] The second acquisition module 602 is used to input the Query corresponding to the second page and the third page into the detection agent and acquire the output of the detection agent.

[0113] The difficult-to-bear sample construction module 603 is used to treat the query corresponding to the second page and the third page as a difficult-to-bear sample when the output of the detection agent indicates that the third page cannot answer the query corresponding to the second page.

[0114] Figure 7 A schematic diagram of a full-modal reordering model training apparatus provided in an exemplary embodiment of this disclosure is shown. See also Figure 7 The full-modal reordering model training device 700 includes: an encoding module 701, an input module 702, a similarity output module 703, and a training optimization module 704.

[0115] The encoding module 701 is used to encode multiple positive samples and multiple difficult negative samples using a visual encoder. The multiple positive samples are obtained using a positive sample construction method, and the multiple difficult negative samples are obtained using a negative sample construction method.

[0116] The input module 702 is used to input the encoded positive samples and multiple hard negative samples into the second multimodal large language model.

[0117] The similarity output module 703 is used to obtain the Softmax probability of the second multimodal large language model on the two tokens "yes" and "no" for any sample input to the second multimodal large language model, and use the probability of the second multimodal large language model on the token "yes" as the similarity between the Query and Document of the sample.

[0118] The training optimization module 704 is used to optimize the samples of the second multimodal large language model using loss functions to obtain a full-modal re-ranking model. The loss functions include Softmax loss and cross-modal loss.

[0119] Optionally, in the training optimization module 704, the original value of the last token in the output layer of the second multimodal large language model is used to calculate the Softmax loss in the loss function; when the fourth page does not contain an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query is calculated as the cross-modal loss among the multiple samples corresponding to the fourth page, where the fourth page is a page in the document used to train the second multimodal large language model; when the fourth page contains an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query is calculated as the cross-modal loss among the samples corresponding to the fourth page that do not contain Markdown text.

[0120] It should be noted that the above-described embodiments of the positive sample construction device, negative sample construction device, and full-modal reordering model training device are only illustrative examples of the division of functional modules. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the positive sample construction device and positive sample construction method embodiments provided in the above embodiments belong to the same concept, as do the negative sample construction device and negative sample construction method embodiments, and the full-modal reordering model training device and full-modal reordering model training method embodiments. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0121] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in the various embodiments of this disclosure can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0122] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. For example... Figure 8 As shown, the computer device 800 includes a processor 801 and a memory 802.

[0124] Processor 801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0125] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 is used to store at least one instruction, which is executed by the processor 801 to implement the positive and negative sample construction method and the full-modal reordering model training method provided in the embodiments of this disclosure.

[0126] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0127] This disclosure also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the positive and negative sample construction method and the full-modal reordering model training method provided in this disclosure.

[0128] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the positive and negative sample construction method and the full-modal reordering model training method provided in this disclosure.

[0129] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for constructing positive samples, characterized in that, Each sample includes a query and a document, the sample type includes positive samples, and the method includes: Based on the first multimodal large language model, construct the text modality and image-text mixed mode query of the first page, where the first page is a page in the first document; Based on the modality of the first page, the first page is converted into multiple Documents with different modalities; The queries of the text modality and the mixed text and image modality of the first page, as well as the documents of multiple different modalities transformed from the first page, are arranged and combined to obtain the positive samples of the full modality corresponding to the first page.

2. The method according to claim 1, characterized in that, If the first page does not contain images, the step of converting the first page into multiple documents with different modalities based on the modality of the first page includes: Convert the first page to Markdown text to obtain the text modal document of the first page; The Markdown text is converted into a Markdown image to obtain the document in the image modality of the first page; If the first page contains an image, the step of converting the first page into multiple documents with different modalities based on the modality of the first page further includes: The Markdown text and the Markdown image are combined to obtain the document of the first page in the mixed text and image format.

3. A method for constructing negative samples, characterized in that, Each sample includes a query and a document, the sample type includes negative samples, and the method includes: Obtain the full-modal positive samples corresponding to the second page and the third page, wherein the second page and the third page are different pages in the first document, and the similarity between the second page and the third page is greater than the similarity threshold. The full-modal positive samples corresponding to the second page are obtained by the method described in claim 1 or 2. Input the Query corresponding to the second page and the third page into the detection agent, and obtain the output of the detection agent; If the output of the detection agent indicates that the third page cannot answer the query corresponding to the second page, the query corresponding to the second page and the third page are treated as a hard negative sample.

4. A method for training a full-modal reordering model, characterized in that, The method includes: A visual encoder is used to encode multiple positive samples and multiple hard negative samples, wherein the multiple positive samples are obtained by the method of claim 1 or 2, and the multiple hard negative samples are obtained by the method of claim 3; The encoded positive samples and multiple hard negative samples are input into the second multimodal large language model; For any sample input to the second multimodal large language model, obtain the Softmax probability of the second multimodal large language model on the two tokens yes and no, and use the probability of the second multimodal large language model on the token yes as the similarity between the Query and Document of the sample; The loss function is used to optimize the samples of the second multimodal large language model to obtain the full modality re-ranking model; The loss function includes Softmax loss and cross-modal loss.

5. The method according to claim 4, characterized in that, In the loss function, the Softmax loss is calculated using the original value of the last token in the output layer of the second multimodal large language model; In the case where the fourth page does not contain an image and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query among the multiple samples corresponding to the fourth page is calculated as the cross-modal loss. The fourth page is a page in a document used to train the second multimodal large language model. When the fourth page contains images and multiple samples corresponding to the fourth page are input into the second multimodal large language model, the KL divergence between different samples with the same query is calculated as the cross-modal loss for the samples corresponding to the fourth page that do not contain Markdown text.

6. A positive sample construction device, characterized in that, Each sample includes a query and a document, and the sample type includes positive samples. The device includes: The Query construction module is used to construct the text modality and image-text mixed modality of the first page based on the first multimodal large language model. The first page is a page in the first document. The Document construction module is used to convert the first page into multiple Documents with different modalities based on the modality of the first page; The positive sample construction module is used to arrange and combine the queries of the text modality and the mixed text and image modality of the first page, as well as the documents of multiple different modalities transformed from the first page, to obtain the positive samples of the full modality corresponding to the first page.

7. A negative sample construction device, characterized in that, Each sample includes a query and a document, and the sample type includes negative samples. The device includes: The first acquisition module is used to acquire the full-modal positive samples corresponding to the second page and the third page, wherein the second page and the third page are different pages in the first document, and the similarity between the second page and the third page is greater than the similarity threshold. The full-modal positive samples corresponding to the second page are obtained by the method described in claim 1 or 2. The second acquisition module is used to input the Query corresponding to the second page and the third page into the detection agent and acquire the output of the detection agent; The difficult-to-bear sample construction module is used to treat the query corresponding to the second page and the third page as a difficult-to-bear sample when the output of the detection agent indicates that the third page cannot answer the query corresponding to the second page.

8. A training device for a full-modal reordering model, characterized in that, The device includes: An encoding module is used to encode multiple positive samples and multiple hard negative samples using a visual encoder, wherein the multiple positive samples are obtained by the method of claim 1 or 2, and the multiple hard negative samples are obtained by the method of claim 3; The input module is used to input the encoded positive samples and multiple hard negative samples into the second multimodal large language model; The similarity output module is used to obtain the Softmax probability of the second multimodal large language model on the two tokens "yes" and "no" for any sample input to the second multimodal large language model, and use the probability of the second multimodal large language model on the token "yes" as the similarity between the Query and Document of the sample. The training optimization module is used to optimize the samples of the second multimodal large language model using a loss function to obtain a full-modal re-ranking model; The loss function includes Softmax loss and cross-modal loss.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method according to any one of claims 1 to 2, claim 3, or any one of claims 4 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method described in any one of claims 1 to 2, claim 3, or any one of claims 4 to 5.

Images