Data enhancement and training method for multi-hop question-answer retrieval model

Abstract

The invention discloses a data enhancement and training method for a multi-hop question and answer retrieval model, which comprises the following steps: step 1, a multi-hop question and answer data set is obtained, the data set is composed of multi-hop questions and a document set corresponding to the multi-hop questions, and the document set comprises a first-hop retrieval document, a second-hop retrieval document and other related documents; step 2, performing positive example denoising on a first hop document and a second hop document in the document set to obtain a denoised document as a new positive example for model training, and taking other residual parts of the document as training supplementary negative examples; and step 3, inputting the obtained data into a multi-hop question-answer pre-training language model for training. According to the positive example denoising strategy based on the proposition clauses, the proposition clauses serve as intermediate steps, information interference irrelevant to questions in document paragraphs is remarkably reduced, and therefore the capture precision and efficiency of the model for relevant information are improved. The strategy enhances the accuracy and robustness of the model in the multi-hop reasoning task.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a training method for a multi-hop question-answering model, and specifically to a data enhancement and training method for a multi-hop question-answering retrieval model. Background Art

[0002] In the training process of the multi-hop question-answering model, the core idea is to use known supporting facts to predict new supporting facts, and then train an efficient sentence encoder. This method achieves effective encoding of document content by converting the entire document into an embedding vector of fixed dimension. However, when dealing with documents with variable length, one of the main challenges faced by the model is how to filter out irrelevant information, the so-called "noise". Due to the huge amount of information contained in the document, not all information is directly related to the training objective, so the noise problem becomes a key factor affecting the performance of the model.

[0003] In addition, although the in-batch negative sampling strategy adopted by the current MDR (multi-hop dense retrieval) model can reduce the cost of negative sample sampling to a certain extent, its selection mechanism is relatively random and simple, which makes it difficult for the model to effectively learn complex logical relationships, further limiting the expressiveness of the model.

[0004] In summary, the problems existing in the prior art are:

[0005] 1. The MDR algorithm used in the multi-hop question-answering retrieval model encounters a noise problem. One of the main challenges is how to filter out irrelevant information, that is, "noise" information that is not related to the multi-hop question.

[0006] 2. Although the in-batch negative sampling strategy used by the current model can reduce the cost of negative sample sampling to a certain extent, its selection mechanism is relatively random and simple, which makes it difficult for the model to effectively learn complex logical relationships, further limiting the expressiveness of the model. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a data enhancement and training method for a multi-hop question-answering retrieval model.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] A data enhancement and training method for a multi-hop question-answering retrieval model comprises the following steps:

[0010] Step 1: Obtain a multi-hop question answering dataset, which consists of complex multi-hop questions and their corresponding document sets, including first-hop retrieval documents, second-hop retrieval documents, and other related documents;

[0011] Step 2: Perform positive example denoising on the first-hop document and the second-hop document in the document collection, and use the denoised documents as new positive examples for model training. The remaining parts of the documents are used as supplementary negative examples for training. The specific steps are as follows:

[0012] Step 21: extract proposition clauses from the first-hop document and the second-hop document in the document set using a large language model and prompt engineering technology to obtain a corresponding proposition clause candidate set;

[0013] Step 22: According to the multi-hop problem, the semantic similarity score of each proposition clause in the candidate set is calculated using BERT Score and sorted in descending order to obtain a sorted proposition clause candidate set;

[0014] Step 23, set the retention percentage α, retain the TopK proposition clauses, and concatenate the TopK proposition clauses in the original order to obtain the denoised positive example text;

[0015] Step 3: Input the data obtained in step 2 into the multi-hop question-answering pre-trained language model for training. The specific steps are as follows:

[0016] Step 31: Based on the Bert text embedding model, the first-hop document, the second-hop document, and the negative example document are converted into corresponding text embedding vectors;

[0017] Step 32: Design a multi-hop question-answering pre-trained language model

[0018] Step 321, selecting the MDR model as the baseline model;

[0019] Step 322: introduce a positive example denoising strategy based on proposition clauses. Proposition clauses are defined as atomic expressions in the text. Each proposition contains an independent fact or information point. The process of positive example denoising is as follows: first, a proposition extraction model is used to identify and extract multiple proposition sets from a single text paragraph. Then, for each proposition in the set, the semantic relevance score of the proposition with the given question q and the related supporting documents is calculated;

[0020] Step 323: Introduce a hyperparameter to control the ratio between the number of retained propositions and the total number of propositions. According to the hyperparameter a, propositions within this ratio are selected to be retained. These propositions will be reassembled according to their order in the original text to maintain the coherence of the text. In addition, in order to avoid losing propositions containing answer information, propositions related to the answer are also included in the retained set. The remaining proposition set = P\ is used to generate additional negative samples to further enrich the training data set.

[0021] Step 33: input the data obtained in step 31 into the multi-hop question-answering pre-trained language model designed in step 32 for training.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This paper proposes a positive example denoising strategy based on propositional clauses. Under this strategy, propositional clauses are defined as atomic expressions in documents that can independently express a single fact or information point, and are presented in a concise and self-contained form. By using propositional clauses as an intermediate step, the interference of information irrelevant to the question in the document paragraph can be significantly reduced, thereby improving the accuracy and efficiency of the model in capturing relevant information. This strategy enhances the accuracy and robustness of the model in multi-hop reasoning tasks by reducing redundant information. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Enhance the flow chart for data;

[0025] Figure 2 Schematic diagram of the training and usage methods for the multi-hop question answering model. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0027] The present invention provides a data enhancement and training method for a multi-hop question-answering retrieval model. Figure 2 As shown, the method comprises the following steps:

[0028] Step 1: Obtain a multi-hop question-answering dataset, which consists of complex multi-hop questions and their corresponding document sets. These document sets include first-hop retrieval documents, second-hop retrieval documents, and other related documents.

[0029] The multi-hop question answering dataset is a dataset specially designed for training multi-hop question answering models. It contains a large number of complex multi-hop questions and corresponding document collections. Each question requires multiple steps (or "hops") of reasoning to find the answer, requiring the model to learn and understand the semantic associations of texts between different hops. The present invention uses the public data resources of HotpotQA and MuSiQue for verification. Among them, HotpotQA is a multi-hop reasoning question answering dataset developed by researchers at Stanford University in the United States. The dataset aims to promote the understanding and solution capabilities of complex problems in the field of natural language processing, especially those that require information integration from multiple sources to answer. HotpotQA requires the model to not only understand a single document, but also effectively integrate information from different documents to complete multi-step reasoning. MuSiQue (Multihop Questions via Single-hop Question Composition) is a multi-hop reasoning question answering dataset introduced by the Knowledge Engineering Research Laboratory of Tsinghua University. The purpose of the design of this dataset is to overcome the problems that can be answered by shortcuts in existing datasets. MuSiQue adopts a bottom-up question synthesis method, which strengthens the logical connection between questions by ensuring that the answer to each sub-question is necessary to solve the subsequent questions.

[0030] Step 2: Perform positive example denoising on the first-hop document and the second-hop document in the document collection, and use the denoised documents as new positive examples for model training, and use the remaining parts of the documents as supplementary negative examples for training. The specific steps are as follows:

[0031] Step 21. Use a large language model and prompt engineering technology to extract proposition clauses from the first-hop document and the second-hop document in the document collection to obtain the corresponding proposition clause candidate set. For example, the original document: Before the restoration work from 1990 to 2001, the Leaning Tower of Pisa tilted 5.5 degrees, but now the Leaning Tower of Pisa tilts about 3.99 degrees. This means that the top of the Leaning Tower of Pisa is horizontally offset from the center by 3.9 meters (12 feet 10 inches). Then the extracted proposition clause set is: 1. Before the restoration work from 1990 to 2001, the Leaning Tower of Pisa tilted 5.5 degrees. 2. The Leaning Tower of Pisa now tilts about 3.99 degrees. 3. The top of the Leaning Tower of Pisa is horizontally offset from the center by 3.9 meters (12 feet 10 inches).

[0032] Step 22: According to the multi-hop problem, the semantic similarity score of each proposition clause in the candidate set is calculated using BERT Score and sorted in descending order to obtain a sorted proposition clause candidate set.

[0033] BERT Score is a semantic similarity calculation method based on the BERT (Bidirectional Encoder Representations from Transformers) model. It uses BERT's powerful language understanding capabilities to convert text into semantic vector representations and evaluate the semantic relevance between texts by calculating the similarity between these vectors. During the calculation process, BERT Score considers vocabulary matching (Precision and Recall) in the text and cosine similarity between embedded vectors (a variant of F1 Score). The BERT Score calculation steps are as follows:

[0034] Step 221, calculate word embedding:

[0035] Perform text word embedding on each proposition clause text N of the proposition clause set corresponding to the multi-hop question text C and the positive example document to obtain vector sequences emb_C and emb_N respectively.

[0036] Step 222, calculate the cosine similarity matrix:

[0037] Calculate the cosine similarity between each word embedding in the multi-hop question text and the proposition clause text to obtain the similarity matrix S, which is expressed in formula (1):

[0038]

[0039] Among them, S ij Indicates the cosine similarity between the embedding of the i-th word in the multi-hop question text and the embedding of the j-th word in the proposition clause text; cos_similarity(emb_C i ,emb_N J ) represents the calculation of two word embedding vectors emb_C i and emb_N J cosine similarity of; emb_C i Represents the word embedding vector of the i-th word in the multi-hop question text; emb_N J Represents the word embedding vector of the jth word in the proposition clause text; In emb_C i *emb_N j represents the dot product of two word embedding vectors, ||emb_C i || and ||emb_N j || respectively represent the modulus of the word embedding vector of the i-th word in the multi-hop question text and the modulus of the word embedding vector of the j-th word in the proposition clause text. By calculating the dot product divided by the product of the two vector moduli, we get the cosine value of the angle between the two vectors, that is, their cosine similarity.

[0040] Step 223, calculate precision, recall and harmonic mean F1:

[0041] For each word in the multi-hop question text, find the word with the highest similarity in the proposition clause text and calculate the average similarity as precision, which is expressed as shown in formula (2):

[0042]

[0043] Among them, P represents precision, that is, the accuracy rate. In the above formula, |C| represents the number of words in the multi-hop question text, c∈C represents traversing each word c in the multi-hop question text, where C is the set of all words in the multi-hop question text, r∈N represents traversing each word r in the proposition clause text, where N is the set of all words in the proposition clause text, s(c,r) represents the similarity between word c in the multi-hop question text and word r in the proposition clause text, and max r∈R s(c,r) means that for a specific word c in the multi-hop question text, the value with the highest similarity among all the words in the proposition clause text is found, Σ c∈ C max r∈R s(c,r) means that for each word in the multi-hop question text, find the word with the highest similarity in the proposition clause text and add up these maximum similarity values.

[0044] For each word in the proposition clause text, find the word with the highest similarity in the multi-hop question text and calculate the average similarity as recall, which is expressed as shown in formula (3):

[0045]

[0046] Among them, R represents recall, that is, the recall rate. The |N| in the sentence represents the number of words in the proposition clause text. c∈N represents traversing each word c in the proposition clause text, where N is the set of all words in the proposition clause text, and r∈C represents traversing each word r in the multi-hop question text, where C is the set of all words in the multi-hop question text. s(c,r) represents the similarity between word c in the proposition clause text and word r in the multi-hop question text. max r∈C s(c,r) means that for a specific word c in the proposition clause text, the value with the highest similarity among all the words in the multi-hop question text is found, Σ c∈R max r∈ Cs(c,r) means that for each word in the proposition clause text, find the word with the highest similarity in the multi-hop question text and add up these maximum similarity values.

[0047] The final BERTScore is the harmonic mean of P and R, and its expression is shown in formula (4):

[0048]

[0049] Among them, F1 represents the final BERTScore. P represents the precision calculated above, which is the average of the maximum similarity of each word in the multi-hop question text found in the proposition clause text, indicating the matching degree of the multi-hop question text and the proposition clause text in terms of accuracy. R represents recall, which is the average of the maximum similarity of each word in the proposition clause text found in the multi-hop question text, indicating the degree to which the information in the proposition clause text is recalled in the multi-hop question text. It is a formula for calculating the harmonic mean of two numbers. Here, the final BERTScore is obtained by calculating the harmonic mean of precision and recall, which comprehensively considers the performance of both precision and recall, making the evaluation result more comprehensive and balanced.

[0050] After calculating the semantic similarity scores between all candidate clauses and the multi-hop question, these scores are sorted in descending order. In the sorted set, clauses with higher scores have stronger semantic relevance to the question and are therefore more likely to contain the answer to the question or relevant information. Several clauses with the highest scores can be selected as inputs for subsequent reasoning steps according to actual needs.

[0051] Step 23: Set the retention percentage α, retain the TopK proposition clauses, and concatenate the TopK proposition clauses in the original order to obtain the denoised positive example text. The specific steps are as follows:

[0052] Step 231, define the retention percentage α: the retention percentage α is a value between 0 and 1, which is used to determine the proportion of clauses that should be retained in the sorted proposition clause candidate set. For example, if α = 0.2, it means that the first 20% of the clauses after sorting will be retained.

[0053] Step 232, calculate TopK: The value of TopK can be calculated by multiplying the retention percentage α by the total number of candidate clauses. For example, if the total number of candidate clauses is 100 and α = 0.2, then TopK = 20, which means that the first 20 clauses after sorting will be retained. Note: In actual applications, if the value calculated by TopK is not an integer, it is usually necessary to round it off or round it off. In addition, if the total number of candidate clauses is small, the value of α may need to be adjusted according to the actual situation to ensure that enough clauses are retained.

[0054] Step 233, concatenate in the original order to obtain the denoised positive example text, which specifically includes the following contents:

[0055] 1) Select the TopK proposition clauses: According to the calculated TopK values, select the topK clauses from the sorted proposition clause candidate set. These clauses are sorted from high to low according to the semantic similarity score with the multi-hop question, so the proposition clause with the highest score has the strongest relevance to the question text.

[0056] 2) Splicing in original order: After selecting the top K clauses, they need to be spliced ​​in the order in which they appear in the original document (i.e. original order). This step is to maintain the logical coherence of the text and the integrity of the information. Although the clauses may be disrupted during the sorting process, splicing them in the original order can restore the logical relationship between them.

[0057] 3) Obtain denoised positive example text: After processing in step 2), a denoised positive example text consisting of the TopK proposition clauses in the original order is obtained. This text removes irrelevant or redundant information to the question and retains the clauses that are most likely to contain the answer or relevant information. It will serve as input for subsequent reasoning and answer generation steps to help the model understand and answer questions more accurately.

[0058] Step 3: input the data obtained in step 2 into the multi-hop question answering pre-trained language model for training, that is, input the data into the multi-hop question answering pre-trained language model designed by the present invention for training. The specific steps are as follows:

[0059] Step 31: The Bert-based text embedding model is used to convert the question into a question embedding vector, and the first-hop document and the second-hop document are converted into corresponding text embedding vectors. These vectors capture the semantic information of the text and can be represented in the semantic vector space, so that similar sentences are closer in the vector space. Similarly, negative example documents are also converted into corresponding text embedding vectors so that they can be compared with positive examples during the training process, thereby optimizing the model and enhancing the retrieval ability of the model. The specific steps are as follows:

[0060] Step 311, convert the multi-hop question text into a question embedding vector. When processing a multi-hop question answering task, the user's question needs to be converted into a question embedding vector first. This is achieved by inputting the question text into a BERT-based text embedding model. BERT is a pre-trained deep bidirectional encoder that can capture the semantic information of the text well through language representation learned on large-scale text data. The question embedding vector captures the semantic information of the question, allowing the model to understand the intent and key content of the question.

[0061] Step 312: The first-hop document and the second-hop document are converted into corresponding text embedding vectors. In multi-hop question answering, the first-hop document usually refers to the text that directly answers the question, while the second-hop document is the information needed after the first-hop document and question text are obtained. These texts are also converted into corresponding text embedding vectors so that the model can understand and process them. These text embedding vectors also capture the semantic information of the text, allowing the model to identify the associations and differences between texts.

[0062] These text embedding vectors are represented in the semantic vector space, where each dimension represents a specific semantic feature of the text. Similar sentences (i.e., semantically close or related sentences) will be closer, while different sentences will be farther away. This representation makes it easier for the model to identify and understand the semantic relationship between texts.

[0063] Step 313: negative example documents are converted into corresponding embedding vectors. In order to optimize the model's ability to identify the correct information flow, negative example documents are usually introduced. Negative example documents refer to texts that are irrelevant or wrong to the question or the correct answer. These negative example documents are also converted into corresponding embedding vectors so that they can be compared with positive examples (i.e., questions, first-hop documents, and second-hop documents) during the training process.

[0064] Step 32: Design a multi-hop question-answering pre-trained language model

[0065] The model idea designed by the present invention is to train the sentence encoder by predicting subsequent supporting facts by using previous supporting facts. This method realizes efficient encoding of the document content by converting the text of the entire document into an embedding vector of fixed dimension. However, when facing documents of variable length, the MDR model will inevitably encounter noise problems when processing, because a large amount of information contained in the document may not be directly related to the goal of training the semantic association of the document. In this regard, the MDR model is selected as the baseline model of the present invention. Although the in-batch strategy adopted by the MDR model plays a role in reducing the cost of negative sampling, its negative sample selection mechanism is too random and simple, which limits the model's ability to fully learn and capture complex logical relationships. In order to solve these problems, the present invention introduces a positive example denoising strategy based on propositional clauses. Propositional clauses are defined as atomic expressions in text, each proposition contains an independent fact or information point, and is presented in a concise, self-contained natural language format. This strategy can effectively avoid too much noisy information in the semantics and sentences contained in the paragraph. By taking the proposition as an intermediate step, the present invention can maximize the reduction of information in the text segment that is not related to the question.

[0066] The process of positive example denoising is as follows Figure 1 Specifically, first, the proposition extraction model is used to identify and extract multiple proposition sets from a single text paragraph. Then, for each proposition in the set, the semantic relevance score between it and the given question q and the related supporting documents is calculated. The score can be calculated using the ordinary Bert Score or the Bert-based Rank model.

[0067] In order to effectively screen out important propositions, the present invention introduces a hyperparameter a, which is used to control the ratio between the number of retained propositions and the total number of propositions. According to the hyperparameter a, the present invention selects propositions within this ratio to be retained, and these propositions will be reassembled according to their order in the original text to maintain the coherence of the text. In addition, in order to avoid losing propositions containing answer information, the present invention also includes those propositions related to the answer into the retained set, and the remaining proposition set = P\ is used to generate additional negative samples to further enrich the training data set. P is the proposition set obtained after a single document is subjected to the proposition extraction model. The formulas are shown in formula (5) and formula (6):

[0068] Pa + ←augment(P + ,q,K ɑ ) (5)

[0069] Pa - ←P + \Pa+ (6)

[0070] Among them, P + is the original positive document, K ɑ is the number of selected sentences, q is the given question of the training set samples, Pa + is the positive example document after denoising, Pa - The part after the positive examples are removed is used as a negative example for training other samples. Through the above method, the present invention not only improves the quality of data and the robustness of the model, but also enhances the model's ability to distinguish between relevant text and irrelevant text by introducing positive and negative samples.

[0071] Step 33: input the data obtained in step 31 into the multi-hop question-answering pre-trained language model designed in step 32 for training.

[0072] During model training, the model learns how to distinguish between positive and negative documents. Through contrastive learning, the model can gradually optimize its ability to identify the correct information flow, that is, it can more accurately judge which text is necessary to answer the question and which text is irrelevant or wrong. This ability is crucial to improving the accuracy and efficiency of multi-hop question answering systems.

[0073] During model training, different contrastive losses are calculated to optimize the relationship between the question embedding vector and the related text embedding vector. Contrastive losses include:

[0074] a. For the question embedding vector, two sets of positive and negative examples are used to calculate the first two contrastive losses. In these two sets, the first-hop document embedding vector is used as the positive example, while the second-hop document embedding vector and the negative example embedding vector in the batch are used as the negative example, thus obtaining a contrastive loss of 1.

[0075] b. Combine the question embedding vector with the first-hop document embedding vector, use the second-hop document embedding vector as the positive example, and the in-batch negative example embedding vector as the negative example, thus obtaining a contrastive loss of 2.

[0076] c. Add a contrast loss 3, which is specifically optimized for the relationship between the question embedding vector and the positive example embedding vectors of two consecutive steps (i.e., the first hop and the second hop). In this case, the positive example is composed of the first hop document embedding vector and the second hop document embedding vector, while the negative example is the negative example embedding vector added after denoising. This design is to further improve the performance of the model in complex multi-hop reasoning scenarios.

[0077] The present invention uses the BM25 algorithm to mine difficult-to-handle negative examples from the Wikipedia knowledge base, and combines the denoising residues of other samples in the batch as negative examples. This comprehensive strategy not only improves the model's ability to recognize positive examples, but also enhances the model's training effect through carefully selected negative examples, thereby improving the overall performance of the model. The training loss function is shown in formula (7), formula (8), and formula (9):

[0078]

[0079] L total =L1+λL2 (9)

[0080] Among them, L1 represents the loss function using denoised positive samples, which is similar to the loss of the original MDR but uses denoised positive samples; L2 represents an additional loss function that includes negative samples formed from discarded positive propositions and previously unused hard negative samples; L total represents the overall loss function; represents the feature mapping function, which converts the input into its feature representation; T represents the temperature hyperparameter, which is used to scale the dot product result and affect the behavior of the softmax function; λ represents the hyperparameter, which balances the influence of the two loss functions of L1 and L2; x i It represents a query, i.e. a question; represents the denoised positive sample; x k represents the negative samples in the batch, including hard negative samples from BM25, B is the total number of batches; x n represents the negative samples formed from the discarded positive propositions and the previously unused hard negative samples, and M is the total number of negative samples.

[0081] Example:

[0082] Assuming that the HotpotQA dataset is used, a specific multi-hop question and its corresponding document set are as follows:

[0083] Question: What is the nationality of the director of the movie "The Godfather"?

[0084] Document collection D:

[0085] First jump document d_1: Francis Ford Coppola is an American film director, producer and screenwriter. His most famous work is directing the "Godfather" trilogy. Coppola was born on April 7, 1939 in Detroit, Michigan. He received a degree in drama from Hofstra University.

[0086] The Godfather is a 1972 American crime film directed by Francis Ford Coppola. The film was adapted from Mario Puzo's 1969 novel of the same name. The Godfather is considered one of the greatest films in film history. The film won several Oscars, including Best Picture.

[0087] Positive example denoising:

[0088] Extract proposition clause:

[0089] d_1:

[0090] 1. Francis Ford Coppola is an American film director.

[0091] 2. His most famous work is directing the Godfather trilogy.

[0092] 3. Coppola was born on April 7, 1939 in Detroit, Michigan.

[0093] 4. He earned a degree in theater from Hofstra University.

[0094] d_2:

[0095] 1. The Godfather is a 1972 American crime film directed by Francis Ford Coppola.

[0096] 2. The film is based on Mario Puzo's 1969 novel of the same name.

[0097] 3. "The Godfather" is considered one of the greatest films in film history.

[0098] 4. The film won several Oscars, including Best Picture.

[0099] Calculate the semantic similarity score:

[0100] d_1:

[0101] 1. Francis Ford Coppola is an American film director. Score: 0.85.

[0102] 2. He is best known for directing the Godfather trilogy. Score: 0.90.

[0103] 3. Coppola was born on April 7, 1939 in Detroit, Michigan. Score: 0.70.

[0104] 4. He earned a degree in theater from Hofstra University. Score: 0.65.

[0105] d_2:

[0106] 1. The Godfather is a 1972 American crime film directed by Francis Ford Coppola. Score: 0.88.

[0107] 2. The film is based on Mario Puzo's 1969 novel of the same name. Score: 0.75.

[0108] 3. The Godfather is considered one of the greatest films in film history. Score: 0.70.

[0109] 4. The film won several Oscars, including Best Picture. Score: 0.65.

[0110] Keep the TopK proposition clauses (assuming \(\alpha=0.5\), keep the first 2):

[0111] Positive example text after denoising\(Pa + )=[d_1*="Francis Ford Coppola is an American film director. He is best known for directing The Godfather trilogy"d_2*="The Godfather is a 1972 American crime film directed by Francis Ford Coppola. The film was based on the 1969 novel of the same name by Mario Puzo."]

[0112] Additional negative examples \(Pa-) = ["Coppola was born on April 7, 1939 in Detroit, Michigan. He received a degree in drama from Hofstra University.", "The Godfather is considered one of the greatest films in film history. The film won several Academy Awards, including Best Picture."]

[0113] Negative example generation:

[0114] Use the BM25 algorithm to mine hard negative examples from the Wikipedia knowledge base, for example:

[0115] "Francis Ford Coppola is a versatile filmmaker whose work spans a wide range of genres."

[0116] "He not only directed the Godfather series, but also classic films like Apocalypse Now."

[0117] "Coppola's family also has several members working in the film industry, including his son Roman Coppola."

[0118] "He is also a renowned film producer who has produced several successful films."

[0119] Loss1 is similar to the original MDR loss but uses denoised positive samples. The few negative examples mined by the BM25 algorithm are used as negative samples.

[0120] The positive example of loss2 is the concatenation of two denoised positive samples. The BM25 algorithm uses the negative examples used in loss1 as negative samples and the denoised residual \(Pa-\) samples combined with other samples in the batch as negative examples of loss2.

[0121] Reasoning process:

[0122] When using the model for reasoning, there is no need to perform steps such as denoising during training. The trained model can be directly used to embed questions and candidate documents, and calculate similarities for document retrieval.

Claims

1. A data augmentation and training method for a multi-hop question-answering retrieval model, characterized in that The method comprises the following steps: Step 1: Obtain a multi-hop question answering dataset, which consists of complex multi-hop questions and their corresponding document sets, including first-hop retrieval documents, second-hop retrieval documents, and other related documents; Step 2: Perform positive example denoising on the first-hop document and the second-hop document in the document collection, and obtain the denoised documents as new positive examples for model training, and the remaining parts of the documents as supplementary negative examples for training; Step 3: Input the data obtained in step 2 into the multi-hop question-answering pre-trained language model for training.

2. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 1, characterized in that The specific steps of step 2 are as follows: Step 21: extract proposition clauses from the first-hop document and the second-hop document in the document set using a large language model and prompt engineering technology to obtain a corresponding proposition clause candidate set; Step 22: According to the multi-hop problem, the semantic similarity score of each proposition clause in the candidate set is calculated using BERT Score and sorted in descending order to obtain a sorted proposition clause candidate set; Step 23: Set the retention percentage α, retain the TopK proposition clauses, concatenate the TopK proposition clauses in the original order, and obtain the denoised positive example text.

3. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 2, characterized in that The specific steps of step 22 are as follows: Step 221, calculate word embedding: Perform text word embedding on each proposition clause text N of the proposition clause set corresponding to the multi-hop question text C and the positive example document to obtain vector sequences emb_C and emb_N respectively; Step 222, calculate the cosine similarity matrix: Calculate the cosine similarity between each word embedding in the multi-hop question text and the proposition clause text to obtain the similarity matrix S, which is expressed as shown in the following formula: Among them, S ij Indicates the cosine similarity between the embedding of the i-th word in the multi-hop question text and the embedding of the j-th word in the proposition clause text; cos_similarity(emb_C i ,emb_N J ) represents the calculation of two word embedding vectors emb_C i and emb_N J cosine similarity of; emb_C i Represents the word embedding vector of the i-th word in the multi-hop question text; emb_N J Represents the word embedding vector of the jth word in the proposition clause text; In emb_C i *emb_N j represents the dot product of two word embedding vectors, ||emb_C i || and ||emb_N j || respectively represent the modulus of the word embedding vector of the i-th word in the multi-hop question text and the modulus of the word embedding vector of the j-th word in the proposition clause text; Step 223, calculate precision, recall and harmonic mean F1: For each word in the multi-hop question text, find the word with the highest similarity in the proposition clause text and calculate the average similarity as precision, which is expressed as follows: Where P represents precision, i.e., accuracy, |C| represents the number of words in the multi-hop question text, c∈C represents traversing each word c in the multi-hop question text, r∈N represents traversing each word r in the proposition clause text, s(c,r) represents the similarity between word c in the multi-hop question text and word r in the proposition clause text, and max r∈R s(c,r) means that for a specific word c in the multi-hop question text, the value with the highest similarity among all the words in the proposition clause text is found, Σ c∈C max r∈ R s(c,r) means that for each word in the multi-hop question text, find the word with the highest similarity in the proposition clause text, and add up these maximum similarity values; For each word in the proposition clause text, find the word with the highest similarity in the multi-hop question text and calculate the average similarity as recall, which is expressed as follows: Where R represents recall, i.e., recall rate, |N| represents the number of words in the proposition clause text, c∈N represents traversing each word c in the proposition clause text, and r∈C represents traversing each word r in the multi-hop question text; The final BERTScore is the harmonic mean of P and R, and its expression is shown in the following formula: Among them, F1 represents the final BERTScore; After calculating the semantic similarity scores between all candidate clauses and the multi-hop question, these scores are sorted in descending order.

4. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 2, characterized in that The specific steps of step 23 are as follows: Step 231, define the retention percentage α: the retention percentage α is a value between 0 and 1, which is used to determine the proportion of clauses that should be retained in the sorted candidate set of propositional clauses; Step 232, calculate TopK: the value of TopK is calculated by multiplying the retention percentage α by the total number of candidate clauses; Step 233: splice in the original word order to obtain the denoised positive example text.

5. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 4, characterized in that The step 233 specifically includes the following contents: 1) Select the TopK proposition clauses: According to the calculated TopK values, select the topK clauses from the sorted proposition clause candidate set; 2) Splicing in original order: After selecting the top K clauses, they need to be spliced ​​in the order in which they appear in the original document; 3) Obtaining the denoised positive example text: After the processing in the above step 2), a denoised positive example text is obtained which is composed of the TopK proposition clauses concatenated in the original order.

6. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 1, characterized in that The specific steps of step 3 are as follows: Step 31: Based on the Bert text embedding model, the first-hop document, the second-hop document, and the negative example document are converted into corresponding text embedding vectors; Step 32: Design a multi-hop question-answering pre-trained language model Step 321, selecting the MDR model as the baseline model; Step 322: introduce a positive example denoising strategy based on proposition clauses. Proposition clauses are defined as atomic expressions in the text. Each proposition contains an independent fact or information point. The process of positive example denoising is as follows: first, a proposition extraction model is used to identify and extract multiple proposition sets from a single text paragraph. Then, for each proposition in the set, the semantic relevance score of the proposition with the given question q and the related supporting documents is calculated; Step 323, introduce a hyperparameter a to control the ratio between the number of retained propositions and the total number of propositions. According to the hyperparameter a, propositions within this ratio are selected and retained. These propositions will be reassembled in the order they appear in the original text to maintain the coherence of the text. In addition, in order to avoid losing propositions containing answer information, propositions related to the answer are also included in the retained set. The remaining proposition set = P\ is used to generate additional negative samples to further enrich the training data set. P is the proposition set obtained after a single document is processed by the proposition extraction model. Step 33: input the data obtained in step 31 into the multi-hop question-answering pre-trained language model designed in step 32 for training.

7. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 6, characterized in that The specific steps of step 31 are as follows: Step 311, converting the multi-hop question text into a question embedding vector; Step 312: The first-hop document and the second-hop document are converted into corresponding text embedding vectors; Step 313: The negative example documents are converted into corresponding embedding vectors.

8. The data enhancement and training method for a multi-hop question-answering retrieval model according to claim 6, characterized in that In step 33, during the model training process, the relationship between the question embedding vector and the related text embedding vector is optimized by calculating the following contrast loss: L total =L1+λL2 Among them, L1 represents the loss function using denoised positive samples, which is similar to the loss of the original MDR but uses denoised positive samples; L2 represents an additional loss function that includes negative samples formed from discarded positive propositions and previously unused hard negative samples; L total represents the overall loss function; represents the feature mapping function, which converts the input into its feature representation; T represents the temperature hyperparameter, which is used to scale the dot product result and affect the behavior of the softmax function; λ represents the hyperparameter, which balances the influence of the two loss functions of L1 and L2; x i It represents a query, i.e. a question; represents the denoised positive sample; x k represents the negative samples in the batch, including hard negative samples from BM25, B is the total number of batches; x n represents the negative samples formed from the discarded positive propositions and the previously unused hard negative samples, and M is the total number of negative samples.

Images