Iterative multi-hop question and answer method based on large language model, medium and equipment

Abstract

The invention discloses an iterative multi-hop question answering method based on a large language model, a medium and equipment, and the method comprises the following steps: decomposing a to-be-solved multi-hop question into a plurality of sub-questions through the large language model, and indicating the dependency relationship among the sub-questions through a specific identifier; for atomic questions in the decomposition result, similar paragraphs are extracted in combination with an information retrieval model, and the similar paragraphs are input into a large language model to obtain answers; performing question reconstruction based on the atom question with the obtained answer and the answer, if a question reconstruction result is a question which still needs to be solved, taking the question as a starting point of the next round of iteration, and continuously and repeatedly iterating until a final answer is obtained; and for the same to-be-solved multi-hop problem, after multiple query chains are generated, selecting an optimal scheme from the query chains by adopting a scoring mechanism containing a trainable module to serve as a final question and answer result. Compared with the prior art, the method has the advantages that the accuracy of multi-hop questions and answers is remarkably improved by iteratively decomposing and reconstructing problems and combining a refined query chain selection mechanism.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to an iterative multi-hop question-answering method, medium and device based on a large language model. Background Art

[0002] As an important application of text intelligence, multi-document question answering is committed to building an intelligent system that can integrate key information from multiple documents and automatically answer complex questions. Compared with single-document question answering, the key challenge of multi-document question answering is that it requires the model to be able to process long text sequences and span multiple documents, identify and associate scattered key information, and perform multi-step reasoning to obtain accurate answers.

[0003] In recent years, large language models have made significant progress in the field of natural language processing, but their long input processing capabilities and multi-step reasoning capabilities are still limited when dealing with multi-document question answering. Recent research trends are to develop comprehensive solutions that can combine the advantages of retrieval enhancement and large language models. These solutions quickly locate key documents and paragraphs through intelligent retrieval strategies, and then use the deep semantic understanding capabilities of large language models to perform refined information extraction and multi-step reasoning. However, multi-document question answering often involves complex multi-hop problems. This scenario requires the model to aggregate key information from each part and perform multi-step reasoning. Direct retrieval is difficult to extract enough effective information.

[0004] Therefore, how to improve the accuracy of retrieval results in multi-hop problem scenarios and increase the coverage of key information is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and to provide an iterative multi-hop question answering method, medium and device based on a large language model.

[0006] The present invention achieves decoupling between sub-problems and answering multi-hop complex problems through the "problem decomposition-problem solving-problem reconstruction cycle", and at the same time adopts a scoring mechanism with a trainable module to select potential optimal results from multiple reasoning chains, effectively solving multi-document complex question and answering. Inspired by the powerful contextual learning ability of large-scale language-based models, the present invention achieves decoupling between sub-problems by adding identifiers to examples, and on this basis combines the retrieval enhancement strategy with the "decomposition-solving-reconstruction" mechanism to successfully answer multi-hop complex problems. The present invention also designs a trainable selector that uses the valuable information generated during the cycle to select the most reliable solution.

[0007] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an iterative multi-hop question answering method based on a large language model, comprising the following steps:

[0009] S1: Obtain a multi-hop problem to be solved, and continuously iterate to solve the multi-hop problem to be solved. In the current iteration round, the problem to be decomposed obtained in the previous iteration round, the decomposition examples of problems similar to the problem to be decomposed, and the prompt words describing the task are input into the large language model together, and the context learning ability of the large language model is used to decompose the problem to be decomposed to obtain the decomposition result of the problem to be decomposed in the current iteration round; the prompt words describing the task use identifiers to indicate the dependency relationship between non-independent problems in the decomposition result and each atomic problem, and each identifier represents the answer to each atomic problem;

[0010] S2: taking each atomic problem in the decomposition result as an atomic problem to be solved, extracting paragraphs similar to the atomic problem to be solved as similar paragraphs using an information retrieval model, inputting the atomic problem to be solved and the similar paragraphs into the large language model, and guiding the large language model to output the answer to the atomic problem to be solved through prompt words for problem solving;

[0011] S3: The atomic questions with answers in the decomposition results are respectively combined with their corresponding answers to form atomic question-answer pairs, and the obtained atomic question-answer pairs and the problem to be decomposed obtained in the previous iteration round are input into the large language model together, and the prompt words of the problem reconstruction are used to guide the large language model to reconstruct the problem to obtain the problem reconstruction result; if the problem reconstruction result indicates that there is no problem to be solved, then the answer is output and used as the final answer to the multi-hop problem to be solved; if the problem reconstruction result is a problem that still needs to be solved, it is used as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained;

[0012] S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic questions used to generate the same final answer. After all query chains are obtained, a scoring mechanism containing a trainable module is used to select the query chain with the highest final score from all query chains as the optimal query chain. The final answer corresponding to the optimal query chain is used as the final question-and-answer result of the multi-hop problem to be solved.

[0013] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, in step S1, the non-independent question refers to the Nth atomic question q N The dependency relationship means that if the problem q is not independent B contains the Nth atomic problem q N The identifier of the answer is #N, then the non-independent question qB Depends on the Nth atomic problem q N .

[0015] As a preferred embodiment of the above-mentioned first aspect, in step S1, the process of obtaining problem decomposition examples is: pre-synthesizing an example library containing 100 problem decomposition examples, calculating the cosine similarity between the problem decomposition examples in the example library and the problem to be decomposed obtained in the previous iteration round, and sorting the problem decomposition examples in the example library according to the cosine similarity calculation result, and selecting the three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration round as problem decomposition examples similar to the problem to be decomposed.

[0016] As a preferred embodiment of the first aspect, in step S2, the information retrieval model adopts a Contriever model.

[0017] As a preferred embodiment of the first aspect, in step S4, the specific process of selecting the query chain with the highest final score from all query chains by the scoring mechanism including the trainable module is as follows:

[0018] S41: taking an atomic question in a query chain as an atomic question to be scored, calculating similarity scores between the atomic question to be scored and each paragraph in the reference document through an information retrieval model, and sorting the obtained similarity scores, selecting the top K highest similarity scores from the sorted results of the similarity scores as inputs of a pre-trained trainable module, and using the trainable module to predict a paragraph ranking threshold, and taking all paragraphs before the paragraph ranking threshold in the reference document as paragraphs participating in the calculation;

[0019] S42: For the atomic question to be scored, the average similarity score of the paragraphs involved in the calculation is taken as the alignment score between it and the reference document;

[0020] S43: For any query chain, the average of the alignment scores obtained in the same query chain is used as the final score of the corresponding query chain, and finally the query chain with the highest final score is used as the optimal query chain.

[0021] As a preferred embodiment of the above-mentioned first aspect, the trainable module adopts a multilayer perceptron. During the training process of the trainable module, the trainable module is trained on a training set, wherein a training sample in the training set consists of an atomic question and a corresponding reference document, and the paragraph containing the answer to the atomic question in the reference document of each training sample is used as an evidence paragraph. The paragraph before the evidence paragraph in the reference document of each training sample is marked as a positive example, and the paragraph after the evidence paragraph is marked as a negative example. The label sequence corresponding to the positive example is 1, and the label sequence corresponding to the negative example is 0; the information retrieval model calculates the similarity score between the atomic question in each training sample and each paragraph in the respective reference document, and the similarity score ranking of each paragraph is used as the retrieval result; for the atomic question in each training sample, , concatenate the top K similarity scores with the highest similarity scores in each reference document as the input of the trainable module to obtain a probability score list, normalize the probability score list through the Sigmoid activation function to obtain a normalized probability score list, wherein a value in the probability score list represents the probability of marking a paragraph with the highest similarity score in the reference document as a positive example; after mapping the normalized probability score list, obtain the predicted index of the evidence paragraph, calculate the mean square error loss between the predicted index of the evidence paragraph and the true index of the evidence paragraph, calculate the binary classification loss between the normalized probability score list and the label sequence, add the mean square error loss and the binary classification loss as the total loss, minimize the total loss and update the parameters of the trainable module until convergence.

[0022] As a preferred embodiment of the first aspect, the prediction index rank of the evidence paragraph is calculated as follows:

[0023]

[0024] Where i represents the index of the probability in the normalized probability score list; s i represents the i-th probability in the normalized probability score list; ω represents the first hyperparameter;

[0025] True index of the evidence passage gold rank The calculation method is as follows:

[0026]

[0027] Among them, label i represents the label sequence corresponding to the i-th probability in the normalized probability score list.

[0028] As a preferred embodiment of the first aspect, the total loss The functional form of is as follows:

[0029]

[0030] Among them, [s1,s2,...,s K ] represents a list of normalized probability scores, s1,s2,...,s K They represent the probability of marking the 1st, 2nd, ..., Kth paragraphs as positive examples; labels represents the label sequence; BCELoss([s1,s2, ..., s K ], labels) represents the binary classification loss; α represents the second hyperparameter; MSELoss(rank, gold rank ) represents the mean square error loss.

[0031] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the iterative multi-hop question-answering method based on a large language model as described in any of the schemes of the first aspect above is implemented.

[0032] In a third aspect, the present invention provides a computer electronic device, comprising a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the iterative multi-hop question-answering method based on a large language model as described in any scheme of the first aspect above when executing the computer program.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Compared with the existing technology, the present invention achieves decoupling between sub-problems through the "problem decomposition-problem solving-problem reconstruction cycle", thereby improving the retrieval accuracy and ultimately solving multi-hop complex problems; at the same time, a scoring mechanism with a trainable module is used to select potential optimal results from multiple reasoning chains to further improve the confidence. Compared with direct retrieval, the present invention has achieved significant performance improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0038] Figure 2 A schematic diagram of the steps of obtaining the optimal query chain in the present invention;

[0039] Figure 3 It is a flow chart of the "problem decomposition-problem solving-problem reconstruction" cycle process in the present invention;

[0040] Figure 4 Schematic diagram of the prompt word template used in the present invention (the input language is English). DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0043] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned iterative multi-hop question answering method based on a large language model includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0044] S1: Obtain the multi-hop problem to be solved, and continuously iterate to solve the multi-hop problem to be solved. In the current iteration round, the problem to be decomposed obtained in the previous iteration round, the decomposition examples of problems similar to the problem to be decomposed, and the prompt words describing the task are input into the large language model together, and the context learning ability of the large language model is used to decompose the problem to be decomposed to obtain the decomposition result of the problem to be decomposed in the current iteration round; the identifier #N in the prompt words describing the task indicates that the non-independent problem in the decomposition result points to the Nth atomic problem q N Dependencies, each identifier represents the answer to each atomic question.

[0045] It should be noted that in step S1 of the present invention, the atomic problem refers to a simple single-hop problem, and the answer to this type of sub-problem can be obtained from the reference document. The present invention refers to the problem containing the identifier "#" as a non-independent problem, and the problem that does not depend on any other atomic problem (i.e., the problem that does not contain the identifier "#") is called an independent problem. Furthermore, the non-independent problem is directed to the Nth atomic problem q N The dependency relationship is defined as: if the non-independent problem q B The atomic problem q is included N The identifier of the answer #N is called the non-independent question q B Depends on the Nth atomic problem q N .

[0046] It should be noted that in step S1 of the present invention, the process of obtaining problem decomposition examples is: pre-synthesizing an example library containing 100 problem decomposition examples, calculating the cosine similarity between the problem decomposition examples in the example library and the problem to be decomposed obtained in the previous iteration round, and sorting the problem decomposition examples in the example library according to the cosine similarity calculation result, and selecting the three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration round as problem decomposition examples similar to the problem to be decomposed.

[0047] In step S1 of this embodiment, in order to indicate that the non-independent problem points to the Nth atomic problem q N The context learning is used to make the large language model decompose the complex problem (problem to be decomposed) into a series of sub-problems (decomposition results) according to the carefully selected problem decomposition examples. The list of sub-problems is recorded as Among them, q1 and q2 represent the first and second atomic problems respectively. Represents a sub-problem (non-independent problem) that needs to be inferred from other atomic problems. Figure 3 As shown, question 3 is related to the answers to questions 1 and 2. Unlike the traditional method of decomposing complex questions into parallel simple sub-questions, the present invention effectively draws the non-independent questions in the decomposition results pointing to the Nth atomic question q by instructing the large language model to use the identifier #N to represent the answer to the Nth atomic question. N dependencies and logical order.

[0048] It should also be noted that the specific form of the large language model used in the present invention is not limited. As a preferred implementation of the embodiment of the present invention, the large language model can be selected from Llama2-Chat or GPT-3.5.

[0049] S2: Treat each atomic problem in the decomposition result as an atomic problem to be solved, use the information retrieval model to extract paragraphs similar to the atomic problem to be solved as similar paragraphs, input the atomic problem to be solved and the similar paragraphs into the large language model together, and guide the large language model to output the answer to the atomic problem to be solved through prompt words for problem solving.

[0050] In step S2 of this embodiment, for independent sub-questions that do not refer to answers to any other sub-questions, they are input into the large language model together with similar paragraphs extracted using the retrieval model to obtain answers. Specifically, first, the atomic question to be solved is treated, and the information retrieval (IR) model is used, which is the Contriever model proposed by Izacard G et al. in this embodiment, to identify the top K paragraphs in the reference document that are most relevant to the atomic question to be solved as similar paragraphs ( Figure 3The first K similar passages in the search result are retrieved, and then the retrieved similar passages and the corresponding atomic questions to be solved are fed into the large language model. At this stage, the large language model uses comprehensive information (similar passages, atomic questions to be solved, and prompt words for problem solving) and understanding of natural language to derive a comprehensive answer to the atomic questions to be solved.

[0051] S3: The atomic questions with answers in the decomposition results are combined with their corresponding answers to form atomic question-answer pairs, and the obtained atomic question-answer pairs and the question to be decomposed obtained in the previous iteration round are input into the large language model together, and the prompt words of question reconstruction are used to guide the large language model to reconstruct the question to obtain the question reconstruction result.

[0052] When using atomic question-answer pairs to reconstruct the problem to be decomposed obtained from the previous iteration round, there are two situations for the problem reconstruction results: if the problem reconstruction result indicates that there is no problem to be solved, then the answer is output and used as the final answer to the multi-hop problem to be solved; if the problem reconstruction result is a problem that still needs to be solved, then it is used as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained.

[0053] It should be noted that in step S3 of the present embodiment, question reconstruction is achieved using atomic question-answer pairs. The method of obtaining a complete question using text replacement (identifier #N is directly replaced with the answer) ignores the requirement of seamlessly integrating the information of the atomic question into a non-independent question, and grammatical or contextual inconsistencies may occur after direct replacement. In order to avoid this situation, the present invention utilizes the powerful language understanding ability of the large language model itself to achieve question reconstruction. In addition, considering that not every atomic question in the decomposition result can be answered, the present invention selects atomic questions that can be answered from the decomposition result, and combines these atomic questions and the corresponding answers into atomic question-answer pairs, which are input as part of the large language model. The atomic question-answer pairs are input into the large language model together with the question to be decomposed obtained in the previous iteration round to obtain a problem reconstruction result. The problem reconstruction result may be a problem that still needs to be solved, similar to Figure 3 In the case shown in (i.e., problem 3 obtained after the problem reconstruction phase), this problem that still needs to be solved is used as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained. However, in another case, when all the required information is obtained by solving the atomic problem, the output of the problem reconstruction is the final answer to the multi-hop problem to be solved.

[0054] In step S3, iteration is a key feature of the present invention. The present invention continues the cycle of problem decomposition, problem solving and problem reconstruction until all relevant aspects of the multi-hop problem to be solved are completely solved, and the output of the problem reconstruction stage is the final answer.

[0055] It should also be noted that in the problem decomposition-problem solving-problem reconstruction cycle of the present invention, prompt words for describing the task, prompt words for problem solving, and prompt words for problem reconstruction are used to guide the large language model to complete the corresponding tasks. These prompt word template diagrams are shown in FIG. Figure 4 as shown (input language is English).

[0056] S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic questions used to generate the same final answer. After all query chains are obtained, a scoring mechanism containing a trainable module is used to select the query chain with the highest final score from all query chains as the optimal query chain. The final answer corresponding to the optimal query chain is used as the final question-and-answer result of the multi-hop problem to be solved.

[0057] It should be noted that in step S4, the present invention explores multiple potential solutions. Repeat steps 1 to 3 for the same multi-hop problem to be solved multiple times to obtain multiple final answers. Each final answer generates a query chain consisting of atomic questions, and each query chain represents a solution. The mth query chain can be represented as P m =q m,1 ,q m,2 ,…,q m,l The multiple query chains finally generated can be expressed as P . = {P1,P2,...,P M}, where l represents the total number of atomic questions contained in the mth query chain; q m,1 ,q m,2 ,…,q m,l represent the 1st, 2nd, ..., lth atomic problems in the mth query chain respectively; P1, P2, ..., P M They represent the 1st, 2nd, …, Mth query chains respectively; M represents the number of solutions (query chains).

[0058] It should be noted that the present invention uses a scoring mechanism with a trainable module to select the best possible solution from multiple query chains to obtain the final question-answering result. Figure 2 As shown in FIG, the specific process of selecting the query chain with the highest final score from all query chains by the scoring mechanism containing the trainable module is as follows:

[0059] S41: An atomic question in a query chain is used as an atomic question to be scored, and similarity scores between the atomic question to be scored and each paragraph in the reference document are calculated through the information retrieval model, and the obtained similarity scores are sorted, and the top K highest similarity scores are selected from the sorted results of the similarity scores as inputs of a pre-trained trainable module, and the trainable module predicts the paragraph ranking threshold K i , set the paragraph ranking threshold K in the reference document i All previous paragraphs are considered as participating paragraphs.

[0060] In this embodiment, for each atomic question on the query chain, the information retrieval (IR) model Contriever is used to calculate similarity scores to identify the top K paragraphs most relevant to each atomic question on the query chain. Subsequently, the similarity scores of the top K most relevant paragraphs are concatenated and fed into a pre-trained trainable module, which predicts the paragraph ranking threshold K i .

[0061] It should be noted that, as a preferred implementation of the embodiment of the present invention, the trainable module adopts a multi-layer perceptron (MLP). During the training process of the trainable module:

[0062] A) training the trainable module on a training set, wherein a training sample in the training set consists of an atomic question and a corresponding reference document, taking the paragraph containing the answer to the atomic question in the reference document of each training sample as the evidence paragraph, marking the paragraph before the evidence paragraph in the reference document of each training sample as a positive example, and marking the paragraph after the evidence paragraph as a negative example, the label sequence corresponding to the positive example is 1, and the label sequence corresponding to the negative example is 0.

[0063] B) The information retrieval model calculates the similarity score between the atomic question in each training sample and each paragraph in the respective reference document, and the similarity score ranking of each paragraph is used as the retrieval result. Taking an atomic question in a training sample as an example, the similarity score between it and each paragraph in the reference document can be expressed as follows:

[0064] c t =IR(query, segment t)

[0065] Where IR represents the information retrieval model; query represents an atomic question in a training sample, segment t represents the text segment t in the reference document; c t Represents the similarity score of paragraph t in the reference document.

[0066] C) For each atomic question in the training sample, concatenate the top K similarity scores with the highest similarity scores in the respective reference documents as the input of the trainable module to obtain a probability score list, and normalize the probability score list through the Sigmoid activation function to obtain a normalized probability score list. Among them, a value in the probability score list represents the probability of marking a paragraph with the highest similarity score in the reference document as a positive example. Taking an atomic question in a training sample as an example, concatenate the top K similarity scores with the highest similarity scores in its reference document [c1,c2,...,c K ], input to the trainable module, and obtain the probability score list MLP ([c1,c2,...,c K ]). After applying the Sigmoid activation function, we get a list of normalized probability scores:

[0067] [s1,s2,...,s K ]=σ(MLP([c1,c2,...,c K ]))

[0068] Among them, s1,s2,...,s K They represent the probability of marking the 1st, 2nd, ..., Kth paragraphs as positive examples, indicating the possibility of being ranked no lower than the evidence paragraph in the retrieval results.

[0069] D) After mapping the normalized probability score list, we get the predicted index of the evidence paragraph:

[0070]

[0071] Where i represents the index of the probability in the normalized probability score list. i represents the i-th probability in the normalized probability score list. ω represents the first hyperparameter such that (s i -0.5)×ω is close to 0 or 1 after the Sigmoid activation function to ensure the differentiability. In this embodiment, the first hyperparameter ω is 10 6 .

[0072] E) Calculate the mean square error loss between the predicted index of the evidence paragraph and the true index of the evidence paragraph, calculate the binary classification loss between the normalized probability score list and the label sequence, add the mean square error loss and the binary classification loss as the total loss, minimize the total loss and update the trainable module parameters until convergence.

[0073] In the above process, the true index of the evidence paragraph is gold rank The calculation method is as follows:

[0074]

[0075] Among them, label i represents the label sequence corresponding to the i-th probability in the normalized probability score list.

[0076] Total loss The functional form of is as follows:

[0077]

[0078] Among them, BCELoss([s1,s2,...,s K ], labels) represents the binary classification loss. α represents the second hyperparameter. In this embodiment, the second hyperparameter α is 0.5. MSELoss(rank, gold rank ) represents the mean square error loss.

[0079] S42: For the atomic question to be scored, the average similarity score of the paragraphs involved in the calculation is taken as the alignment score between it and the reference document.

[0080] S43: For any query chain, the average of the alignment scores (average sub-question alignment) obtained in the same query chain is used as the final score of the corresponding query chain, and finally the query chain with the highest final score is used as the optimal query chain.

[0081] In order to better demonstrate the specific implementation and technical effects of the present invention, the iterative multi-hop question-answering method based on a large language model shown in steps S1 to S4 in the above preferred implementation is applied to a specific example.

[0082] Example

[0083] The specific implementation process of the iterative multi-hop question-answering method based on a large language model used in this embodiment is as described above and will not be repeated here. For ease of description, the above-mentioned iterative multi-hop question-answering method based on a large language model is referred to as DSRC-QCS. This embodiment uses the multi-document multi-hop question-answering benchmark introduced in the article LongBench to comprehensively evaluate the performance of the present invention on this task. The LongBench benchmark is adapted from three multi-hop question-answering datasets: HotpotQA, 2WikiMultiHop and MuSiQue. Among them, HotpotQA, as a multi-document question-answering dataset, focuses on testing the model's ability to extract and integrate information from multiple documents to form answers. The dataset consists of two-hop questions carefully designed by native speakers, each of which is associated with two related paragraphs, aiming to examine the model's ability to navigate between different paragraphs and obtain accurate answers. The 2WikiMultiHop dataset generates questions of up to five hops through a carefully planned template system, aiming to prevent the model from taking shortcuts to answer and ensure that the model must perform deep comprehensive reasoning across multiple hops to obtain accurate answers. The MuSiQue dataset covers everything from simple queries to complex reasoning processes with up to four hops, and each question is explained by annotators to prevent the model from taking shortcuts while maintaining the linguistic naturalness of the questions. Each question in the three multi-hop question-answering datasets is accompanied by two to four supporting paragraphs to provide evidence for one-step reasoning, and also contains several distracting paragraphs to increase the complexity of the question. As the first bilingual, multi-task long-context understanding benchmark, LongBench includes 200 samples in the long-context versions of the above three datasets. In order to better meet the actual needs of long-context evaluation, LongBench uses complete Wikipedia paragraphs, including supportive and distracting paragraphs, as context.

[0084] This embodiment uses three instruction fine-tuning language models to verify the effectiveness of the method, namely Alpaca-7B, LLama2-Chat-7B and GPT-3.5-Turbo models. This implementation verifies the performance of the method on three data sets according to the processes described in S1 to S5 above, and the results are shown in Table 1. In Table 1, Cover-EM represents the coverage expectation, that is, the percentage of correct answers that can be covered; EM represents the exact match rate, that is, the percentage of correct answers that are completely matched. The direct method (Directly) uses the original question as a query to directly generate answers. Visconde decomposes the problem into multiple parallel sub-problems to collect relevant information; while ITER-RETGEN iteratively collaborates in retrieval and generation, and it uses the model output of the previous iteration as a specific context to help retrieve more relevant knowledge; Toolformer requires the large language model to autonomously use IR as a tool and call the IR information retrieval model to retrieve relevant information when necessary.

[0085] Table 1 Model performance test results

[0086]

[0087]

[0088] The DSRC-QCS method of the present invention mainly includes two components: "problem decomposition-problem solving-problem reconstruction cycle" and query chain evaluation. In order to verify the effectiveness of each part, this embodiment conducted an ablation experiment on the above three data sets, and the average results of the three data sets are shown in Table 2.

[0089] Table 2 F1 score table of the method of the present invention in the ablation experiment on three large language models

[0090]

[0091] From the results in Table 2, it can be seen that the "problem decomposition-problem solving-problem reconstruction cycle" and query chain evaluation have produced a significant enhancement effect on the multi-document multi-hop question answering performance of the present invention. Through continuous cycles of problem decomposition, independent solving and result reconstruction, the complex multi-hop problem is effectively decomposed into multiple relatively simple sub-problems. This process not only achieves the decoupling between sub-problems, but also follows the implicit logical hierarchy within the multi-hop problem, so that each sub-problem can be solved in a targeted manner. This processing method enables the system to focus on more specific single-hop problems, thereby demonstrating excellent reasoning capabilities. At the same time, the query chain selector module effectively evaluates the credibility of the solution by utilizing the information in the reasoning process, helping the model to screen out the most reasonable solution from multiple candidate answers, further improving the robustness and reliability of the entire reasoning process.

[0092] In summary, the method of the present invention uses a large language model to decompose the multi-hop problem to be solved into multiple atomic problems, and uses specific identifiers to represent the dependency relationship between non-independent problems and each atomic problem. For the atomic problems in the decomposition results, similar paragraphs are extracted in combination with the information retrieval model, and the large language model is input to obtain the answer; the problem is reconstructed based on the atomic problems and answers that have been answered, which serve as the starting point for the next round of iterations; the above steps are repeated until the final answer to the multi-hop problem to be solved is obtained; the above steps are performed multiple times for the same multi-hop problem to be solved, and after generating multiple query chains, the optimal solution is selected from them using a scoring mechanism containing a trainable module as the final question-answering result.

[0093] It can be understood that the iterative multi-hop question answering method based on a large language model described in S1 to S4 above can be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the iterative multi-hop question answering method based on a large language model provided in the above embodiment, which includes a memory and a processor;

[0094] The memory is used to store computer programs;

[0095] The processor is used to implement the iterative multi-hop question-answering method based on a large language model in the above embodiment when executing the computer program.

[0096] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0097] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the iterative multi-hop question-answering method based on a large language model provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the iterative multi-hop question-answering method based on a large language model in the above embodiment can be implemented.

[0098] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0099] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0100] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0101] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An iterative multi-hop question answering method based on a large language model, characterized in that: The following steps are involved: S1: Obtain a multi-hop problem to be solved, and continuously iterate to solve the multi-hop problem to be solved. In the current iteration round, the problem to be decomposed obtained in the previous iteration round, a problem decomposition example similar to the problem to be decomposed, and a prompt word describing the task are input into the large language model, and the context learning ability of the large language model is used to decompose the problem to be decomposed to obtain the decomposition result of the problem to be decomposed in the current iteration round; The prompt words describing the task use identifiers to indicate the dependency relationship between the non-independent questions in the decomposition results and the atomic questions. Each identifier indicates the answer to each atomic question. S2: taking each atomic problem in the decomposition result as an atomic problem to be solved, extracting paragraphs similar to the atomic problem to be solved as similar paragraphs using an information retrieval model, inputting the atomic problem to be solved and the similar paragraphs into the large language model, and guiding the large language model to output the answer to the atomic problem to be solved through prompt words for problem solving; S3: The atomic questions with answers obtained in the decomposition results are respectively combined with their corresponding answers to form atomic question-answer pairs, and the obtained atomic question-answer pairs and the question to be decomposed obtained in the previous iteration round are input into the large language model together, and the prompt words of question reconstruction are used to guide the large language model to reconstruct the question, and obtain the question reconstruction result; If the problem reconstruction result indicates that there is no problem to be solved, then the answer is output and used as the final answer to the multi-hop problem to be solved; if the problem reconstruction result is a problem that still needs to be solved, then it is used as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained; S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic questions used to generate the same final answer. After all query chains are obtained, a scoring mechanism containing a trainable module is used to select the query chain with the highest final score from all query chains as the optimal query chain. The final answer corresponding to the optimal query chain is used as the final question-and-answer result of the multi-hop problem to be solved.

2. The iterative multi-hop question answering method based on a large language model as claimed in claim 1, characterized in that: In step S1, the non-independent question points to the Nth atomic question q N The dependency relationship means that if the problem q is not independent B contains the Nth atomic problem q N The identifier of the answer is #N, then the non-independent question q B Depends on the Nth atomic problem q N .

3. The iterative multi-hop question answering method based on a large language model as claimed in claim 1, characterized in that: In step S1, the process of obtaining problem decomposition examples is as follows: pre-synthesize an example library containing 100 problem decomposition examples, calculate the cosine similarity between the problem decomposition examples in the example library and the problem to be decomposed obtained in the previous iteration round, and sort the problem decomposition examples in the example library according to the cosine similarity calculation result, and select the three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration round as problem decomposition examples similar to the problem to be decomposed.

4. The iterative multi-hop question answering method based on a large language model as claimed in claim 1, characterized in that: In step S2, the information retrieval model adopts the Contriever model.

5. The iterative multi-hop question answering method based on a large language model as claimed in claim 1, characterized in that: In step S4, the specific process of selecting the query chain with the highest final score from all query chains by the scoring mechanism containing the trainable module is as follows: S41: taking an atomic question in a query chain as an atomic question to be scored, calculating similarity scores between the atomic question to be scored and each paragraph in the reference document through an information retrieval model, and sorting the obtained similarity scores, selecting the top K highest similarity scores from the sorted results of the similarity scores as inputs of a pre-trained trainable module, and using the trainable module to predict a paragraph ranking threshold, and taking all paragraphs before the paragraph ranking threshold in the reference document as paragraphs participating in the calculation; S42: For the atomic question to be scored, the average similarity score of the paragraphs involved in the calculation is taken as the alignment score between it and the reference document; S43: For any query chain, the average of the alignment scores obtained in the same query chain is used as the final score of the corresponding query chain, and finally the query chain with the highest final score is used as the optimal query chain.

6. The iterative multi-hop question answering method based on a large language model as claimed in claim 5, characterized in that: The trainable module adopts a multi-layer perceptron. During the training process of the trainable module, the trainable module is trained on a training set. A training sample in the training set consists of an atomic question and a corresponding reference document. The paragraph containing the answer to the atomic question in the reference document of each training sample is used as an evidence paragraph. The paragraph before the evidence paragraph in the reference document of each training sample is marked as a positive example, and the paragraph after the evidence paragraph is marked as a negative example. The label sequence corresponding to the positive example is 1, and the label sequence corresponding to the negative example is 0. The information retrieval model calculates the similarity score between the atomic question in each training sample and each paragraph in the respective reference document, and the similarity score ranking of each paragraph is used as the retrieval result. For each atomic question in the training sample, the respective reference documents are ranked. The first K similarity scores with the highest similarity scores in the document are concatenated as the input of the trainable module to obtain a probability score list, and the probability score list is normalized by a Sigmoid activation function to obtain a normalized probability score list, wherein a value in the probability score list represents the probability of marking a paragraph with the highest similarity score in the reference document as a positive example; after mapping the normalized probability score list, the predicted index of the evidence paragraph is obtained, the mean square error loss between the predicted index of the evidence paragraph and the true index of the evidence paragraph is calculated, the binary classification loss between the normalized probability score list and the label sequence is calculated, the mean square error loss and the binary classification loss are added as the total loss, the total loss is minimized and the parameters of the trainable module are updated until convergence.

7. The iterative multi-hop question answering method based on a large language model as claimed in claim 6, characterized in that: The prediction index rank of the evidence paragraph is calculated as follows: Where i represents the index of the probability in the normalized probability score list; s i represents the i-th probability in the normalized probability score list; ω represents the first hyperparameter; True index of the evidence passage gold rank The calculation method is as follows: Among them, label i represents the label sequence corresponding to the i-th probability in the normalized probability score list.

8. The iterative multi-hop question answering method based on a large language model as claimed in claim 7, characterized in that: Total loss The functional form of is as follows: Among them, [s1,s2,...,s K ] represents a list of normalized probability scores, s1,s2,...,s K They represent the probability of marking the 1st, 2nd, ..., Kth paragraphs as positive examples; labels represents the label sequence; BCELoss([s1,s2, ..., s K ], labels) represents the binary classification loss; α represents the second hyperparameter; MSELoss(rank, gold rank ) represents the mean square error loss.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the iterative multi-hop question answering method based on a large language model as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the iterative multi-hop question answering method based on a large language model as described in any one of claims 1 to 7 when executing the computer program.

Images