Iterative multi-hop question answering method based on large language model, medium and device

Abstract

This invention discloses an iterative multi-hop question answering method, medium, and device based on a large language model. The method includes the following steps: the large language model decomposes the multi-hop question to be solved into multiple sub-questions, and uses specific identifiers to indicate the dependencies between sub-questions; for atomic questions in the decomposition results, similar paragraphs are extracted using an information retrieval model and input into the large language model to obtain the answer; based on the atomic questions and answers that have already been answered, the question is reconstructed; if the reconstructed question is a question that still needs to be solved, it is used as the starting point for the next iteration, and the iteration is repeated until the final answer is obtained; for the same multi-hop question to be solved, multiple query chains are generated, and a scoring mechanism with trainable modules is used to select the optimal solution as the final question answering result. Compared with the prior art, this invention significantly improves the accuracy of multi-hop question answering by iteratively decomposing and reconstructing the question, combined with a refined query chain selection mechanism.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, and particularly relates to an iterative multi-hop question answering method, medium, and device based on a large language model. Background Technology

[0002] Multi-document question answering (MLE) systems, as an important application of text intelligence, aim to build intelligent systems capable of integrating key information from multiple documents and automatically answering complex questions. Compared with single-document question answering, the key challenge of MLE lies in the need for models to be able to handle long text sequences and span multiple documents, identify and associate scattered key information, and perform multi-step reasoning to arrive at accurate answers.

[0003] In recent years, large-scale language models have made significant progress in the field of natural language processing, but their long input processing capabilities and multi-step reasoning abilities remain limited when handling multi-document question answering. Recent research trends focus on developing comprehensive solutions that combine retrieval enhancement with the advantages of large language models. These solutions quickly locate key documents and paragraphs through intelligent retrieval strategies, and then leverage the deep semantic understanding capabilities of large language models for refined information extraction and multi-step reasoning. However, multi-document question answering often involves complex multi-hop problems, requiring models to aggregate key information from various parts and perform multi-step reasoning; direct retrieval alone is insufficient to extract enough effective information.

[0004] Therefore, improving the accuracy of search results and increasing the coverage of key information in multi-hop scenarios are urgent technical problems that need to be solved. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to provide an iterative multi-hop question answering method, medium and device based on a large language model.

[0006] This invention achieves decoupling between sub-problems and solves complex multi-hop questions through a "problem decomposition-problem solving-problem reconstruction loop." It also employs a scoring mechanism with trainable modules to select the potentially optimal result from multiple inference chains, effectively solving complex multi-document question answering. Inspired by the powerful context learning capabilities of large-scale language models, this invention decouples sub-problems by adding identifiers to examples. Based on this, it successfully solves complex multi-hop questions by combining retrieval enhancement strategies with the "decomposition-solving-reconstruction" mechanism. This invention also designs a trainable selector that utilizes valuable information generated during the loop to select the most reliable solution.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides an iterative multi-hop question-answering method based on a large language model, comprising the following steps:

[0009] S1: Obtain the multi-hop problem to be solved, and iteratively solve the multi-hop problem to be solved. In the current iteration, the problem to be decomposed obtained in the previous iteration, decomposed examples of similar problems, and prompts describing the task are input into the large language model. The context learning capability of the large language model is used to decompose the problem to be decomposed, and the decomposition result of the problem to be decomposed in the current iteration is obtained. In the prompts describing the task, identifiers are used to indicate the dependency relationship between non-independent problems and atomic problems in the decomposition result. Each identifier represents the answer to each atomic problem.

[0010] S2: Take each atomic problem in the decomposition result as an atomic problem to be solved, use the information retrieval model to extract paragraphs similar to the atomic problems to be solved as similar paragraphs, input the atomic problems to be solved and the similar paragraphs into the large language model, and guide the large language model to output the answer to the atomic problems to be solved through the prompt words for problem solving;

[0011] S3: Combine the atomic problems with answers obtained from the decomposition results with their corresponding answers to form atomic problem-answer pairs. Input the resulting atomic problem-answer pairs and the problems to be decomposed from the previous iteration into the large language model. Guide the large language model to reconstruct the problems using problem reconstruction prompts to obtain the problem reconstruction results. If the problem reconstruction results indicate that there are no problems to be solved, output the answer and use it as the final answer to the multi-hop problem to be solved. If the problem reconstruction results indicate that there are still problems to be solved, use it as the problem to be decomposed in the next iteration. Repeat the iteration until the final answer to the multi-hop problem to be solved is obtained.

[0012] S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic problems used to generate the same final answer. After obtaining all the query chains, a scoring mechanism with trainable modules is used to select the query chain with the highest final score from all the query chains as the optimal query chain. The final answer corresponding to the optimal query chain is the final question and answer result of the multi-hop problem to be solved.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] As a preferred embodiment of the first aspect mentioned above, in step S1, the non-independent problem refers to the Nth atomic problem q. N Dependency refers to: if the problem q is not independent B It contains the problem q representing the Nth atom. N If the identifier of the answer is #N, then the problem q is not independent.B Depends on the Nth atom problem q N .

[0015] As a preferred embodiment of the first aspect above, in step S1, the process of obtaining problem decomposition examples is as follows: a sample library containing 100 problem decomposition examples is pre-synthesized, the cosine similarity between the problem decomposition examples in the sample library and the problem to be decomposed obtained in the previous iteration is calculated, and the problem decomposition examples in the sample library are sorted according to the cosine similarity calculation results, and the three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration are selected as problem decomposition examples similar to the problem to be decomposed.

[0016] As a preferred embodiment of the first aspect mentioned above, in step S2, the information retrieval model adopts the Contriever model.

[0017] As a preferred embodiment of the first aspect mentioned above, the specific process of selecting the query chain with the highest final score from all query chains by a scoring mechanism containing trainable modules in step S4 is as follows:

[0018] S41: Take an atomic problem in a query chain as the atomic problem to be scored, calculate the similarity score between the atomic problem to be scored and each paragraph in the reference document through the information retrieval model, sort the obtained similarity scores, select the top K highest similarity scores from the similarity score ranking results as the input of the pre-trained trainable module, and the trainable module predicts the paragraph ranking threshold. In the reference document, all paragraphs before the paragraph ranking threshold are taken as the paragraphs to be calculated.

[0019] S42: For the atomic problem of scoring, take the average similarity score of the paragraphs involved in the calculation as its alignment score with the reference document;

[0020] S43: For any query chain, take the average alignment score obtained in the same query chain as the final score of the corresponding query chain, and finally take the query chain with the highest final score as the optimal query chain.

[0021] As a preferred embodiment of the first aspect, the trainable module employs a multilayer perceptron. During the training process of the trainable module, it is trained on a training set. A training sample in the training set consists of an atomic problem and a corresponding reference document. The paragraphs in the reference document of each training sample containing the answer to the atomic problem are designated as evidence paragraphs. Paragraphs in the reference document of each training sample that precede the evidence paragraphs are marked as positive examples, while those that follow are marked as negative examples. The label sequence corresponding to positive examples is 1, and the label sequence corresponding to negative examples is 0. The information retrieval model calculates the similarity score between the atomic problem in each training sample and each paragraph in its respective reference document, and the ranking of the similarity scores of each paragraph is used as the retrieval result. For each training sample's atomic problem... The K highest similarity scores from each reference document are concatenated and used as input to the trainable module to obtain a probability score list. This probability score list is then normalized using a Sigmoid activation function, resulting in a normalized probability score list. Each value in the probability score list represents the probability of labeling the paragraph with the highest similarity score in the reference document as a positive example. After mapping the normalized probability score list, the predicted index of the evidence paragraph is obtained. The mean squared error loss between the predicted and true indices of the evidence paragraph is calculated, as is the binary classification loss between the normalized probability score list and the label sequence. The mean squared error loss and the binary classification loss are summed to obtain the total loss. The parameters of the trainable module are updated by minimizing this total loss until convergence.

[0022] As a preferred approach to the first aspect mentioned above, the predicted index rank of the evidence paragraph is calculated as follows:

[0023]

[0024] Where i represents the index of the probability in the list of normalized probability scores; s i ω represents the i-th probability in the list of normalized probability scores; ω represents the first hyperparameter.

[0025] The true index of the evidence paragraph (gold) rank The calculation method is as follows:

[0026]

[0027] Where, label i Let represent the label sequence corresponding to the i-th probability in the normalized probability score list.

[0028] As the preferred option in the first aspect mentioned above, total loss The function form is as follows:

[0029]

[0030] Where, [s1,s2,...,s K ] represents a normalized list of probability scores, s1, s2, ..., s K s1, s2, ..., sk represent the probabilities of marking the 1st, 2nd, ..., Kth paragraphs as positive examples; labels represent the label sequence; BCELoss([s1, s2, ..., sk ... K ],labels) represents the binary classification loss; α represents the second hyperparameter; MSELoss(rank,gold) rank ) represents the mean square error loss.

[0031] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the iterative multi-hop question-answering method based on a large language model as described in any of the solutions of the first aspect above.

[0032] Thirdly, the present invention provides a computer electronic device, including a memory and a processor;

[0033] The memory is used to store computer programs;

[0034] The processor is configured to, when executing the computer program, implement the iterative multi-hop question-answering method based on a large language model as described in any of the solutions of the first aspect above.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] Compared with existing technologies, this invention achieves decoupling between sub-problems through a "problem decomposition-problem solving-problem reconstruction loop," thereby improving retrieval accuracy and ultimately solving complex multi-hop problems. Simultaneously, it employs a scoring mechanism with trainable modules to select potentially optimal results from multiple inference chains, further enhancing confidence. Compared to direct retrieval, this invention achieves a significant performance improvement. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the steps of the method of the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the steps for obtaining the optimal query chain in this invention;

[0039] Figure 3 This is a flowchart of the "problem decomposition-problem solving-problem reconstruction" cyclic process in this invention;

[0040] Figure 4 This is a schematic diagram of the prompt word template used in this invention (input language is English). Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0042] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0043] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned iterative multi-hop question-answering method based on a large language model includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0044] S1: Obtain the multi-hop problem to be solved, and iteratively solve the multi-hop problem. In the current iteration, input the problem to be decomposed from the previous iteration, similar problem decomposition examples, and task description prompts into the large language model. Utilize the context learning capability of the large language model to decompose the problem, obtaining the decomposition result of the problem to be decomposed in the current iteration. The prompts describing the task use the identifier #N to indicate that the non-independent problem in the decomposition result points to the Nth atomic problem q. N The dependencies are defined such that each identifier represents the answer to each atomic problem.

[0045] It should be noted that in step S1 of this invention, the atomic problem refers to a simple single-hop problem, the answer of which can be obtained from the reference document. This invention refers to problems containing the identifier "#" as non-independent problems, and problems that do not depend on any other atomic problems (i.e., problems that do not contain the identifier "#") as independent problems. Furthermore, the non-independent problem is directed to the Nth atomic problem q. N The dependency relationship is defined as follows: if the problem q is not independent B It includes the atomic problem q N The identifier #N for the answer is called the non-independent problem q. B Depends on the Nth atom problem q N .

[0046] It should be noted that in step S1 of the present invention, the process of obtaining problem decomposition examples is as follows: a sample library containing 100 problem decomposition examples is pre-synthesized, the cosine similarity between the problem decomposition examples in the sample library and the problem to be decomposed obtained in the previous iteration is calculated, and the problem decomposition examples in the sample library are sorted according to the cosine similarity calculation results, and the three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration are selected as problem decomposition examples similar to the problem to be decomposed.

[0047] In step S1 of this embodiment, in order to indicate that the non-independent problem points to the Nth atomic problem q N By leveraging the dependencies and contextual learning, the large language model decomposes a complex problem (the problem to be decomposed) into a series of sub-problems (the decomposition results) according to carefully selected problem decomposition examples. The resulting list of sub-problems is denoted as... Where q1 and q2 represent the first and second atomic problems, respectively. This represents a subproblem that depends on other atomic problems for deduction (a non-independent problem). For example... Figure 3 As shown, the answers to questions 3 are related to those to questions 1 and 2. Unlike traditional methods that decompose complex problems into parallel simple subproblems, this invention effectively plots the non-independent problems in the decomposition results pointing to the Nth atomic problem q by instructing the large language model to use the identifier #N to represent the answer to the Nth atomic problem. N Dependencies and logical order.

[0048] It should also be noted that the specific form of the large language model used in this invention is not limited. As a preferred implementation of this invention, the large language model described above can be Llama2-Chat or GPT-3.5.

[0049] S2: Each atomic problem in the decomposition result is taken as an atomic problem to be solved. The information retrieval model is used to extract paragraphs similar to the atomic problems to be solved as similar paragraphs. The atomic problems to be solved and the similar paragraphs are input into the large language model. The large language model is guided to output the answer to the atomic problems to be solved by prompt words for problem solving.

[0050] In step S2 of this embodiment, for independent sub-problems that do not reference the answers to any other sub-problems, they are input together with similar paragraphs extracted using the retrieval model into the large language model to obtain the answer. Specifically, firstly, for the atomic problem to be solved, an information retrieval (IR) model, specifically the Contriever model proposed by Izacard G et al. in this embodiment, is used to identify the top K paragraphs in the reference document that are most relevant to the atomic problem to be solved as similar paragraphs. Figure 3The first K most similar segments are retrieved, and then the retrieved similar segments and their corresponding atomic problems are fed into the large language model. At this stage, the large language model uses comprehensive information (similar segments, atomic problems to be solved, and clue words for solving the problem) and its understanding of natural language to derive a comprehensive answer to the atomic problems to be solved.

[0051] S3: Combine the atomic questions with their corresponding answers from the decomposition results to form atomic question-answer pairs. Input the resulting atomic question-answer pairs and the questions to be decomposed from the previous iteration into the large language model. Guide the large language model to reconstruct the questions through question reconstruction prompts to obtain the question reconstruction results.

[0052] When using the atomic problem-answer pair to reconstruct the problem to be decomposed from the previous iteration, there are two possible outcomes: if the reconstructed outcome indicates that there are no problems to be solved, then the answer is output and used as the final answer to the multi-hop problem to be solved; if the reconstructed outcome indicates that there is still a problem to be solved, then it is used as the problem to be decomposed in the next iteration, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained.

[0053] It should be noted that in step S3 of this embodiment, atomic problem-answer pairs are used to reconstruct the problem. The method of obtaining the complete problem using text replacement (identifier #N is directly replaced with the answer) ignores the requirement of seamlessly integrating the information of atomic problems into non-independent problems; direct replacement may result in grammatical or contextual inconsistencies. To avoid this, this invention utilizes the powerful language understanding capabilities of the large language model to reconstruct the problem. Furthermore, considering that not every atomic problem in the decomposition result can be answered, this invention selects atomic problems with answers from the decomposition result and forms atomic problem-answer pairs with their corresponding answers, which are then used as input to the large language model. The atomic problem-answer pairs, along with the problem to be decomposed obtained in the previous iteration, are input into the large language model to obtain the problem reconstruction result. The problem reconstruction result may still be a problem requiring a solution, similar to... Figure 3 In the case shown (i.e., problem 3 obtained after the problem reconstruction stage), this problem that still needs to be solved is treated as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained. However, in another case, when all the necessary information is obtained by solving the atomic problem, the output of the problem reconstruction is the final answer to the multi-hop problem to be solved.

[0054] In step S3, iteration is a key feature of this invention. The invention continuously cycles through problem decomposition, problem solving, and problem reconstruction until all relevant aspects of the multi-hop problem to be solved are thoroughly resolved. The output of the problem reconstruction phase is the final answer.

[0055] It should also be noted that in the problem decomposition-problem solving-problem reconstruction cycle of this invention, prompts describing the task, prompts for problem solving, and prompts for problem reconstruction are used respectively to guide the large language model to complete the corresponding tasks. These are illustrated in the prompt template diagram below. Figure 4 As shown (input language is English).

[0056] S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic problems used to generate the same final answer. After obtaining all the query chains, a scoring mechanism with trainable modules is used to select the query chain with the highest final score from all the query chains as the optimal query chain. The final answer corresponding to the optimal query chain is the final question and answer result of the multi-hop problem to be solved.

[0057] It should be noted that in step S4, this invention explores multiple potential solutions. Steps 1-3 are repeated multiple times for the same multi-hop problem to be solved, obtaining multiple final answers. Each final answer corresponds to a query chain composed of atomic problems, and each query chain represents a solution. The m-th query chain can be represented as P. m =q m,1 ,q m,2 ,…,q m,l The resulting multiple query chains can be represented by the formula P. . ={P1,P2,...,P M}, where l represents the total number of atomic problems contained in the m-th query chain; q m,1 ,q m,2 ,…,q m,l Let P1, P2, ..., P be the 1st, 2nd, ..., 1st atomic problems in the m-th query chain, respectively. M These represent the 1st, 2nd, ..., Mth query chains, respectively; M represents the number of solutions (query chains).

[0058] It should be noted that this invention uses a scoring mechanism with trainable modules to select the most likely optimal solution from multiple query chains to obtain the final question-answering result. In step S4 of this invention, as... Figure 2 As shown, the specific process of selecting the query chain with the highest final score from all query chains by a scoring mechanism containing trainable modules is as follows:

[0059] S41: Take an atomic problem in a query chain as the atomic problem to be scored, calculate the similarity score between the atomic problem to be scored and each paragraph in the reference document using the information retrieval model, sort the obtained similarity scores, and select the K highest similarity scores from the sorted results as the input of a pre-trained trainable module. The trainable module then predicts the paragraph ranking threshold K. i In the reference document, the paragraph ranking threshold K is set. i All preceding paragraphs are included in the calculation.

[0060] In this embodiment, for each atomic question in the query chain, the Information Retrieval (IR) model Contriever is used to identify the top K most relevant paragraphs by calculating similarity scores. Subsequently, the similarity scores of the top K most relevant paragraphs are concatenated and fed into a pre-trained trainable module, which predicts a paragraph ranking threshold K. i .

[0061] It should be noted that, as a preferred implementation of this invention, the trainable module employs a multilayer perceptron (MLP), and during the training process of the trainable module:

[0062] A) Train the trainable module on the training set, wherein a training sample in the training set consists of an atomic problem and a corresponding reference document. The paragraph in the reference document of each training sample that contains the answer to the atomic problem is taken as the evidence paragraph. The paragraph in the reference document of each training sample that is listed before the evidence paragraph is marked as a positive example, and the paragraph that is listed after the evidence paragraph is marked as a negative example. The label sequence corresponding to the positive example is 1, and the label sequence corresponding to the negative example is 0.

[0063] B) The information retrieval model calculates the similarity score between the atomic problem in each training sample and each paragraph in its respective reference document, and ranks the similarity scores of each paragraph as the retrieval results. Taking an atomic problem in a training sample as an example, the similarity score between it and each paragraph in the reference document can be expressed as follows:

[0064] c t =IR(query, segment t)

[0065] Where IR represents the information retrieval model; query represents the atomic problem in a training sample; segment t represents a text segment t in the reference document; c t This represents the similarity score of text segment t in the reference document.

[0066] C) For each atomic problem in a training sample, the top K similarity scores from the respective reference documents are concatenated and used as input to the trainable module to obtain a probability score list. This probability score list is then normalized using the Sigmoid activation function to obtain a normalized probability score list. Each value in the probability score list represents the probability of marking the paragraph with the highest similarity score in the reference document as a positive example. Taking the atomic problem in a training sample as an example, the top K similarity scores from its reference documents are concatenated [c1, c2, ..., c...]. K The input is fed into the trainable module to obtain a list of probability scores of shape (1,K) MLP([c1,c2,...,c...). K After applying the Sigmoid activation function, a normalized list of probability scores is obtained:

[0067] [s1,s2,...,s K ]=σ(MLP([c1,c2,...,c K ]))

[0068] Where s1, s2, ..., s K These represent the probabilities of marking paragraphs 1, 2, ..., K as positive examples, indicating the likelihood that a paragraph will rank at least as high as the evidence paragraph in the search results.

[0069] D) After mapping the normalized probability score list, the predicted index of the evidence paragraph is obtained:

[0070]

[0071] Where i represents the index of the probability in the list of normalized probability scores. i Let represent the i-th probability in the normalized probability score list. ω represents the first hyperparameter, such that (s... i -0.5)×ω approaches 0 or 1 after passing through the Sigmoid activation function to ensure differentiability. In this embodiment, the first hyperparameter ω is set to 10. 6 .

[0072] E) Calculate the mean squared error loss between the predicted index of the evidence paragraph and the true index of the evidence paragraph, calculate the binary classification loss between the normalized probability score list and the label sequence, add the mean squared error loss and the binary classification loss as the total loss, minimize the total loss to update the parameters of the trainable module until convergence.

[0073] In the above process, the true index of the evidence paragraph is gold. rank The calculation method is as follows:

[0074]

[0075] Where, label i Let represent the label sequence corresponding to the i-th probability in the normalized probability score list.

[0076] Total loss The function form is as follows:

[0077]

[0078] Wherein, BCELoss([s1,s2,...,s K The binary classification loss is represented by `[,labels]`. α represents the second hyperparameter, which is 0.5 in this embodiment. `MSELoss(rank,gold)` rank ) represents the mean square error loss.

[0079] S42: For atomic problems in scoring, take the average similarity score of the paragraphs involved in the calculation as its alignment score with the reference document.

[0080] S43: For any query chain, take the average alignment score (average subproblem alignment) obtained in the same query chain as the final score of the corresponding query chain, and finally take the query chain with the highest final score as the optimal query chain.

[0081] To better demonstrate the specific implementation and technical effects of the present invention, the iterative multi-hop question-answering method based on a large language model shown in steps S1 to S4 of the above preferred implementation is applied to a specific example.

[0082] Example

[0083] The implementation process of the iterative multi-hop question answering method based on a large language model used in this embodiment is as described above and will not be repeated here. For ease of description, the above-mentioned iterative multi-hop question answering method based on a large language model is referred to as DSRC-QCS. This embodiment uses the multi-document multi-hop question answering benchmark introduced in the LongBench paper to comprehensively evaluate the performance of the present invention on this task. The LongBench benchmark is modified from three multi-hop question answering datasets: HotpotQA, 2WikiMultiHop, and MuSiQue. Among them, HotpotQA, as a multi-document question answering dataset, focuses on testing the model's ability to extract and integrate information from multiple documents to form an answer. This dataset consists of two-hop questions carefully designed by native speakers, with each question associated with two relevant paragraphs, aiming to examine the model's ability to navigate between different paragraphs and arrive at an accurate answer. The 2WikiMultiHop dataset generates up to five-hop questions through a carefully curated template system, aiming to prevent the model from taking shortcuts and ensure that the model must perform deep comprehensive reasoning across multiple hops to arrive at an accurate answer. The MuSiQue dataset covers complex reasoning processes ranging from simple queries to up to four hops. Each question is annotated to prevent the model from taking shortcuts while maintaining the naturalness of the question language. Each question in the three multi-hop question-answering datasets is accompanied by two to four supporting paragraphs, providing empirical evidence for one-step reasoning, while also including several distracting paragraphs to increase the complexity of the questions. LongBench, as the first bilingual, multi-task long-context understanding benchmark, includes long-context versions of the above three datasets, each containing 200 samples. To better align with the practical needs of long-context evaluation, LongBench uses complete Wikipedia paragraphs, including both supporting and distracting paragraphs, as context.

[0084] This embodiment uses three instruction-tuned language models to verify the effectiveness of the method: Alpaca-7B, LLama2-Chat-7B, and GPT-3.5-Turbo models. Following the procedures described in S1-S5 above, this embodiment verifies the performance of the method on three datasets, and the results are shown in Table 1. In Table 1, Cover-EM represents the expected coverage rate, i.e., the percentage of correct answers covered; EM represents the exact match rate, i.e., the percentage of correct answers completely matched. The Directly method uses the original question as the query to directly generate the answer. Visconde decomposes the question into multiple parallel sub-questions to collect relevant information; while ITER-RETGEN iteratively and collaboratively retrieves and generates information, using the model output of the previous iteration as a specific context to help retrieve more relevant knowledge; Toolformer requires the large language model to autonomously use IR as a tool, calling the IR information retrieval model to retrieve relevant information when necessary.

[0085] Table 1 Model performance test results

[0086]

[0087]

[0088] The DSRC-QCS method of this invention mainly comprises two components: a "problem decomposition-problem solving-problem reconstruction loop" and query chain evaluation. To verify the effectiveness of each component, ablation experiments were conducted on the three datasets mentioned above, and the average results for the three datasets are shown in Table 2.

[0089] Table 2. F1 scores of the ablation experiments conducted on three large language models using the method of the present invention.

[0090]

[0091] As can be seen from the results in Table 2, both the "problem decomposition-problem solving-problem reconstruction loop" and query chain evaluation significantly enhance the multi-document, multi-hop question-answering performance of this invention. Through continuous iterative problem decomposition, independent solving, and result reconstruction, the complex multi-hop problem is effectively decomposed into several relatively simple sub-problems. This process not only decouples the sub-problems but also follows the implicit logical hierarchy of the multi-hop problem, ensuring that each sub-problem is addressed specifically. This approach allows the system to focus on more specific single-hop problems, thus demonstrating superior reasoning capabilities. Simultaneously, the query chain selector module effectively evaluates the credibility of solutions by utilizing information from the reasoning process, helping the model select the most reasonable solution from multiple candidate answers, further improving the robustness and reliability of the entire reasoning process.

[0092] In summary, the method of this invention utilizes a large language model to decompose the multi-hop problem into multiple atomic problems, and uses specific identifiers to represent the dependencies between non-independent problems and each atomic problem. For the atomic problems in the decomposition result, similar paragraphs are extracted using an information retrieval model and input into the large language model to obtain the answer; based on the atomic problems and answers obtained, the problem is reconstructed as the starting point for the next iteration; the above steps are repeated until the final answer to the multi-hop problem is obtained; the above steps are performed multiple times on the same multi-hop problem to generate multiple query chains, and a scoring mechanism with trainable modules is used to select the optimal solution as the final question-answering result.

[0093] It is understood that the iterative multi-hop question-answering method based on a large language model described in S1 to S4 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the iterative multi-hop question-answering method based on a large language model provided in the above embodiments, which includes a memory and a processor;

[0094] The memory is used to store computer programs;

[0095] The processor is configured to implement the iterative multi-hop question-answering method based on a large language model in the above embodiments when executing the computer program.

[0096] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0097] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the iterative multi-hop question-answering method based on a large language model provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can implement the iterative multi-hop question-answering method based on a large language model in the above embodiments.

[0098] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0099] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0101] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. An iterative multi-hop question-answering method based on a large language model, characterized in that, Includes the following steps: S1: Obtain the multi-hop problem to be solved, and continuously iterate to solve the multi-hop problem to be solved. In the current iteration round, input the problem to be decomposed obtained in the previous iteration round, the decomposition examples of similar problems to be decomposed, and the prompt words describing the task into the large language model. Use the context learning ability of the large language model to decompose the problem to be decomposed and obtain the decomposition result of the problem to be decomposed in the current iteration round. The prompts describing the task use identifiers to indicate the dependencies between non-independent problems and individual atomic problems in the decomposition results, with each identifier representing the answer to each atomic problem; S2: Take each atomic problem in the decomposition result as an atomic problem to be solved, use the information retrieval model to extract paragraphs similar to the atomic problems to be solved as similar paragraphs, input the atomic problems to be solved and the similar paragraphs into the large language model, and guide the large language model to output the answer to the atomic problems to be solved through the prompt words for problem solving; S3: Combine the atomic problems with answers obtained from the decomposition results with their respective answers to form atomic problem-answer pairs. Input the obtained atomic problem-answer pairs and the problems to be decomposed obtained from the previous iteration into the large language model. Guide the large language model to reconstruct the problem through prompt words for problem reconstruction to obtain the problem reconstruction result. If the problem reconstruction result indicates that there are no problems to be solved, then the answer is output and used as the final answer to the multi-hop problem to be solved; if the problem reconstruction result is a problem that still needs to be solved, then it is used as the problem to be decomposed in the next iteration round, and the iteration is repeated until the final answer to the multi-hop problem to be solved is obtained. S4: Solve the same multi-hop problem multiple times to obtain multiple final answers. A query chain is formed by atomic problems used to generate the same final answer. After obtaining all the query chains, a scoring mechanism with trainable modules is used to select the query chain with the highest final score from all the query chains as the optimal query chain. The final answer corresponding to the optimal query chain is the final question and answer result of the multi-hop problem to be solved. In step S4, the specific process of selecting the query chain with the highest final score from all query chains by the scoring mechanism containing trainable modules is as follows: S41: Take an atomic problem in a query chain as the atomic problem to be scored, calculate the similarity score between the atomic problem to be scored and each paragraph in the reference document through the information retrieval model, sort the obtained similarity scores, select the top K highest similarity scores from the similarity score ranking results as the input of the pre-trained trainable module, and the trainable module predicts the paragraph ranking threshold. In the reference document, all paragraphs before the paragraph ranking threshold are taken as the paragraphs to be calculated. S42: For the atomic problem of scoring, take the average similarity score of the paragraphs involved in the calculation as its alignment score with the reference document; S43: For any query chain, take the average alignment score obtained in the same query chain as the final score of the corresponding query chain, and finally take the query chain with the highest final score as the optimal query chain. The trainable module employs a multilayer perceptron. During training, the trainable module is trained on a training set. A training sample in the training set consists of an atomic problem and a corresponding reference document. The paragraph in the reference document of each training sample containing the answer to the atomic problem is designated as an evidence paragraph. Paragraphs in the reference document preceding the evidence paragraphs are labeled as positive examples, while paragraphs following the evidence paragraphs are labeled as negative examples. The label sequence for positive examples is 1, and the label sequence for negative examples is 0. The information retrieval model calculates the similarity score between the atomic problem in each training sample and each paragraph in its respective reference document. The ranking of the similarity scores for each paragraph is used as the retrieval result. For each atomic problem in a training sample, the corresponding reference document... The top K similarity scores with the highest similarity scores in the document are concatenated and used as input to the trainable module to obtain a probability score list. This probability score list is then normalized using a sigmoid activation function to obtain a normalized probability score list. Each value in the probability score list represents the probability of labeling the paragraph with the highest similarity score in the reference document as a positive example. After mapping the normalized probability score list, the predicted index of the evidence paragraph is obtained. The mean squared error loss between the predicted index and the true index of the evidence paragraph is calculated. A binary classification loss between the normalized probability score list and the label sequence is also calculated. The mean squared error loss and the binary classification loss are added together to obtain the total loss. The parameters of the trainable module are updated by minimizing this total loss until convergence.

2. The iterative multi-hop question answering method based on a large language model as described in claim 1, characterized in that, In step S1, the non-independent problem points to the Nth atomic problem. Dependency refers to: if not independent problems It includes the problem representing the Nth atom. If the answer is identified by the identifier #N, then it is not an independent problem. Dependence on the Nth atom problem .

3. The iterative multi-hop question answering method based on a large language model as described in claim 1, characterized in that, In step S1, the process of obtaining problem decomposition examples is as follows: a sample library containing 100 problem decomposition examples is pre-synthesized, the cosine similarity between the problem decomposition examples in the sample library and the problem to be decomposed obtained in the previous iteration is calculated, and the problem decomposition examples in the sample library are sorted according to the cosine similarity calculation results. The three problem decomposition examples with the highest similarity to the problem to be decomposed obtained in the previous iteration are selected as problem decomposition examples similar to the problem to be decomposed.

4. The iterative multi-hop question-answering method based on a large language model as described in claim 1, characterized in that, In step S2, the information retrieval model adopts the Contriever model.

5. The iterative multi-hop question answering method based on a large language model as described in claim 1, characterized in that, Predicted index of evidence paragraphs The calculation method is as follows: ； in, Indicates the index of the probability in the list of normalized probability scores; The first in the list of normalized probability scores One probability; Indicates the first hyperparameter; True Index of Evidence Paragraphs The calculation method is as follows: ； in, The first in the list of normalized probability scores represents the... The label sequence corresponding to each probability.

6. The iterative multi-hop question-answering method based on a large language model as described in claim 5, characterized in that, Total loss The function form is as follows: ； in, Represents a list of normalized probability scores. They respectively represent the first The probability of a paragraph being marked as a positive example; Represents a label sequence; This represents the binary classification loss; Indicates the second hyperparameter; This represents the mean square error loss.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the iterative multi-hop question-answering method based on a large language model as described in any one of claims 1 to 6.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the iterative multi-hop question-answering method based on a large language model as described in any one of claims 1 to 6 when executing the computer program.

Images