Reasoning method, device and storage medium for text answers

Abstract

The present application provides a method, device and storage medium for reasoning about text answers, the method comprising: inputting a first text into a language model, obtaining multiple candidate answers output by the language model; the first text is a text to be recognized; determining a verification score corresponding to each candidate answer with each candidate answer as a condition; the verification score represents the confidence of the candidate answer; and outputting the candidate answer corresponding to the highest verification score as the reasoning answer of the first text. The method, device and storage medium for reasoning about text answers provided in the present application replace the original condition with the answer generated by the language model, and then input it into the language model again, thereby verifying the answer. This self-verification method improves the robustness and accuracy of the output results of the language model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device and storage medium for reasoning about text answers. Background Art

[0002] Recently, with the promotion of Chain-of-thoughts (CoT), large language models (LLM), such as Generative Pre-trained Transformer3 (GPT-3), have shown strong reasoning capabilities in natural language processing tasks such as arithmetic, common sense, and logical reasoning. Simply put, CoT allows LLM to simulate the human thinking process, helping large language models generate a reasoning path, decomposing complex reasoning problems into multiple simple steps, rather than just a final answer, thereby enhancing the model's reasoning ability.

[0003] While CoT enables large models to solve complex reasoning tasks, it is highly sensitive to single errors and vulnerable to error accumulation. Multi-step prompts and multi-token predictions are required in CoT, which may lack robustness when using autoregressive mechanisms to handle complex reasoning tasks. If a tiny error occurs, it will change the meaning bias of the entire sentence, leading to an incorrect answer. This is particularly problematic when using CoT to solve multi-step precise reasoning. Summary of the invention

[0004] The embodiments of the present application provide a method, device and storage medium for reasoning about text answers, so as to solve the technical problem of low robustness in the related art when reasoning about text answers.

[0005] In a first aspect, the present invention provides a method for reasoning about a text answer, comprising:

[0006] Input a first text into a language model, and obtain a plurality of candidate answers output by the language model; the first text is a text to be recognized;

[0007] Determine a verification score corresponding to each candidate answer based on each candidate answer as a condition; the verification score indicates the confidence of the candidate answer;

[0008] The candidate answer corresponding to the highest verification score is output as the inference answer of the first text.

[0009] In some embodiments, determining the verification score corresponding to each candidate answer based on each candidate answer as a condition includes:

[0010] Using a target candidate answer as a condition to replace a target condition in a first condition set included in the first text, and determining at least one second condition set; the target candidate answer is one of the candidate answers;

[0011] Based on each second condition set, a second text is generated; the questions included in the second text are the questions corresponding to the target condition;

[0012] Input a second text into the language model, and obtain a second answer output by the language model;

[0013] A verification score corresponding to the target candidate answer is determined based on the second answer.

[0014] In some embodiments, determining a verification score corresponding to the target candidate answer based on the second answer includes:

[0015] determining whether each second answer is the same as the corresponding target condition;

[0016] When the second answer is the same as the corresponding target condition, the verification score corresponding to the target candidate answer is increased by one unit; when the second answer is different from the corresponding target condition, the verification score corresponding to the target candidate answer remains unchanged.

[0017] In some embodiments, it is determined repeatedly P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

[0018] In some embodiments, generating a second text based on each second condition set includes:

[0019] Determining the target condition based on regular matching;

[0020] Using the question corresponding to the target condition as the question of the second text;

[0021] A second text is generated based on each second condition set and the question corresponding to the target condition.

[0022] In some embodiments, the multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

[0023] In some embodiments, the language model is trained by a large-scale Chinese and English general corpus dataset, and the result is obtained by using one or more of the following datasets as input:

[0024] GSM8K;

[0025] SingleEq;

[0026] AddSub;

[0027] MultiArith;

[0028] AQUA-RAT;

[0029] SVAMP;

[0030] CSQA;

[0031] DU.

[0032] In a second aspect, the present invention provides a text answer reasoning device, comprising:

[0033] An acquisition module, used for inputting a first text into a language model and acquiring a plurality of candidate answers output by the language model; the first text is a text to be recognized;

[0034] A determination module, used to determine a verification score corresponding to each candidate answer based on each candidate answer as a condition; the verification score represents the confidence of the candidate answer;

[0035] The output module is used to output the candidate answer corresponding to the highest verification score as the inference answer of the first text.

[0036] In some embodiments, the determination module includes a first determination submodule, a first generation submodule, a first acquisition submodule, and a second determination submodule, wherein:

[0037] The first determination submodule is used to replace a target condition in a first condition set included in the first text with a target candidate answer as a condition, and determine at least one second condition set; the target candidate answer is one of the candidate answers;

[0038] The first generating submodule is used to generate a second text based on each second condition set; the questions included in the second text are the questions corresponding to the target condition;

[0039] The first acquisition submodule is used to input a second text into the language model to obtain a second answer output by the language model;

[0040] The second determination submodule is used to determine a verification score corresponding to the target candidate answer based on the second answer.

[0041] In some embodiments, the second determination submodule includes a first judgment unit and a first processing unit, wherein:

[0042] The first judgment unit is used to judge whether each second answer is the same as the corresponding target condition;

[0043] The first processing unit is used to increase the verification score corresponding to the target candidate answer by one unit when the second answer is the same as the corresponding target condition; and to keep the verification score corresponding to the target candidate answer unchanged when the second answer is different from the corresponding target condition.

[0044] In some embodiments, the first judgment unit is specifically used to repeatedly judge P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

[0045] In some embodiments, the first generating submodule includes a first determining unit, a second processing unit and a first generating unit:

[0046] The first determination unit is used to determine the target condition based on regular matching;

[0047] The second processing unit is used to use the question corresponding to the target condition as the question of the second text;

[0048] The first generating unit is used to generate a second text based on each second condition set and the question corresponding to the target condition.

[0049] In some embodiments, the multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

[0050] In some embodiments, the language model obtains results by using one or more of the following data sets as input:

[0051] GSM8K;

[0052] SingleEq;

[0053] AddSub;

[0054] MultiArith;

[0055] AQUA-RAT;

[0056] SVAMP;

[0057] CSQA;

[0058] DU.

[0059] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for reasoning about text answers as described above is implemented.

[0060] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for reasoning about text answers as described in any one of the above.

[0061] The text answer reasoning method, device and storage medium provided in the embodiment of the present application replace the original condition with the answer generated by the language model and then input it into the language model again to verify the answer. Through this self-verification method, the robustness of the language model output result is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 It is a flowchart of the reasoning method of the text answer provided in the embodiment of the present application;

[0064] Figure 2 It is a structural schematic diagram of a text answer reasoning device provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] Previous approaches have addressed the above issues by training another verifier to evaluate the correctness of the model output. However, these works have some drawbacks. On the one hand, training the verifier requires a lot of manual annotations and additional fine-tuning models, which limits its wide application in other tasks and fields. On the other hand, the verifier fine-tuned by the language model is not easy to interpret, so it is difficult to evaluate the reliability of the model based on its output scores. Therefore, for LLM, having a verifier that avoids manual annotations and additional training when solving complex reasoning problems can better expand and migrate to other fields and tasks.

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] Figure 1 is a flow chart of the reasoning method of the text answer provided in the embodiment of the present application, such as Figure 1 As shown, the embodiment of the present application provides a method for reasoning about text answers, including:

[0069] Step 101, input a first text into a language model, and obtain a plurality of candidate answers output by the language model; the first text is the text to be recognized.

[0070] Specifically, after inputting the first text into the language model, the input can be augmented with several CoT prompts similar to the original query and then sent to the language model. The language model performs sampling decoding to generate multiple candidates for verification. Among them, the language model can be LLM. The first text can be an inference dataset, such as GSM8K, SingleEq, AddSub, MultiArith, AQUA-RAT, SVAMP, CSQA, and DU datasets.

[0071] Step 102, determining a verification score corresponding to each candidate answer with each candidate answer as a condition; the verification score represents the confidence of the candidate answer.

[0072] Specifically, assuming that the first text is X1, there are R conditions f and a question q in the first text, then X1 can be written as {f1, f2, f3…f R , q}, f and q are all expressed as natural language sentences. To ensure the diversity of different answers, we use sampling decoding to generate K answers y. When the answer y1 needs to be verified, f1 in X1 can be replaced by y1, and f1 can be used as a new question to obtain the second text X2, then X2 can be written as {y1, f2, f3…f R ,q f1}. Then input X2 into the language model to obtain the result z. By judging whether the replaced condition f1 is the same as the answer z of the second text, the verification score of the answer y1 of the first text is determined.

[0073] For example, the first text is that Xiao Zhang has 10 apples and Xiao Li has 8 apples, and the question is how many more apples Xiao Zhang has than Xiao Li. The answer is that Xiao Zhang has two more apples than Xiao Li.

[0074] Then the second text of the verification process can be: Xiao Zhang has two more apples than Xiao Li, Xiao Li has 8 apples, and Xiao Zhang is asked how many apples he has. The answer is that Xiao Zhang has 10 apples.

[0075] The condition for judging whether the first text is replaced, "Xiao Zhang has 10 apples", is the same as the answer to the second text, "Xiao Zhang has 10 apples", so the verification score corresponding to the answer "Xiao Zhang has two more apples than Xiao Li" can be increased by one point.

[0076] Step 103: output the candidate answer corresponding to the highest verification score as the inference answer of the first text.

[0077] Specifically, each candidate answer has a corresponding verification score. The language model can select the candidate answer with the highest verification score among all candidate answers as the result and output this result.

[0078] The text answer reasoning method provided in the embodiment of the present application replaces the original condition with the answer generated by the language model and then inputs it into the language model again to verify the answer. Through this self-verification method, the robustness of the language model output result is improved.

[0079] In some embodiments, determining the verification score corresponding to each candidate answer based on each candidate answer as a condition includes:

[0080] Using a target candidate answer as a condition to replace a target condition in a first condition set included in the first text, and determining at least one second condition set; the target candidate answer is one of the candidate answers;

[0081] Based on each second condition set, a second text is generated; the questions included in the second text are the questions corresponding to the target condition;

[0082] Input a second text into the language model, and obtain a second answer output by the language model;

[0083] A verification score corresponding to the target candidate answer is determined based on the second answer.

[0084] Specifically, assuming that the first text is X1, there are R conditions f and a question q in the first text, then X1 can be written as {f1, f2, f3…f R , q}, f and q are all expressed as natural language sentences. To ensure the diversity of different answers, we use sampling decoding to generate K answers y. When the answer y1 needs to be verified, f1 in X1 can be replaced by y1, and f1 can be used as a new question to obtain the second text X2, then X2 can be written as {y1, f2, f3…f R ,q f1}. Then input X2 into the language model to obtain the result z. Then replace all R conditions in sequence to obtain R z. By judging whether each replaced condition and the answer corresponding to each second text are the same, the verification score of the answer y1 of the first text is determined.

[0085] Among them, the target condition is f1, the target candidate answer is y1, and the first condition set is {f1, f2, f3…f R}, the second condition set is {y1, f2, f3…fR}, the problem corresponding to the target condition is q f1 .

[0086] For example, the first text is that Xiao Zhang has 10 apples and Xiao Li has 8 apples, and the question is how many more apples Xiao Zhang has than Xiao Li. The answer is that Xiao Zhang has 7 more apples than Xiao Li.

[0087] Then the second text of the verification process can be: Xiao Zhang has 7 more apples than Xiao Li, Xiao Li has 8 apples, and Xiao Zhang is asked how many apples he has. The answer is that Xiao Zhang has 15 apples.

[0088] The condition for judging whether the first text is replaced, "Xiao Zhang has 10 apples", is different from the answer to the second text, "Xiao Zhang has 15 apples", so the verification score corresponding to the answer "Xiao Zhang has 7 more apples than Xiao Li" remains unchanged.

[0089] The text answer reasoning method provided in the embodiment of the present application replaces the original condition with the answer generated by the language model and then inputs it into the language model again to verify the answer. Through this self-verification method, the robustness of the language model output result is improved.

[0090] In some embodiments, determining a verification score corresponding to the target candidate answer based on the second answer includes:

[0091] determining whether each second answer is the same as the corresponding target condition;

[0092] When the second answer is the same as the corresponding target condition, the verification score corresponding to the target candidate answer is increased by one unit; when the second answer is different from the corresponding target condition, the verification score corresponding to the target candidate answer remains unchanged.

[0093] Specifically, all R conditions are replaced in sequence with y1 to obtain R z. The verification score of the answer y1 of the first text is determined by judging whether each replaced condition is the same as the answer corresponding to each second text. If the second answer is the same as the corresponding target condition, the verification score corresponding to the target candidate answer is increased by one point. If the second answer is different from the corresponding target condition, the verification score corresponding to the target candidate answer remains unchanged.

[0094] The verification score here is used to count whether the second answer is the same as the corresponding target condition, to determine which candidate answer to output. It can also be counted by the percentage of the same situation in all situations.

[0095] The reasoning method for text answers provided in the embodiment of the present application uses y1 to replace all R conditions in sequence to obtain R z, and then verifies the answers re-predicted based on the language model, thereby improving the reliability and accuracy of the verification.

[0096] In some embodiments, it is determined repeatedly P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

[0097] Specifically, due to the limited performance of the language model itself, during the self-verification process, a single decoding will cause deviations in the verification results due to randomness, making it difficult to ensure a more accurate verification score. To solve this problem, the sampling decoding process will be repeated P times, so that the verification score can more accurately reflect the model's confidence in a given conclusion.

[0098] For example, when P is 10, whether each second answer is the same as the corresponding target condition will be judged 10 times.

[0099] The text answer reasoning method provided in the embodiment of the present application reduces the deviation of the verification result caused by randomness and improves the accuracy of the verification score through the process of repeated sampling and decoding.

[0100] In some embodiments, generating a second text based on each second condition set includes:

[0101] Determining the target condition based on regular matching;

[0102] Using the question corresponding to the target condition as the question of the second text;

[0103] A second text is generated based on each second condition set and the question corresponding to the target condition.

[0104] Specifically, due to the diversity of questions, it is difficult to balance the need for coherence and factual consistency between questions and answers in actual operations. Therefore, the value of the target condition can be found through regular matching, and the value of the target condition can be masked as a question, and the reasoning process and answer for the masked original condition can be generated in sequence using the few-sample prompting technology.

[0105] For example, the first text is that Xiao Zhang has 10 apples and Xiao Li has 8 apples, and the question is how many more apples Xiao Zhang has than Xiao Li. The answer is that Xiao Zhang has 7 more apples than Xiao Li.

[0106] Then the second text of the verification process can be: Xiao Zhang has N apples, Xiao Li has 8 apples, Xiao Zhang has 7 more apples than Xiao Li, what is N.

[0107] The text answer reasoning method provided in the embodiment of the present application uses regular matching to cover the numerical value in the original condition, and utilizes the few-sample prompting technology to sequentially generate the reasoning process and answer for the covered original condition, thereby improving the robustness of the language model output result.

[0108] In some embodiments, the multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

[0109] Specifically, after inputting the first text into the language model, the language model is required to output multiple different candidate answers. The input can be increased by using several CoT prompts similar to the original query and then sent to the language model. The language model then performs sampling decoding to generate multiple candidate answers for verification.

[0110] The text answer reasoning method provided in the embodiment of the present application uses CoT prompts to increase input, so that the language model can generate multiple candidate answers for verification, thereby improving the robustness of the language model output results.

[0111] In some embodiments, the language model is trained by a large-scale Chinese and English general corpus dataset, and the result is obtained by using one or more of the following datasets as input:

[0112] GSM8K;

[0113] SingleEq;

[0114] AddSub;

[0115] MultiArith;

[0116] AQUA-RAT;

[0117] SVAMP;

[0118] CSQA;

[0119] DU.

[0120] Specifically, the GSM8K, SingleEq, AddSub, MultiArith, AQUA-RAT and SVAMP datasets are math problem test sets, the CSQA dataset is a question-answering dataset with multiple options about the real world, and the DU dataset is a dataset from Baidu. By testing the language model with one or more datasets and evaluating the above self-verification methods on multiple datasets, the practicality of the self-verification methods can be determined.

[0121] The text answer reasoning method provided in the embodiment of the present application tests the language model through various data sets and evaluates the self-verification method to determine the practicality of the self-verification method.

[0122] The method in the above embodiment is further described below with reference to a specific example.

[0123] Step 1: LLM reasoners generate candidate answers through thought chain prompts.

[0124] We augment the input with several CoT prompts similar to the original query and send it to the LLM. The LLM then performs sample decoding to generate multiple candidate answers for verification.

[0125] For example, given a question text X, the large language model is prompted with a set of thought chains C. In the few-shot setting, the entire prompt also contains other question CoT prompts. The input X can be further subdivided into X = {f1, f2, f3…f R , q}, where each f is a condition, and there are R f in X, and q is a question, all expressed as natural language sentences. Before generating the final answer y, LLM is required to generate a thought chain according to the prompts of set C.

[0126] To ensure the diversity of different answers, we use sampling decoding to generate K y. Specifically, sampling decoding is a stochastic decoding method that selects the next word by sampling from the probability distribution of possible words at each step. When sampling decoding is used repeatedly, multiple candidate answers can be obtained.

[0127] We use the command “Please change the questions and answers into completed eclarative sentences [q] The answer is [y]” to ask the LLM to change q and y into new declarative sentences. y .

[0128] Step 2: Due to the diversity of questions, it is difficult to balance the need for coherence and factual consistency between questions and answers in practice, so we choose to directly mask the conditions. First, find the value in f through regular matching and rewrite it as "[N]", and add "What is the answer of '[N]'?" at the end of the new question, so as to prompt LLM to indicate the target to be predicted.

[0129] We found that if a given X does not satisfy all necessary conditions for the conclusion, masking only the first condition will have limitations and it is difficult to accurately evaluate its verification score, such as "Dana worked 9 hours on Friday, 10 hours on Saturday, and 3 hours on Sunday. She earns $13 per hour. How much money did Dana earn on weekends?". Since condition 1 (9 hours) does not affect the conclusion, it is difficult to predict correctly. To solve this problem, a method of verifying multiple conditions in sequence can be adopted: replace all condition f that appear in the original X with "[N]" in sequence, and ask LLM to re-predict it to improve the reliability and accuracy of verification.

[0130] Step 3: Due to the limited performance of LLM itself, during the reverse verification process, a single decoding will cause deviations in the verification results due to randomness, making it difficult to ensure a more accurate verification score. To solve this problem, the sampling decoding process will be repeated P times, so that the verification score can more accurately reflect the model's confidence in a given conclusion. The verification score is calculated as follows:

[0131]

[0132] Among them, LLM p (Xf r +f y ) = f r Indicates that when the original condition fr is replaced by the answer, the output of LLM is judged to be different from the replaced original condition f r Are they the same? is an indicator function, indicating that when LLM is satisfied p (Xf r +f y ) = f r In the case of not satisfying LLM p (Xf r +f y ) = f r The result is 0 in this case.

[0133] Afterwards, we can select the one with the highest verification score from the generated K candidate answers as the result:

[0134] Output = argmax(Score k ), k∈[0,K]

[0135] To illustrate the effectiveness of the system, the present invention conducts experimental exploration on eight different reasoning datasets, namely GSM8K, SingleEq, AddSub, MultiArith, AQUA-RAT, SVAMP, CSQA and DU datasets. The methods used for comparison include: direct generation of large language models, CoT prompts and a series of fine-tuning-based methods.

[0136] Table 1 shows the experimental results on multiple datasets. We selected two large language models, GPT-3 and Instruct-GPT, for comparison to illustrate their effectiveness.

[0137] Table 1 Comparison results on different datasets

[0138] ％ GPT-3 CoT GPT-3 SV Instruct-GPT CoT Instruct-GPT SV GSM8K 13.84 13.92 60.81 65.14 SingleEq 60.20 60.61 91.01 93.40 AddSub 58.55 59.07 82.78 86.33 MultiArith 45.85 46.19 96.13 99.15 AQUA-RAT 18.90 27.04 45.30 47.95 SVAMP 38.42 38.96 75.87 76.99 CSQA 46.75 47.68 77.42 77.83 DU 38.72 39.03 65.43 66.57

[0139] It can be found in the table that the performance of the self-verification method (SV) is significantly better than the baseline method.

[0140] In addition, larger language models are better self-verifying reasoners. When the model parameters are small, the self-verification ability of the model is weaker, even worse than the original performance of CoT. The main reason is that the verification of the language model is an emergent ability, which may appear in larger models.

[0141] The reasoning ability of the model using self-verification increases significantly with the number of minority shots. Even with only 2-shot, the self-verification method is more robust with smaller samples (here it has 99.6% of the 8-shot performance, while CoT has only 98.7%). In addition, we find that even with only 4 samples (2 CoT samples + 2 self-verification samples), self-verification outperforms CoT with 8 samples, which highlights the importance of answer verification in the limited data scenario.

[0142] And the more conditions used for verification, the better the reasoning ability relying on self-verification. Observe the effect of using a single conditional mask on six different arithmetic datasets. Since each number in the input of these datasets can be regarded as a condition, we can study the impact of increasing the number of verification conditions. In most experiments, we find that multi-conditional masks perform better than single-conditional masks, and both perform better than the original CoT. These results show that the accuracy of the verification score improves as the number of available conditions increases.

[0143] The text answer reasoning method, device and storage medium provided in the embodiment of the present application replace the original condition with the answer generated by the language model and then input it into the language model again to verify the answer. Through this self-verification method, the robustness of the language model output result is improved.

[0144] Figure 2 is a schematic diagram of a text answer reasoning device provided in an embodiment of the present application, such as Figure 2 As shown, the text answer reasoning device provided in the embodiment of the present application includes an acquisition module 201, a determination module 202, and an output module 203, wherein:

[0145] The acquisition module 201 is used to input a first text into a language model and obtain a plurality of candidate answers output by the language model; the first text is the text to be recognized;

[0146] A determination module 202, for determining a verification score corresponding to each candidate answer based on each candidate answer as a condition; the verification score indicates the confidence of the candidate answer;

[0147] The output module 203 is used to output the candidate answer corresponding to the highest verification score as the inference answer of the first text.

[0148] In some embodiments, the determination module includes a first determination submodule, a first generation submodule, a first acquisition submodule, and a second determination submodule, wherein:

[0149] The first determination submodule is used to replace a target condition in a first condition set included in the first text with a target candidate answer as a condition, and determine at least one second condition set; the target candidate answer is one of the candidate answers;

[0150] The first generating submodule is used to generate a second text based on each second condition set; the questions included in the second text are the questions corresponding to the target condition;

[0151] The first acquisition submodule is used to input a second text into the language model to obtain a second answer output by the language model;

[0152] The second determination submodule is used to determine a verification score corresponding to the target candidate answer based on the second answer.

[0153] In some embodiments, the second determination submodule includes a first judgment unit and a first processing unit, wherein:

[0154] The first judgment unit is used to judge whether each second answer is the same as the corresponding target condition;

[0155] The first processing unit is used to increase the verification score corresponding to the target candidate answer by one unit when the second answer is the same as the corresponding target condition; and to keep the verification score corresponding to the target candidate answer unchanged when the second answer is different from the corresponding target condition.

[0156] In some embodiments, the first judgment unit is specifically used to repeatedly judge P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

[0157] In some embodiments, the first generating submodule includes a first determining unit, a second processing unit and a first generating unit:

[0158] The first determination unit is used to determine the target condition based on regular matching;

[0159] The second processing unit is used to use the question corresponding to the target condition as the question of the second text;

[0160] The first generating unit is used to generate a second text based on each second condition set and the question corresponding to the target condition.

[0161] In some embodiments, the multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

[0162] In some embodiments, the language model obtains a result by using one or more of the following data sets as input:

[0163] GSM8K;

[0164] SingleEq;

[0165] AddSub;

[0166] MultiArith;

[0167] AQUA-RAT;

[0168] SVAMP;

[0169] CSQA;

[0170] DU.

[0171] Specifically, the reasoning device for the above-mentioned text answer provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned text answer reasoning method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0172] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the reasoning method of the text answer, which includes:

[0173] Input a first text into a language model, and obtain a plurality of candidate answers output by the language model; the first text is a text to be recognized;

[0174] Determine a verification score corresponding to each candidate answer based on each candidate answer as a condition; the verification score indicates the confidence of the candidate answer;

[0175] The candidate answer corresponding to the highest verification score is output as the inference answer of the first text.

[0176] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0177] In some embodiments, determining the verification score corresponding to each candidate answer based on each candidate answer as a condition includes:

[0178] Using a target candidate answer as a condition to replace a target condition in a first condition set included in the first text, and determining at least one second condition set; the target candidate answer is one of the candidate answers;

[0179] Based on each second condition set, a second text is generated; the questions included in the second text are the questions corresponding to the target condition;

[0180] Input a second text into the language model, and obtain a second answer output by the language model;

[0181] A verification score corresponding to the target candidate answer is determined based on the second answer.

[0182] In some embodiments, determining a verification score corresponding to the target candidate answer based on the second answer includes:

[0183] determining whether each second answer is the same as the corresponding target condition;

[0184] When the second answer is the same as the corresponding target condition, the verification score corresponding to the target candidate answer is increased by one unit; when the second answer is different from the corresponding target condition, the verification score corresponding to the target candidate answer remains unchanged.

[0185] In some embodiments, it is determined repeatedly P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

[0186] In some embodiments, generating a second text based on each second condition set includes:

[0187] Determining the target condition based on regular matching;

[0188] Using the question corresponding to the target condition as the question of the second text;

[0189] A second text is generated based on each second condition set and the question corresponding to the target condition.

[0190] In some embodiments, the multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

[0191] In some embodiments, the language model obtains a result by using one or more of the following data sets as input:

[0192] GSM8K;

[0193] SingleEq;

[0194] AddSub;

[0195] MultiArith;

[0196] AQUA-RAT;

[0197] SVAMP;

[0198] CSQA;

[0199] DU.

[0200] Specifically, the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented by the method embodiment in which the execution subject is the electronic device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0201] On the other hand, the present application further provides a computer program product, the computer program product comprising a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the text answer reasoning method provided by the above-mentioned method embodiments, the method comprising:

[0202] Input a first text into a language model, and obtain a plurality of candidate answers output by the language model; the first text is a text to be recognized;

[0203] Determine a verification score corresponding to each candidate answer based on each candidate answer as a condition; the verification score indicates the confidence of the candidate answer;

[0204] The candidate answer corresponding to the highest verification score is output as the inference answer of the first text.

[0205] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0206] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0207] It should also be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0208] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0209] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0210] In this application, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reasoning about text answers, characterized in that: include: Input a first text into a language model, and obtain a plurality of candidate answers output by the language model; the first text is a text to be recognized; Using a target candidate answer as a condition to replace a target condition in a first condition set included in the first text, and determining at least one second condition set; the target candidate answer is one of the candidate answers; Based on each second condition set, a second text is generated; the questions included in the second text are the questions corresponding to the target condition; Inputting a second text into the language model, obtaining a second answer output by the language model; determining a verification score corresponding to the target candidate answer based on the second answer; The verification score represents the confidence of the candidate answer; The candidate answer corresponding to the highest verification score is output as the inference answer of the first text.

2. The method for reasoning text answers according to claim 1, characterized in that: The determining, based on the second answer, a verification score corresponding to the target candidate answer includes: determining whether each second answer is the same as the corresponding target condition; When the second answer is the same as the corresponding target condition, the verification score corresponding to the target candidate answer is increased by one unit; when the second answer is different from the corresponding target condition, the verification score corresponding to the target candidate answer remains unchanged.

3. The method for reasoning text answers according to claim 2, characterized in that: Repeat the determination P times whether each second answer is the same as the corresponding target condition, where P is a positive integer greater than 1.

4. The method for reasoning text answers according to claim 1, characterized in that: The step of generating a second text based on each second condition set includes: Determining the target condition based on regular matching; Using the question corresponding to the target condition as the question of the second text; A second text is generated based on each second condition set and the question corresponding to the target condition.

5. The method for reasoning text answers according to claim 1, characterized in that: The multiple candidate answers output by the language model are prompts based on the thought chain CoT and are generated by the language model.

6. The method for reasoning text answers according to claim 1, characterized in that: The language model is trained by a large-scale Chinese and English general corpus dataset, and the results are obtained by using one or more of the following datasets as input: GSM8K; SingleEq; AddSub; MultiArith; AQUA-RAT; SVAMP; CSQA; DU.

7. A text answer reasoning device, characterized in that: include: An acquisition module, used for inputting a first text into a language model and acquiring a plurality of candidate answers output by the language model; the first text is a text to be recognized; A determination module, configured to replace a target condition in a first condition set included in the first text with a target candidate answer as a condition, and determine at least one second condition set; the target candidate answer is one of the candidate answers; Based on each second condition set, a second text is generated; the questions included in the second text are the questions corresponding to the target condition; Inputting a second text into the language model, obtaining a second answer output by the language model; determining a verification score corresponding to the target candidate answer based on the second answer; The verification score represents the confidence of the candidate answer; The output module is used to output the candidate answer corresponding to the highest verification score as the inference answer of the first text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for inferring a text answer as claimed in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for inferring a text answer as claimed in any one of claims 1 to 6 is implemented.

Images