Question and answer model training method and system based on thinking chain

Abstract

The invention relates to a question and answer model training method and system based on a thinking chain. The method comprises the steps of obtaining a plurality of question thinking training samples; inputting the plurality of question thinking training samples into an initial thinking chain model, and outputting a thinking chain according to the initial thinking chain model; inputting the thinking chains into a question and answer model, and obtaining scores of the thinking chains according to thinking answers output by the question and answer model; and according to the plurality of thinking chains and the scores thereof, training the initial thinking chain model by adopting a group relative strategy optimization fine tuning method to obtain a target thinking chain model. According to the method, the understanding of the thinking model on user query can be improved, and the accuracy of the answers output according to the thinking chain and the correlation between the answers and the questions can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of adjustment of thought chain models, and in particular to a question-answering model training method and system based on thought chain. Background Art

[0002] Existing thought chain generation methods have certain limitations, primarily because they can only select the appropriate thought chain based on a given problem type. This method often struggles to generalize across different types of problems. In other words, it cannot efficiently and adaptively generate effective, debiased thought chains for a variety of problems. This limitation stems primarily from the existing methods' lack of in-depth understanding and flexible application of the nature and logical structure of the problem. This makes it difficult to quickly and accurately adjust the thought chain generation strategy when faced with new problems, thus affecting its applicability and effectiveness in different scenarios. Summary of the Invention

[0003] Based on this, the purpose of this application is to provide a question-answering model training method and system based on thought chain, which can overcome the shortcomings of the existing technology.

[0004] In order to achieve the above objectives, the technical solutions adopted in this application are: Obtain multiple problem thinking training samples; Inputting the plurality of problem thinking training samples into an initial thinking chain model, and outputting a thinking chain according to the initial thinking chain model; Inputting the thought chain into the question-answering model, and obtaining a score for each thought chain according to the thought answer output by the question-answering model; According to the multiple thinking chains and their scores, the initial thinking chain model is trained by adopting a fine-tuning method of group relative strategy optimization to obtain a target thinking chain model.

[0005] As an implementation method, the problem thinking training sample includes thinking problems and background knowledge; The step of obtaining multiple problem thinking training samples includes: Generate the thinking questions according to the knowledge structure of the preset field; The knowledge structure includes relevant knowledge including prompt words as the background knowledge; the prompt words are used to indicate the correct answer to the thinking question; According to the thinking problem and the background knowledge, the problem thinking training sample is obtained.

[0006] As an embodiment, the step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

[0007] As an embodiment, the step of inputting the thought chains into a question-answering model and obtaining a score for each thought chain according to the thought answers output by the question-answering model includes: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; According to the output probability of the candidate answer being the same as the preset standard answer, the score of the corresponding thinking chain is obtained.

[0008] As an embodiment, the step of training the initial thinking chain model based on the multiple thinking chains and their scores using a group relative strategy optimization fine-tuning method to obtain a target thinking chain model includes: Training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; Training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is equal to a preset number threshold, the candidate thinking chain model after training is determined as the target thinking chain model.

[0009] The second embodiment of the present application provides a question-answering model training system based on thought chaining, including: A sample acquisition module is used to obtain multiple problem thinking training samples; a thinking chain acquisition module, configured to input the plurality of problem thinking training samples into an initial thinking chain model, and output a thinking chain according to the initial thinking chain model; A scoring acquisition module is used to input the thought chain into the question-answering model and obtain the score of each thought chain according to the thought answer output by the question-answering model; The training module is used to train the initial thinking chain model based on the multiple thinking chains and their scores by adopting a fine-tuning method of group relative strategy optimization to obtain a target thinking chain model.

[0010] As an embodiment, the step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

[0011] As an embodiment, the step of inputting the thought chains into a question-answering model and obtaining a score for each thought chain according to the thought answers output by the question-answering model includes: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; According to the output probability of the candidate answer being the same as the preset standard answer, the score of the corresponding thinking chain is obtained.

[0012] As an embodiment, the step of training the initial thinking chain model based on the multiple thinking chains and their scores using a group relative strategy optimization fine-tuning method to obtain a target thinking chain model includes: Training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; Training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is equal to a preset number threshold, the candidate thinking chain model after training is determined as the target thinking chain model.

[0013] Compared with traditional technologies, the beneficial effects of this application are: This application inputs the multiple question thinking training samples into the initial thinking chain model, and outputs the thinking chain according to the initial thinking chain model; then inputs the thinking chain into the question-answering model, and obtains the score of each thinking chain according to the thinking answer output by the question-answering model; then, based on the multiple thinking chains and their scores, the initial thinking chain model is trained using the fine-tuning method of group relative strategy optimization to obtain the target thinking chain model. This can improve the thinking model's understanding of user queries and significantly improve the accuracy of the answers output according to the thinking chain, as well as the relevance of the answers to the questions. This improvement in ability enables the thinking chain model to more deeply explore and utilize its knowledge base, providing users with richer and more detailed information.

[0014] For better understanding and implementation, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flowchart of the steps of a question-answering model training method based on a thought chain according to an embodiment of the present application; Figure 2 A training diagram of a question-answering model training method based on a thought chain according to an embodiment of the present application; Figure 3 This is a module connection diagram of a question-answering model training system based on thought chain according to an embodiment of the present application.

[0016] 100. Question-answering model training system based on thought chain; 101. Sample acquisition module; 102. Thought chain acquisition module; 103. Score acquisition module; 104. Training module. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0018] It should be clear that the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0019] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates other meanings. The words "if" / "if" used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0020] In this application, unless otherwise specified, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0021] See also Figure 1-2 The question-answering model training method based on thought chain of the first embodiment of the present application includes: S1: Obtain multiple problem thinking training samples; S2: inputting the plurality of problem thinking training samples into an initial thinking chain model, and outputting a thinking chain according to the initial thinking chain model; S3: Inputting the thought chain into the question-answering model, and obtaining a score for each thought chain according to the thought answer output by the question-answering model; S4: Based on the multiple thinking chains and their scores, a fine-tuning method of group relative strategy optimization is adopted to train the initial thinking chain model to obtain a target thinking chain model.

[0022] As an implementation method, the problem thinking training sample includes thinking problems and background knowledge; The step of S1: obtaining a plurality of problem thinking training samples comprises: S11: generating the thinking question according to the knowledge structure of the preset domain; S12: Relevant knowledge including the prompt word in the knowledge structure is used as the background knowledge; the prompt word is used to indicate the correct answer to the thinking question; S13: Obtain the problem thinking training sample according to the thinking problem and the background knowledge.

[0023] As an embodiment, the step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

[0024] As an embodiment, the step of S3: inputting the thought chains into a question-answering model and obtaining scores for each thought chain according to the thought answers output by the question-answering model, includes: S31: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; S32: Obtaining a score for the corresponding thought chain based on the output probability of the candidate answer being the same as the preset standard answer.

[0025] Among them, the higher the score, the higher the probability that the corresponding thinking chain guides the question-answering model to output a candidate answer that is the same as the preset standard answer.

[0026] For example, the background knowledge records that "the capital of the United States" is "New York" (that is, the preset standard answer). When the question-answering model answers "the capital of the United States" based on the thought chain, the probability that the output candidate answer is "New York" is 0.6 (that is, the output probability of the candidate answer being the same as the preset standard answer). Then the thought chain score guiding the question-answering model is 0.6.

[0027] As an embodiment, the step S4: training the initial thinking chain model using a group relative strategy optimization fine-tuning method based on the multiple thinking chains and their scores to obtain a target thinking chain model includes: S41: training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; S42: training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; S43: If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; S44: If the number of training times is equal to a preset number threshold, the candidate thought chain model after training is determined as the target thought chain model.

[0028] Among them, such as Figure 2As shown, the initial thinking chain model, the candidate thinking chain model and the target thinking chain model are all thinking chain generation models, and the number threshold is set by the user.

[0029] Compared with the existing technology, this application inputs the multiple problem thinking training samples into the initial thinking chain model, and outputs the thinking chain according to the initial thinking chain model; then inputs the thinking chain into the question-answering model, and obtains the score of each thinking chain according to the thinking answer output by the question-answering model; then, based on the multiple thinking chains and their scores, the fine-tuning method of group relative strategy optimization is adopted to train the initial thinking chain model to obtain the target thinking chain model. This can improve the thinking model's understanding of user queries and significantly improve the accuracy of the answers output by the question-answering model based on the thinking chain, as well as the relevance of the answers to the questions. This improvement in ability enables the thinking chain model to more deeply explore and utilize its knowledge base, providing users with richer and more detailed information.

[0030] See also Figure 3 The second embodiment of the present application provides a question-answering model training system based on thought chain, including: A sample acquisition module is used to obtain multiple problem thinking training samples; a thinking chain acquisition module, configured to input the plurality of problem thinking training samples into an initial thinking chain model, and output a thinking chain according to the initial thinking chain model; A scoring acquisition module is used to input the thought chain into the question-answering model and obtain the score of each thought chain according to the thought answer output by the question-answering model; The training module is used to train the initial thinking chain model based on the multiple thinking chains and their scores by adopting a fine-tuning method of group relative strategy optimization to obtain a target thinking chain model.

[0031] As an embodiment, the step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

[0032] As an embodiment, the step of inputting the thought chains into a question-answering model and obtaining a score for each thought chain according to the thought answers output by the question-answering model includes: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; According to the output probability of the candidate answer being the same as the preset standard answer, the score of the corresponding thinking chain is obtained.

[0033] As an embodiment, the step of training the initial thinking chain model based on the multiple thinking chains and their scores using a group relative strategy optimization fine-tuning method to obtain a target thinking chain model includes: Training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; Training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is equal to a preset number threshold, the candidate thinking chain model after training is determined as the target thinking chain model.

[0034] It should be noted that the question-answering model training system based on the thinking chain provided by the second embodiment of the present application only uses the division of the above-mentioned functional modules as an example when executing the question-answering model training method based on the thinking chain. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the question-answering model training system based on the thinking chain provided by the second embodiment of the present application and the question-answering model training method based on the thinking chain provided by the first embodiment of the present application belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0035] The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. Those of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0036] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0037] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function selected in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 function selected in a box or multiple boxes.

[0038] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 steps for the function selected in a box or multiple boxes.

[0039] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0040] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0041] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0042] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0043] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A question-answering model training method based on thought chain, characterized in that: include: Obtain multiple problem thinking training samples; Inputting the plurality of problem thinking training samples into an initial thinking chain model, and outputting a thinking chain according to the initial thinking chain model; Inputting the thought chain into the question-answering model, and obtaining a score for each thought chain according to the thought answer output by the question-answering model; According to the multiple thinking chains and their scores, the initial thinking chain model is trained by adopting a fine-tuning method of group relative strategy optimization to obtain a target thinking chain model.

2. The question-answering model training method based on thought chain according to claim 1 is characterized in that: The problem thinking training samples include thinking problems and background knowledge; The step of obtaining multiple problem thinking training samples includes: Generate the thinking questions according to the knowledge structure of the preset field; The knowledge structure includes relevant knowledge including prompt words as the background knowledge; the prompt words are used to indicate the correct answer to the thinking question; According to the thinking problem and the background knowledge, the problem thinking training sample is obtained.

3. The question-answering model training method based on thought chain according to claim 2 is characterized in that: The step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

4. The question-answering model training method based on thought chain according to claim 1 is characterized in that: The step of inputting the thought chains into the question-answer model and obtaining scores for each thought chain according to the thought answers output by the question-answer model includes: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; According to the output probability of the candidate answer being the same as the preset standard answer, the score of the corresponding thinking chain is obtained.

5. The question-answering model training method based on thought chain according to claim 1 is characterized in that: The step of training the initial thinking chain model based on the multiple thinking chains and their scores using a group relative strategy optimization fine-tuning method to obtain a target thinking chain model includes: Training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; Training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is equal to a preset number threshold, the candidate thinking chain model after training is determined as the target thinking chain model.

6. A question-answering model training system based on thought chain, characterized in that: include: A sample acquisition module is used to obtain multiple problem thinking training samples; a thinking chain acquisition module, configured to input the plurality of problem thinking training samples into an initial thinking chain model, and output a thinking chain according to the initial thinking chain model; A scoring acquisition module is used to input the thought chain into the question-answering model and obtain the score of each thought chain according to the thought answer output by the question-answering model; The training module is used to train the initial thinking chain model based on the multiple thinking chains and their scores by adopting a fine-tuning method of group relative strategy optimization to obtain a target thinking chain model.

7. The question-answering model training system based on thought chain according to claim 6 is characterized in that: The problem thinking training samples include thinking problems and background knowledge; The step of obtaining multiple problem thinking training samples includes: Generate the thinking questions according to the knowledge structure of the preset field; The knowledge structure includes relevant knowledge including prompt words as the background knowledge; the prompt words are used to indicate the correct answer to the thinking question; According to the thinking problem and the background knowledge, the problem thinking training sample is obtained.

8. The question-answering model training system based on thought chain according to claim 7 is characterized in that: The step of inputting the plurality of problem thinking training samples into an initial thinking chain model and outputting a plurality of thinking chains according to the initial thinking chain model includes: The thinking problem and the background knowledge are input into the initial thinking chain model, so that the initial thinking chain model constructs a logically coherent thinking chain corresponding to the thinking problem and the background knowledge through an internal language generation mechanism.

9. The question-answering model training system based on thought chain according to claim 6, characterized in that: The step of inputting the thought chains into the question-answer model and obtaining scores for each thought chain according to the thought answers output by the question-answer model includes: Inputting the thought chain into a question-answering model to obtain a plurality of candidate answers output by the question-answering model and an output probability of each candidate answer; According to the output probability of the candidate answer being the same as the preset standard answer, the score of the corresponding thinking chain is obtained.

10. The question-answering model training system based on thought chain according to claim 6, characterized in that: The step of training the initial thinking chain model based on the multiple thinking chains and their scores using a group relative strategy optimization fine-tuning method to obtain a target thinking chain model includes: Training the initial thought chain model according to the multiple thought chains and their scores, and obtaining a new thought chain and a new score output by the trained candidate thought chain model; Training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is less than a preset number threshold, obtaining a new thought chain and a new score output by the trained candidate thought chain model, and training the candidate thought chain model according to the new thought chain and the new score, accumulating the number of training times; If the number of training times is equal to a preset number threshold, the candidate thinking chain model after training is determined as the target thinking chain model.

Images