Question and answer model training method, industry knowledge question and answer method, device and equipment

Abstract

The invention provides a question and answer model training method, an industry knowledge question and answer method, device and equipment, and is applied to the technical field of natural languages. The question and answer model training method comprises the steps that an initial question and answer model of a target industry is obtained, the initial question and answer model is obtained based on training of a first question and answer pair, and the question and answer form of the first question and answer pair is question-answer; a question and answer reward model of the target industry is obtained, the question and answer reward model is obtained based on training of a second question and answer pair, the question and answer form of the second question and answer pair is one question and multiple answers, and answers in the first question and answer pair and the second question and answer pair are described in a thinking chain mode; and based on the question-answer reward model and the question sentence sample of the target industry, performing reinforcement learning training on the initial question-answer model to obtain a knowledge question-answer model of the target industry. Therefore, the large language model can answer the industry knowledge conforming to user preferences, the industry questions are thinking completely in a thinking chain mode, and the accuracy of the industry knowledge answer is improved.

Description

Technical Field

[0001] This application is applied to the field of natural language technology, and particularly relates to a method for training a question-and-answer model, a method for industry knowledge question-and-answer, a device, and a device. Background Art

[0002] With the continuous development of social technology, large model technology in the field of artificial intelligence has been implemented in multiple industries to form industry large models. As one of the important achievements in the development of artificial intelligence technology, industry large models promote the digital transformation of various fields of society, provide intelligent solutions for multiple industries, optimize business processes, improve decision-making efficiency, and also promote the emergence of innovative service models.

[0003] The large language model for industry knowledge question-and-answer is one of the industry large models. For example, the large language model for medical knowledge question-and-answer belongs to the medical industry large model. Taking medical knowledge question-and-answer as an example, in the related technology, question samples of medical knowledge and answer samples corresponding to the question samples are collected, and based on the question samples and the answer samples corresponding to the question samples, a general large language model is trained in a supervised manner to obtain a large language model suitable for medical knowledge question-and-answer.

[0004] However, the accuracy of knowledge answers of the large language model in the above method still needs to be improved. Summary of the Invention

[0005] To solve the above problems, this application proposes a method for training a question-and-answer model, a method for industry knowledge question-and-answer, a device, and a device, which can improve the accuracy of knowledge answers of the large language model.

[0006] The first aspect of this application provides a method for training a question-and-answer model, including: obtaining an initial question-and-answer model for a target industry, where the initial question-and-answer model is trained based on a first question-and-answer pair, the question-and-answer form of the first question-and-answer pair is one question and one answer, and the answer in the first question-and-answer pair is described in the form of a thought chain; obtaining a question-and-answer reward model for the target industry, where the question-and-answer reward model is trained based on a second question-and-answer pair, the question-and-answer form of the second question-and-answer pair is one question and multiple answers, and the answer in the second question-and-answer pair is described in the form of a thought chain; and performing reinforcement learning training on the initial question-and-answer model based on the question-and-answer reward model and the question sample of the target industry to obtain the knowledge question-and-answer model of the target industry.

[0007] In some embodiments, the training process of the question-and-answer reward model includes: determining the expected answer to the question in the second question-and-answer pair and the non-expected answer to the question in the second question-and-answer pair among the multiple answers included in the second question-and-answer pair; and training the question-and-answer reward model based on the question in the second question-and-answer pair, the expected answer, and the non-expected answer.

[0008] In some embodiments, among the multiple answers, each answer includes multiple thinking steps forming a chain of thought and step scores respectively corresponding to the multiple thinking steps; determining the expected answer to the question in the second Q&A pair and the non-expected answer to the question in the second Q&A pair among the multiple answers included in the second Q&A pair includes: determining the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers; and determining the expected answer and the non-expected answer among the multiple answers according to the answer scores respectively corresponding to the multiple answers.

[0009] In some embodiments, the determining the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers includes: in the multiple thinking steps included in the first answer, if the step score of the last thinking step is zero, determining that the answer score of the first answer is zero; if the step score of the last thinking step is not zero, calculating the mean value of the step scores respectively corresponding to the multiple thinking steps included in the first answer, and determining that the answer score of the first answer is the mean value; where the first answer is any one of the multiple answers.

[0010] In some embodiments, the determining the expected answer and the non-expected answer among the multiple answers according to the answer scores respectively corresponding to the multiple answers includes: sorting the multiple answers according to the answer scores; performing answer sampling among the sorted multiple answers; and among the sampled answers, determining the answer with the highest answer score as the expected answer and the answer with the lowest answer score as the non-expected answer.

[0011] In some embodiments, the Q&A reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; training the Q&A reward model according to the question in the second Q&A pair, the expected answer and the non-expected answer includes: training the first reward model according to the question in the second Q&A pair, the expected answer and the non-expected answer; and training the second reward model according to the question in the second Q&A pair, the individual thinking steps of the expected answer and the individual thinking steps of the non-expected answer.

[0012] In some embodiments, training the first reward model according to the question in the second question-answer pair, the expected answer, and the unexpected answer includes: inputting the question in the second question-answer pair and the expected answer into the first reward model, predicting the score of the expected answer in the first reward model, and obtaining a first predicted score output by the first reward model; inputting the question in the second question-answer pair and the unexpected answer into the first reward model, predicting the score of the unexpected answer in the first reward model, and obtaining a second predicted score output by the first reward model; inputting the question in the second question-answer pair and the expected answer into the first prediction model, predicting the probability that the answer to the question in the second question-answer pair is the expected answer in the first prediction model, and obtaining a first predicted probability output by the first prediction model; inputting the question in the second question-answer pair and the unexpected answer into the first prediction model, predicting the probability that the answer to the question in the second question-answer pair is the unexpected answer in the first prediction model, and obtaining a second predicted probability output by the first prediction model; determining a first loss value according to the first predicted score, the second predicted score, the first predicted probability, and the second predicted probability; and adjusting the parameters of the backbone network of the first reward model according to the first loss value, where the first prediction model and the first reward model share the same backbone network.

[0013] In some embodiments, training the second reward model according to the question in the second Q&A pair, the single thinking step of the expected answer, and the single thinking step of the unexpected answer includes: inputting the question in the second Q&A pair and the single thinking step of the expected answer into the second reward model, predicting the score of the single thinking step of the expected answer in the second reward model, and obtaining the third predicted score output by the second reward model; inputting the question in the second Q&A pair and the single thinking step of the unexpected answer into the second reward model, predicting the score of the single thinking step of the unexpected answer in the second reward model, and obtaining the fourth predicted score output by the second reward model; inputting the question in the second Q&A pair and the single thinking step of the expected answer into the second prediction model, predicting the probability that the answer step of the question in the second Q&A pair is the single thinking step of the expected answer in the second prediction model, and obtaining the third predicted probability output by the second prediction model; inputting the question in the second Q&A pair and the single thinking step of the unexpected answer into the second prediction model, predicting the probability that the answer step of the question in the second Q&A pair is the single thinking step of the unexpected answer in the second prediction model, and obtaining the fourth predicted probability output by the second prediction model; determining a second loss value according to the third predicted score, the fourth predicted score, the third predicted probability, and the fourth predicted probability; and adjusting the parameters of the backbone network of the second reward model according to the second loss value, where the second prediction model and the second reward model share the same backbone network.

[0014] In some embodiments, the algorithm used for the reinforcement learning training is the Proximal Policy Optimization algorithm. Training the initial Q&A model with reinforcement learning based on the Q&A reward model and the question samples in the target industry to obtain the knowledge Q&A model of the target industry includes: initializing the actor model and the reference model in the training framework of the Proximal Policy Optimization algorithm to the initial Q&A model respectively; initializing the critic model and the reward model in the training framework to the Q&A reward model respectively; in the training framework, freezing the parameters of the reference model and the parameters of the reward model, and training the actor model and the critic model based on the loss function of the actor model, the loss function of the critic model, and the question samples; and determining the trained actor model as the knowledge Q&A model.

[0015] In some embodiments, the Q&A reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; initializing the critic model in the training framework and the reward model in the training framework as the Q&A reward model respectively includes: initializing the critic model as the first reward model, and initializing the reward model as the first reward model and the second reward model.

[0016] In some embodiments, the loss function of the actor model includes an advantage variable corresponding to the predicted token output by the actor model, and the process of determining the value of the advantage variable includes: predicting the immediate gain generated by the predicted token from a global perspective through the first reward model to obtain the first predicted gain of the predicted token; predicting the immediate gain generated by the predicted token from the perspective of individual thinking steps through the second reward model to obtain the second predicted gain of the predicted token; and determining the advantage variable according to the first predicted gain and the second predicted gain.

[0017] In some embodiments, the generation process of the first Q&A pair and the second Q&A pair includes: obtaining a pre-trained corpus of the target industry; performing text segmentation and keyword extraction on the pre-trained corpus to obtain the extracted text in the pre-trained corpus and the keywords in the pre-trained corpus; generating a plurality of questions through a question generation model according to the extracted text and the keywords; generating answers corresponding to the plurality of questions through an answer generation model set with a thought chain instruction template to obtain a plurality of initial Q&A pairs; and preprocessing the plurality of initial Q&A pairs to obtain a first Q&A data set and a second Q&A data set, where the first Q&A data set is a data set composed of the first Q&A pairs, and the second Q&A data set is a data set composed of the second Q&A pairs; wherein, the Q&A forms of the plurality of initial Q&A pairs include one question and one answer and one question and multiple answers.

[0018] In some embodiments, generating a plurality of questions through a question generation model according to the extracted text and the keywords includes: if the keyword appears in the extracted text, inputting the extracted text and the keyword that appears in the extracted text into the question generation model, otherwise inputting the extracted text into the question generation model; and generating questions in the question generation model to obtain the plurality of questions.

[0019] In some embodiments, before generating answers corresponding to the plurality of questions through an answer generation model set with a thought chain instruction template to obtain a plurality of initial Q&A pairs, it further includes: performing anaphora resolution on the plurality of questions; and / or, performing intent recognition on the plurality of questions, and deleting questions whose intent recognition results do not meet the requirements from the plurality of questions.

[0020] In some embodiments, after preprocessing the multiple initial question-and-answer pairs to obtain a first question-and-answer data set and a second question-and-answer data set, the method further includes: for each answer in the second question-and-answer pair, obtaining the step scores corresponding to the multiple thinking steps included in each answer and annotating them in the second question-and-answer pair.

[0021] A second aspect of the present application provides an industry knowledge question-and-answer method, including: obtaining a question to be answered in a target industry; inputting the question to be answered into a knowledge question-and-answer model of the target industry, and generating an answer to the question to be answered through the knowledge question-and-answer model, where the knowledge question-and-answer model is trained according to the question-and-answer model training method provided in the first aspect or any embodiment of the first aspect; and outputting the answer to the question to be answered.

[0022] A third aspect of the present application provides a question-and-answer model training device, including: a model acquisition unit, configured to acquire an initial question-and-answer model of a target industry, where the initial question-and-answer model is trained based on first question-and-answer pairs, the question-and-answer form of the first question-and-answer pairs is question-and-answer, and the answers in the first question-and-answer pairs are described in the form of a thinking chain; acquiring a question-and-answer reward model of the target industry, where the question-and-answer reward model is trained based on second question-and-answer pairs, the question-and-answer form of the second question-and-answer pairs is question-and-multiple-answers, and the answers in the second question-and-answer pairs are described in the form of a thinking chain; and a model training unit, configured to perform reinforcement learning training on the initial question-and-answer model based on the question-and-answer reward model and question samples of the target industry to obtain a knowledge question-and-answer model of the target industry.

[0023] A fourth aspect of the present application provides an industry knowledge question-and-answer device, including: a question acquisition unit, configured to acquire a question to be answered in a target industry; an answer generation unit, configured to input the question to be answered into a knowledge question-and-answer model of the target industry, and generate an answer to the question to be answered through the knowledge question-and-answer model, where the knowledge question-and-answer model is trained according to the question-and-answer model training method provided in the first aspect or any embodiment of the first aspect; and an output unit, configured to output the answer to the question to be answered.

[0024] A fifth aspect of the present application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is configured to store a program; the processor is configured to implement the question-and-answer model training method as described in the first aspect or any embodiment of the first aspect, or implement the industry knowledge question-and-answer method as described in the second aspect by running the program in the memory.

[0025] The sixth aspect of the present application provides a chip, including a processor and a data interface. The processor reads and runs a program stored on a memory through the data interface to execute the Q&A model training method described in the first aspect or any embodiment of the first aspect, or to execute the industry knowledge Q&A method described in the second aspect.

[0026] The seventh aspect of the present application provides a computer program product, including a computer program which, when executed by a processor, implements the Q&A model training method described in the first aspect or any embodiment of the first aspect, or implements the industry knowledge Q&A method described in the second aspect.

[0027] The eighth aspect of the present application provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, it implements the Q&A model training method described in the first aspect or any embodiment of the first aspect, or implements the industry knowledge Q&A method described in the second aspect.

[0028] A Q&A model training method, an industry knowledge Q&A method, a device and a device proposed according to the present application, based on a Q&A reward model and question samples in a target industry, perform reinforcement learning training on an initial Q&A model obtained through preliminary training to obtain a knowledge Q&A model for the target industry. On the one hand, the Q&A reward model is trained based on Q&A pairs with a question-and-many-answers Q&A form, which helps to improve the knowledge Q&A model's perception ability of user preferences in the reinforcement learning training and generate answers that meet user preferences. On the other hand, in the Q&A pairs used for training the initial Q&A model and the Q&A reward model, the answers are described in the way of a thought chain, so that the knowledge Q&A model after reinforcement learning can perform complete logical thinking on the questions according to the thought chain during the process of answering questions and show the thinking process in the answers. Thus, the question answering accuracy of the knowledge Q&A model is improved. Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0030] Figure 1 It is a schematic diagram of the implementation environment involved in the embodiments of the present application;

[0031] Figure 2 It is a schematic flowchart of the Q&A model training method provided by the embodiments of the present application;

[0032] Figure 3Schematic diagram of the training process of the Q&A reward model provided according to an embodiment of the present application;

[0033] Figure 4 Schematic diagram of the reinforcement learning training process provided according to an embodiment of the present application;

[0034] Figure 5 Schematic diagram of the generation process of Q&A pairs provided according to an embodiment of the present application;

[0035] Figure 6 Example diagram of the generation process of the first Q&A dataset and the second Q&A dataset provided according to an embodiment of the present application;

[0036] Figure 7 Example flowchart of generating from Q&A pairs to Q&A model training provided according to an embodiment of the present application;

[0037] Figure 8 Schematic diagram of the structure of the Q&A model training device provided according to an embodiment of the present application;

[0038] Figure 9 Schematic diagram of the structure of the industry knowledge Q&A device provided according to an embodiment of the present application;

[0039] Figure 10 Schematic diagram of the structure of the electronic device provided according to an embodiment of the present application. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0041] In the field of industry applications, user requirements are gradually diversifying. While retaining the good general knowledge generalization ability of the large language model (such as general language understanding, text generation, knowledge fusion, etc.), how to enable the large language model to master industry knowledge that meets user preferences has become one of the difficult problems to be solved for the large language model to be implemented in the field of industry applications.

[0042] The embodiments of the present application propose a method for training a question-and-answer model, an industry knowledge question-and-answer method, a device, and a device. Through a question-and-answer reward model trained based on question-and-answer pairs with multiple answers for one question, the initial question-and-answer model trained based on question-and-answer pairs with one answer for one question is trained by reinforcement learning to obtain a knowledge question-and-answer model, enhancing the knowledge question-and-answer model's ability to perceive user preferences, enabling the knowledge question-and-answer model to generate answers that conform to user preferences; in the question-and-answer pairs used to train the question-and-answer reward model and the initial question-and-answer model, the answers are described in the form of a chain of thought (Cot), enabling the knowledge question-and-answer model to think logically according to the chain of thought for industry knowledge questions. Thus, the knowledge question-and-answer model trained through the embodiments of the present application can generate professional field knowledge answers that conform to the user preference form and have a complete thinking process, improving the accuracy of knowledge answers.

[0043] Exemplary implementation environment

[0044] Please refer to Figure 1 , Figure 1 FIG. is a schematic diagram of the implementation environment related to the embodiments of the present application. In the implementation environment related to the present application, it includes a knowledge question-and-answer device 100 and a model training device 101. On the model training device 101, according to the question-and-answer model training method provided by the embodiments of the present application, a knowledge question-and-answer model can be trained; the knowledge question-and-answer model is deployed on the knowledge question-and-answer device 100, and according to the industry knowledge question-and-answer method provided by the embodiments of the present application, an answer to the industry question input by the user is provided.

[0045] Among them, the knowledge question-and-answer device 100 and the model training device 101 can be a terminal or a server. Figure 1 Taking the knowledge question-and-answer device 100 and the model training device 101 as servers as an example. When the knowledge question-and-answer device 100 is a server, the implementation environment may further include a user terminal 103. The knowledge question-and-answer device 100 can receive the question sentence to be answered input by the user through the user terminal 103. The user terminal 103 sends the question sentence to be answered to the knowledge question-and-answer device 100. In the knowledge question-and-answer device 100, an answer to the question sentence to be answered is generated through the knowledge question-and-answer model, and the answer is output to the user terminal 103.

[0046] Among them, the above implementation environment is an intelligent question-and-answer environment in a professional field, such as a medical question-and-answer environment.

[0047] Exemplary method

[0048] Please refer to Figure 2 , in an exemplary embodiment, a method for training a question-and-answer model is provided. The method for training a question-and-answer model includes:

[0049] S201. Obtain an initial Q&A model for the target industry. The initial Q&A model is trained based on the first Q&A pairs. The Q&A form of the first Q&A pairs is one question and one answer, and the answers in the first Q&A pairs are described in the form of a chain of thought.

[0050] Among them, the target industry is, for example, industries such as medicine, finance, and education.

[0051] Among them, there are multiple first Q&A pairs belonging to the target industry. Each first Q&A pair contains a question and an answer. The answer is described in the form of a chain of thought, that is, the answer contains multiple thought steps described in sequence, and these multiple thought steps form a chain of thought.

[0052] In this embodiment, the process of training the initial Q&A model based on the first Q&A pairs and the reinforcement learning training process of the initial Q&A model in the subsequent steps can occur on the same device or on different devices. If it occurs on different devices, the initial Q&A model can be obtained from the device on which the initial Q&A model is trained. Otherwise, the initial Q&A model can be obtained on the current device.

[0053] Regarding the training of the initial Q&A model, the following examples are provided:

[0054] In one example, based on the first Q&A pairs, a general large language model is trained to obtain an initial Q&A model, so that the initial Q&A model initially has the knowledge answering ability of the target industry.

[0055] In another example, based on the pre-training corpus of the target industry, a general large language model is trained to obtain an industry large model that has learned the relevant knowledge of the target industry. Based on the first Q&A pairs, the parameters of this industry large model are adjusted (which can be fine-tuning of all the parameters of the industry large model) to obtain the initial Q&A model. Thus, by first pre-training the large language model and then adjusting the parameters based on the Q&A pairs, the knowledge answering ability of the initial Q&A model is improved. Among them, the pre-training corpus can cover one or more dimensions such as knowledge Q&A, reading comprehension, information extraction, and text generation. For example, the pre-training corpus contains texts from sources such as academic articles, textbooks, and interviews in the target industry.

[0056] S202. Obtain a Q&A reward model for the target industry. The Q&A reward model is trained based on the second Q&A pairs. The Q&A form of the second Q&A pairs is one question and multiple answers, and the answers in the second Q&A pairs are described in the form of a chain of thought.

[0057] Among them, there are multiple second Q&A pairs belonging to the target industry. Each second Q&A pair contains a question and multiple answers corresponding to the question. Each answer is described in the form of a chain of thought, that is, the answer contains multiple thought steps described in sequence, and these multiple thought steps form a chain of thought.

[0058] Among them, since the Q&A form of the second Q&A pair is one question with multiple answers, for a question, among its corresponding multiple answers, some answers are more in line with the user's preferences, and some answers are less in line with the user's preferences. Therefore, the Q&A reward model trained based on the second Q&A pair can perceive whether the answer is in line with the user's preferences. For example, it outputs a higher score for the answer that meets the user's preferences and a lower score for the answer that does not meet the user's preferences.

[0059] In this embodiment, the process of training the Q&A reward model based on the second Q&A pair and the reinforcement learning training process of the initial Q&A model in the subsequent steps can occur on the same device or on different devices. If it occurs on different devices, the Q&A reward model can be obtained from the device where the Q&A reward model is trained. Otherwise, the Q&A reward model can be obtained on the current device.

[0060] S203. Based on the Q&A reward model and the question samples of the target industry, perform reinforcement learning training on the initial Q&A model to obtain the knowledge Q&A model of the target industry.

[0061] Among them, in the reinforcement learning training, there are an agent and an environment. The agent makes actions based on the current policy, and the environment feedbacks a reward signal for the actions made by the agent. The agent adjusts the current policy based on this reward signal and then makes actions based on the new policy. In this way, by adjusting the policy multiple times, the reward signal feedback by the environment is maximized to obtain the optimal policy.

[0062] In this embodiment, the initial Q&A model is the agent in the reinforcement learning training, and the Q&A reward model is the environment in the reinforcement learning training. During the reinforcement training process, the predicted answer of the question sample can be generated through the initial Q&A model (this predicted answer is the action made by the agent based on the current policy). For this predicted answer, the corresponding reward signal can be generated through the Q&A reward model, and the parameters of the initial Q&A model (i.e., adjusting the policy) are adjusted based on this reward signal, and finally the knowledge Q&A model is obtained.

[0063] In the embodiments of the present application, the Q&A reward model trained based on the Q&A pair of one question with multiple answers is used to perform reinforcement training on the initial Q&A model, so that the finally obtained knowledge Q&A model has the ability to perceive the user-preferred answers; both the first Q&A pair and the second Q&A pair adopt the answers described in the way of the thought chain, so that the knowledge Q&A model can think and answer questions according to the thought chain. Thus, the knowledge Q&A model can generate professional field knowledge answers that conform to the user-preferred form and have a complete thinking process, improving the accuracy of the knowledge answers.

[0064] Next, an embodiment is provided for the training process of the Q&A reward model.

[0065] Please refer to Figure 3 , in yet another exemplary embodiment, a training process of a question-and-answer reward model is provided. The training process of the question-and-answer reward model includes:

[0066] S301. Among the multiple answers included in the second question-and-answer pair, determine the expected answer to the question in the second question-and-answer pair and the non-expected answers to the question in the second question-and-answer pair.

[0067] Wherein, for the question in the second question-and-answer pair, the expected answer is better than the non-expected answers. Compared with the non-expected answers, the expected answer is the preferred answer of the user.

[0068] In this embodiment, the multiple answers included in the second question-and-answer pair are different in one or more aspects such as accuracy, source reliability, answer detail level, and logical rigor level. These aspects all affect the answer quality. Among the multiple answers included in the second question-and-answer pair, the expected answer and the non-expected answers to the question in the second question-and-answer pair can be selected according to the quality of the multiple answers.

[0069] In a possible implementation manner, among the multiple answers included in the second question-and-answer pair, each answer includes multiple thinking steps forming a thinking chain and step scores respectively corresponding to the multiple thinking steps. Based on this, S301 may include: S3011. According to the step scores respectively corresponding to the multiple thinking steps in the multiple answers, determine the answer scores respectively corresponding to the multiple answers; S3012. According to the answer scores respectively corresponding to the multiple answers, determine the expected answer and the non-expected answers among the multiple answers.

[0070] Wherein, the step scores respectively corresponding to the multiple thinking steps can be obtained by scoring the multiple thinking steps from one or more aspects affecting the answer quality, so as to improve the accuracy and rationality of the step scores respectively corresponding to the multiple thinking steps. Regarding the scoring of the thinking steps, reference can be made to the subsequent embodiments of the generation process of the first question-and-answer pair and the second question-and-answer pair, and details will not be repeated here.

[0071] In this implementation manner, for each answer in the second question-and-answer pair, the answer score of the answer is determined by comprehensively considering the step scores respectively corresponding to the multiple thinking steps in the answer, so that the answer score of the answer takes into account each step in the thinking chain, and the accuracy of the answer score is improved. After obtaining the answer scores respectively corresponding to the multiple answers in the second question-and-answer pair, since the answer scores respectively corresponding to the multiple answers can reflect the answer quality respectively corresponding to the multiple answers, the answer with a higher answer score can be selected as the expected answer among the multiple answers, and the answer with a lower answer score can be selected as the non-expected answer.

[0072] In one example, S3011 may include: among the multiple thinking steps included in the first answer, if the step score of the last thinking step is zero, determine that the answer score of the first answer is zero; if the step score of the last thinking step is not zero, calculate the average value of the step scores corresponding to the multiple thinking steps included in the first answer, and determine that the answer score of the first answer is this average value; where the first answer is any answer among the multiple answers included in the second question-and-answer pair.

[0073] In this example, for the convenience of representing the answer score calculation process of any answer in the second question-and-answer pair, the first answer is used to represent any answer in the second question-and-answer pair, that is, the answer score of each answer in the second question-and-answer pair can be determined with reference to the answer score determination process of the first answer provided in this example. Among the multiple thinking steps included in the first answer, the last thinking step is the answer conclusion, and whether the answer conclusion is correct has a decisive impact on an answer. Therefore, if the step score of the last thinking step is zero, determine that the answer score of the first answer is zero. If the step score of the last thinking step is not zero, determine to calculate the average value of the step scores corresponding to the multiple thinking steps included in the first answer, and determine that the answer score of the first answer is this average value. Thus, in the process of determining the answer score, the importance of the last thinking step is taken into account, and different treatments are carried out according to whether the score of the last thinking step is zero, improving the accuracy of the answer score.

[0074] Optionally, when the step score of the last thinking step is not zero, calculate the sum of the step scores corresponding to the multiple thinking steps included in the first answer to obtain a first value, and divide the first value by the number of multiple thinking steps to obtain the above average value.

[0075] Among them, the calculation formula for the answer score of the first answer is expressed as:

[0076]

[0077] Among them, n represents the number of thinking steps in the first answer, z i represents the i-th thinking step in the first answer, z n that is, the last thinking step, and score represents the answer score of the first answer.

[0078] It should be noted that when the step score of the last thinking step is not zero, the step scores corresponding to the multiple thinking steps included in the first answer can also be weighted and summed according to the step weights corresponding to the multiple thinking steps included in the first answer to obtain the above average value. Thus, different step weights can be assigned to the multiple thinking steps according to the importance levels of the multiple thinking steps, and the accuracy of the average value, that is, the accuracy of the answer score, can be improved through the weighting method.

[0079] In one example, S3012 includes: sorting the multiple answers corresponding to the multiple answers included in the second Q&A pair according to the answer scores respectively; performing answer sampling on the sorted multiple answers; among the sampled answers, determining the answer with the highest answer score as the expected answer and determining the answer with the lowest answer score as the non-expected answer.

[0080] In this example, the multiple answers can be sorted in descending or ascending order according to the answer scores corresponding to the multiple answers included in the second Q&A pair to obtain the sorted multiple answers; sampling is performed on the sorted multiple answers to obtain the sampled answers; the answer scores of the sampled answers are compared to determine the answer with the highest answer score as the expected answer and the answer with the lowest answer score as the non-expected answer. Thus, on the one hand, through answer sampling, the randomness of the expected answer and the non-expected answer is improved, and further the training effect of the Q&A reward model is improved; on the other hand, by selecting the expected answer and the non-expected answer according to the answer scores among the sampled answers, the accuracy of selecting the expected answer and the non-expected answer is improved.

[0081] Optionally, among the sorted multiple answers, two answers are sampled, the answer scores corresponding to the two answers are compared, the answer with the highest answer score is determined as the expected answer, and the answer with the lowest answer score is determined as the non-expected answer.

[0082] Optionally, random sampling is performed among the sorted multiple answers to improve the randomness and diversity of the expected answer and the non-expected answer.

[0083] Alternatively, the training process of the Q&A reward model is performed multiple times. In the i-th training process, answers different from the answers sampled in the previous i - 1 training processes are sampled, so that different expected answers and different non-expected answers may appear in different training processes, improving the randomness and diversity of the expected answer and the non-expected answer. For example, assume that the second Q&A pair includes five answers. In the first training process, the sampled answers are the first answer and the third answer. In the second training process, the sampled answers are the third answer and the fifth answer.

[0084] It should be noted that in addition to adopting this example, S3012 can also select the expected answer and the non-expected answer by setting a score threshold. Specifically, among the multiple answers included in the second answer pair, the answers with answer scores greater than or equal to the score threshold are selected as the expected answers, and the answers with answer scores less than the score threshold are selected as the non-expected answers.

[0085] S302, train the Q&A reward model according to the question, the expected answer and the non-expected answer in the second Q&A pair.

[0086] Among them, the Q&A reward model can be: based on a general large language model, or based on an industry large model trained with pre-training corpus of the target industry, or based on an initial Q&A model, with a value head added. The value head is a linear layer used to evaluate decisions or states, and the value head in this embodiment is used for answer evaluation.

[0087] In this embodiment, the input data of the Q&A reward model can be determined according to the question, expected answer and unexpected answer in the second Q&A pair; the input data is input into the Q&A reward model for processing to obtain the output data of the Q&A reward model; according to the output data of the Q&A reward model and the loss function of the Q&A reward model, the loss value of the Q&A reward model is determined; according to the loss value of the Q&A reward model, the parameters of the Q&A reward model are adjusted. By executing S301 and S302 multiple times, the parameters of the Q&A reward model can be adjusted multiple times to achieve multiple trainings of the Q&A reward model.

[0088] In a possible implementation, the Q&A reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer. Based on this, S302 includes: S3021, training the first reward model according to the question, expected answer and unexpected answer in the second Q&A pair; S3022, training the second reward model according to the individual thinking steps of the expected answer and the individual thinking steps of the unexpected answer in the second Q&A pair. Thus, the first reward model can perceive the global optimal solution, and the second reward model can perceive the local optimal solution, enabling the Q&A reward model to feedback reward signals from both global and local perspectives.

[0089] Among them, the individual thinking steps of the expected answer and the individual thinking steps of the unexpected answer can be flexibly selected during the training process of the second reward model. For example, in the first training process of the second reward model, the first thinking step of the expected answer and the first thinking step of the unexpected answer can be used. In the second training process of the second reward model, the second thinking step of the expected answer and the second thinking step of the unexpected answer can be used, and so on.

[0090] In this implementation manner, the input data of the first reward model can be determined according to the question in the second question-and-answer pair, the expected answer to the question, and the unexpected answer to the question; the input data is input into the first reward model for processing to obtain the output data of the first reward model; according to the output data of the first reward model and the loss function of the first reward model, the loss value of the first reward model is determined; according to the loss value of the first reward model, the parameters of the first reward model are adjusted. The input data of the second reward model can be determined according to the question in the second question-and-answer pair, the single thinking step of the expected answer to the question, and the single thinking step of the unexpected answer to the question; the input data is input into the second reward model for processing to obtain the output data of the second reward model; according to the output data of the second reward model and the loss function of the second reward model, the loss value of the second reward model is determined; according to the loss value of the second reward model, the parameters of the second reward model are adjusted.

[0091] In one example, S3021 includes: inputting the question and the expected answer in the second question-and-answer pair into the first reward model, predicting the score of the expected answer in the first reward model, and obtaining the first predicted score output by the first reward model; inputting the question and the unexpected answer in the second question-and-answer pair into the first reward model, predicting the score of the unexpected answer in the first reward model, and obtaining the second predicted score output by the first reward model; inputting the question and the expected answer in the second question-and-answer pair into the first prediction model, predicting the probability that the answer to the question in the second question-and-answer pair is the expected answer in the first prediction model, and obtaining the first predicted probability output by the first prediction model; inputting the question and the unexpected answer in the second question-and-answer pair into the first prediction model, predicting the probability that the answer to the question in the second question-and-answer pair is the unexpected answer in the first prediction model, and obtaining the second predicted probability output by the first prediction model; determining the first loss value according to the first predicted score, the second predicted score, the first predicted probability, and the second predicted probability; adjusting the parameters of the backbone network of the first reward model according to the first loss value, and the first prediction model and the first reward model share the same backbone network.

[0092] Among them, the difference between the first reward model and the first prediction network is that: the first reward model adds a linear layer for value evaluation (i.e., the value head mentioned in the foregoing content) on the basis of the backbone network to output the predicted score of the answer, and the first prediction model adds a linear layer for probability prediction on the basis of the backbone network to output the predicted probability of the answer.

[0093] Optionally, the dimension of the output layer of the first reward model (i.e., the aforementioned linear layer for value evaluation) is the vocabulary size, and the dimension of the output layer of the first prediction model (i.e., the aforementioned linear layer for probability prediction) is the vocabulary size. The vocabulary size is determined according to the number of output tokens of the backbone network. For each output token of the backbone network, the output layer of the first reward model can output the corresponding predicted score, and the predicted score of the last output token can be taken as the first predicted score; for each output token of the backbone network, the output layer of the first prediction model can output the corresponding predicted probability, and the predicted probability of the last output token can be taken as the first predicted probability.

[0094] In this example, during the training process of the first reward model, a first prediction model sharing the same backbone network with the first reward model is introduced. The first predicted score and the second predicted score output by the first prediction model participate in the determination process of the loss value of the first reward model. On the basis of the loss of the reward model itself, a regularization constraint on the output probability of the expected answer and the output probability of the non-expected answer is added, making the first reward model more sensitive to the expected answer. Furthermore, in the subsequent reinforcement training, the reward signal feedback by the first reward model will promote the generation probability of the expected answer (i.e., the generation probability of the preferred corpus) of the knowledge Q&A model and reduce the generation probability of the non-expected answer (i.e., the generation probability of the non-preferred corpus) of the knowledge Q&A model.

[0095] Optionally, the loss function of the first reward model is expressed as:

[0096]

[0097] Among them, represents the first reward model, represents the first prediction model, represents the parameters of the backbone network shared by the first reward model and the first prediction model, represents the loss value of the first reward model, that is, the aforementioned calculated first loss value; x represents the question sentence in the second Q&A pair, and y w represents the expected answer to this question sentence, and y l represents the non-expected answer to this question sentence. represents the first predicted score, represents the second predicted score, represents the first predicted probability, represents the second predicted probability; σ represents the activation function that converts the score into a probability, such as the sigmoid function; β and γ represent the weight coefficients corresponding to the first predicted probability and the second predicted probability respectively.

[0098] Among them, is the cross-entropy loss function, which is used to widen the gap between the first predicted score and the second predicted score; Based on the cross-entropy loss function, a loss function for the output probabilities of the expected answer and the unexpected answer is added, which is used to make the predicted output probability distribution closer to a good response (i.e., closer to the expected answer).

[0099] Therefore, based on the cross-entropy loss function of the first reward model, a penalty term (i.e., regularization constraint) for the output probabilities of the expected answer and the unexpected answer is added, making the first reward model more sensitive to the expected response.

[0100] In one example, S3022 includes: inputting the single thinking step of the question and the expected answer in the second Q&A pair into the second reward model, predicting the score of the single thinking step of the expected answer in the second reward model, and obtaining the third predicted score output by the second reward model; inputting the single thinking step of the question and the unexpected answer in the second Q&A pair into the second reward model, predicting the score of the single thinking step of the unexpected answer in the second reward model, and obtaining the fourth predicted score output by the second reward model; inputting the single thinking step of the question and the expected answer in the second Q&A pair into the second prediction model, predicting the probability that the answer step of the question in the second Q&A pair is the single thinking step of the expected answer in the second prediction model, and obtaining the third predicted probability output by the second prediction model; inputting the single thinking step of the question and the unexpected answer in the second Q&A pair into the second prediction model, predicting the probability that the answer step of the question in the second Q&A pair is the single thinking step of the unexpected answer in the second prediction model, and obtaining the fourth predicted probability output by the second prediction model; determining the second loss value according to the third predicted score, the fourth predicted score, the third predicted probability, and the fourth predicted probability; adjusting the parameters of the backbone network of the second reward model according to the second loss value, and the second prediction model shares the same backbone network with the second reward model. Therefore, by utilizing the characteristic that the answer is described in a chain-of-thought manner, by inputting the single thinking step into the second reward model and the second prediction model, the second reward model perceives the answer preference from the perspective of the single thinking step and searches for the local optimal solution.

[0101] Among them, the second reward model and the second prediction model are different from the first reward model and the first prediction model in terms of the input data. The implementation principle and technical effect of the above training process of the second reward model and the second prediction model can refer to the foregoing description of the first reward model and the first prediction model. For example, replacing y in the loss function of the first reward model w with the single thinking step of the expected answer, and replacing y in the loss function of the first reward model l with the single thinking step of the unexpected answer, the loss function of the second reward model can be obtained.

[0102] In the embodiments of the present application, a process for training a question-and-answer reward model based on a second question-and-answer pair is provided. By selecting an expected question and an unexpected question from multiple questions included in the second question-and-answer pair, the ability of the question-and-answer reward model to perceive the user's preferred answers is improved, which is beneficial to promoting the knowledge question-and-answer model to generate answers that meet the user's preferences through the question-and-answer reward model during the reinforcement training process.

[0103] In some embodiments, in the reinforcement learning training of the initial question-and-answer model, the algorithm adopted is the proximal policy optimization (PPO) algorithm. Compared with other reinforcement learning algorithms, such as the direct preference optimization (DPO) algorithm, the proximal policy optimization algorithm can enable the large language model to have better performance in complex industry scenarios, while ensuring that the general capabilities (language understanding, text generation, knowledge question-and-answer, etc.) of the large language model do not decrease significantly. Therefore, using the PPO algorithm to perform reinforcement learning training on the initial question-and-answer model can improve the performance of the initial question-and-answer model in the knowledge question-and-answer field of the target industry.

[0104] Please refer to Figure 4 , in another exemplary embodiment, a reinforcement learning training process for an initial question-and-answer model is provided. The reinforcement learning training process may include:

[0105] S401, initialize the actor model and the reference model in the training framework of the proximal policy optimization algorithm to the initial question-and-answer model respectively.

[0106] S402, initialize the critic model and the reward model in the training framework to the question-and-answer reward model respectively.

[0107] Among them, in the training framework of the proximal policy optimization algorithm, there are an actor model, a reference model, a critic model, and a reward model. The actor model and the reference model adopt the same large language model. During the training process, the reference model is used to compare with the actor model to constrain the actor model from deviating too much from the original large language model. During the training process, the critic model is used to evaluate the long-term rewards brought by the behavior of the actor model, and the reward model is used to evaluate the immediate benefits brought by the behavior of the actor model. It can be understood that: the critic model is used to evaluate the rewards brought by the current output token of the actor model after the actor model outputs the last output token, and the reward model is used to analyze the immediate benefits brought by the current output token of the actor model.

[0108] In this embodiment, the actor model and the reference model in the training framework of the proximal policy optimization algorithm are respectively initialized as the initial Q&A model, so that the actor model and the reference model use the same large language model; the critic model in the training framework and the reward model in the training framework are respectively initialized as the Q&A reward model. Since the Q&A reward model already has the ability to analyze immediate rewards through the foregoing training, the parameters of the reward model in the training framework can be frozen, and the parameters of the critic model in the training framework can be adjusted to improve the ability of the critic model to predict long-term rewards.

[0109] In one example, the Q&A reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; based on this, S402 includes: S4021, initializing the critic model as the first reward model and initializing the reward model as the first reward model and the second reward model. Among them, considering that the critic model predicts long-term rewards, the critic model is initialized as the first reward model that scores from the global perspective of the answer; considering that the reward model predicts immediate rewards, the reward model is initialized as the first reward model and the second reward model, so that in the reinforcement training process, the first reward model can feedback the reward signal considering the whole answer to the actor model and the second reward model can feedback the reward signal considering individual thinking steps to the actor model, improving the policy optimization effect of the actor model.

[0110] S403, in the training framework, freeze the parameters of the reference model and the parameters of the reward model, and train the actor model and the critic model based on the loss function of the actor model, the loss function of the critic model, and the question samples.

[0111] In this embodiment, since the reference model is used to constrain the actor model not to deviate too much from the original model, that is, too much from the initial Q&A model, in the reinforcement training process, the parameters of the reference model are frozen, that is, the reference model is kept as the initial Q&A model; since the Q&A reward model is a model that has been maturely trained, in the reinforcement training process, the parameters of the reward model in the training framework of the proximal policy optimization algorithm are frozen, that is, the reward model is kept as the Q&A reward model. The reason for adjusting the parameters of the critic model is on the one hand that the critic model needs to learn to evaluate the long-term rewards of the answer, and on the other hand that the critic model is initialized as the Q&A reward model. The Q&A reward model scores the complete sentence of the answer during training, while in the reinforcement learning training, the critic model needs to score the individual tokens output by the actor model.

[0112] In one example, in the training framework of the Proximal Policy Optimization (PPO) algorithm, the process of the i-th training of the actor model and the critic model includes: taking the input sequence in the i-th training (if the i-th training process is the initial input of a question sample, the input sequence is the token sequence obtained by transforming the question sample; otherwise, the input token sequence is obtained by concatenating the input sequence of the actor model in the (i - 1)-th training and the predicted tokens output by the actor model in the (i - 1)-th training process), generating an answer in the actor model according to the input sequence to obtain the predicted tokens output by the actor model; concatenating the input sequence and the predicted tokens to obtain the input sequence in the (i + 1)-th training; inputting the input sequence in the (i + 1)-th training into the actor model, the reference model, the critic model, and the reward model to obtain the predicted probability output by the actor model (i.e., the predicted probability corresponding to the predicted tokens in the i-th training), the predicted probability output by the reference model (i.e., the predicted probability corresponding to the predicted tokens in the i-th training, and the predicted probability output by the reference model may be different from the predicted probability output by the actor model), the token value output by the critic model (the long-term return corresponding to the predicted tokens in the i-th training), and the immediate reward output by the reward model (the immediate reward corresponding to the predicted tokens in the i-th training); determining the loss value of the actor model for the predicted tokens in the i-th training and the loss value of the critic model for the predicted tokens in the i-th training according to the predicted probability output by the actor model, the predicted probability output by the reference model, the token value output by the critic model, the immediate reward output by the reward model, the loss function of the actor model, and the loss function of the critic model. Thus, for multiple predicted tokens output by the actor model during multiple training processes, the loss value of the actor model for these multiple predicted tokens and the loss value of the critic model corresponding to these multiple predicted tokens can be determined respectively. Finally, by synthesizing the loss values of the actor model for multiple predicted tokens in one epoch (including multiple trainings), the parameters of the actor model are adjusted, and by synthesizing the loss values of the critic model for multiple predicted tokens in one epoch, the parameters of the critic model are adjusted.

[0113] Among them, multiple epochs of training can be performed in one batch, and all question samples in one batch are traversed during each epoch of training. For example, one batch contains 5 question samples, and each epoch of training needs to traverse these 5 question samples, and the parameters of the actor model and the critic model are adjusted by synthesizing the loss values of multiple predicted tokens corresponding to each question sample.

[0114] In a possible implementation, in the loss function of the actor model, an advantage variable corresponding to the predicted token output by the actor model is included. The process of determining the value of the advantage variable includes: predicting the immediate reward generated by the predicted token from a global perspective through a first reward model to obtain the first predicted reward of the predicted token; predicting the immediate reward generated by the predicted token from the perspective of a single thinking step through a second reward model to obtain the second predicted reward of the predicted token; and determining the advantage variable based on the first predicted reward and the second predicted reward.

[0115] In this implementation, the advantage variable corresponding to the predicted token can be understood as the reward signal corresponding to the predicted token. By combining the immediate reward obtained from the global perspective of the answer and the immediate reward obtained from the perspective of a single thinking step to determine the advantage variable, the immediate reward from the global perspective and the immediate reward from the local perspective are fed back to the actor model through the loss value, improving the reinforcement learning training effect of the actor model. This enables the actor model to not only explore the global optimal solution but also explore the step-by-step optimal solution, improving the knowledge answering accuracy of the trained actor model.

[0116] In one example, the loss function of the actor model is expressed as:

[0117]

[0118] Adv t =(R t +β*R′ t +γ*V t+1 -V t )+γ*λ*Adv t+1

[0119] where actor_loss represents the loss value of the actor model. A t represents the t-th predicted token output by the actor model in one round of training, Adv t represents the corresponding advantage variable, S t represents the context information of A t , that is, the input sequence of the t-th time of the actor model in this round of training; P represents the actor model after multiple rounds of training updates, and P old represents the actor model of the previous round of update of P; k and j are constants, k is less than j, and clip is a clipping function, which means that if is less than k or greater than j, then the value of is restricted within the range of k to j.

[0120] where, R t represents the first predicted reward corresponding to A t (predicted by the first reward model), and R′ t represents At The corresponding second predicted revenue (predicted by the second reward model), V t+1 Denotes the token value obtained by the critic model for evaluating the value of the (t + 1)-th predicted token, V t Denotes the token value obtained by the critic model for evaluating the value of the t-th predicted token. β, γ, and λ denote constants.

[0121] In one example, the loss function of the critic model is expressed as:

[0122]

[0123] vf_loss1 = (value - return) 2

[0124] vf_loss2 = (value_clipped - return) 2

[0125] value_clipped = clip(value, h, g)

[0126] Where M is a constant, N represents the number of predicted tokens output by the actor model in one round of training. value represents the token value obtained by the critic model for evaluating the value of the current token, return represents the actual value obtained by the critic model for evaluating the value of the current token after forward adjustment in one round of training, and h and g are constants, with h less than g.

[0127] S404. Determine the trained actor model as the knowledge Q&A model.

[0128] In this embodiment, after completing the reinforcement learning training, determine the trained actor model as the knowledge Q&A model.

[0129] In the embodiment of the present application, a process of performing reinforcement training on the initial Q&A model through the proximal policy optimization algorithm is provided. In this process, the Q&A reward model trained in the foregoing embodiment is used, and the calculation of the loss value is improved, so that the scores obtained by the Q&A reward model from global and local evaluations are fed back to the initial Q&A model through the loss value, and finally the knowledge Q&A model for the target industry is trained, improving the question answering accuracy of the knowledge Q&A model.

[0130] Please refer to Figure 5 , in another exemplary embodiment, a generation process of Q&A pairs (including first Q&A pairs and second Q&A pairs) is provided. The generation process of the Q&A pairs may include:

[0131] S501. Obtain the pre-trained corpus of the target industry.

[0132] Among them, the pre-training corpus can come from different sources. For example, the pre-training corpus contains texts from sources such as academic articles, textbooks, and interviews in the target industry, and can cover one or more dimensions such as knowledge Q&A, reading comprehension, information extraction, and text generation.

[0133] In this embodiment, the pre-training corpus can be obtained from a database, or the pre-training corpus sent by other devices can be received, or the pre-training corpus input by the user can be received.

[0134] S502. Perform text segmentation and keyword extraction on the pre-training corpus to obtain the extracted text in the pre-training corpus and the keywords in the pre-training corpus.

[0135] In this embodiment, the texts in the pre-training corpus appear in the form of long paragraphs, with excessive length and possible redundant information, and a large number of secondary information. Therefore, the text paragraphs in the pre-training corpus can be segmented into multiple texts. For the convenience of distinction, the segmented texts are called extracted texts; keyword recognition can be performed in the pre-training corpus to extract the recognized keywords.

[0136] In one example, semantic recognition is performed on the text paragraphs in the pre-training corpus, and according to the semantic recognition results, the text paragraphs are segmented to obtain the extracted texts. Thus, from the semantic perspective, the accuracy of text paragraph segmentation is improved, and the semantic integrity of the extracted texts is ensured.

[0137] In one example, the occurrence frequency of words in the pre-training corpus can be counted. If the occurrence frequency of a word is greater than or equal to the frequency threshold, then the word is determined to be a keyword in the pre-training corpus. Thus, by identifying high-frequency words, the accuracy of keyword extraction is improved.

[0138] S503. Generate multiple questions through a question generation model according to the extracted text and keywords.

[0139] Among them, the question generation model is a pre-trained large language model that can complete the task of question generation.

[0140] In this embodiment, the extracted text and keywords can be input into the question generation model, and questions are generated in the question generation model based on the extracted text and keywords to obtain the output multiple questions.

[0141] In a possible implementation, S503 includes: if a keyword appears in the extracted text, input the extracted text and the keyword that appears in the extracted text into the answer generation model; otherwise, input the extracted text into the answer generation model; generate multiple questions in the answer generation model. Thus, when a keyword appears in the extracted text, use the keyword as enhanced data for the extracted text and input it into the answer generation model together with the extracted text. The answer generation model can focus on the keyword during the question generation process and generate more accurate questions.

[0142] In one example, the accuracy of identifying whether a keyword appears in the extracted text can be improved by matching the extracted text with the keyword in terms of text.

[0143] In another example, after obtaining the extracted text by segmentation, the extracted text and its text position information can be saved; after extracting the keyword, the keyword and its word position information can be saved. By comparing the text position information of the extracted text and the word position information of the keyword, it is determined whether a keyword appears in the extracted text. Thus, by comparing the position information, the accuracy of identifying whether a keyword exists in the extracted text is improved.

[0144] Among them, the text position information of the extracted text indicates the position of the extracted text in the pre-trained corpus, and the word position information of the keyword indicates the position of the keyword in the pre-trained corpus. For example, the identification information of the affiliated corpus, the row and column positions in the corpus.

[0145] S504, generate answers corresponding to multiple questions through an answer generation model with a thought chain instruction template to obtain multiple initial question-answer pairs.

[0146] Among them, the question-answer forms of the multiple initial question-answer pairs include one question and one answer and one question and multiple answers.

[0147] Among them, there are multiple fields in the large language model. For example, an input field (i.e., the "input" field, used to store input data), a target field (i.e., the "taget" field, used to store output data), and a system field (i.e., the "system" field). The answer generation model is a large language model, and the thought chain instruction template can be set in the system field of the answer generation model.

[0148] Among them, the thought chain instruction template indicates that the answer generation model is an expert in the target industry. The answer generation model needs to generate corresponding answers for the input questions, and the generated answers need to be described in the form of a thought chain and give an example of the answer described in the thought chain. Thus, through a detailed thought chain instruction template, sufficient information is provided for the answer generation model.

[0149] For example, taking the medical industry as an example, the thought chain instruction template includes the following three points: 1. You are an expert in the medical industry, responsible for answering knowledge questions in the medical industry. The task goal is to expand the input question into a usable answer according to the thought chain. 2. The style requirement for the output answer (i.e., the template for the output answer): "First, XXX. Next, XXX. Then, XXX. Therefore, XXX." 3. The following is an example. Please refer to this thought process and apply it to answer generation: XXX.

[0150] In this embodiment, for each of the multiple questions, the question can be input into the answer generation model to obtain the answer corresponding to the question. According to the multiple questions and the answers respectively corresponding to the multiple questions, multiple initial question-answer pairs are obtained. For each question, the answer generation model can generate one or more answers. Therefore, among the multiple initial question-answer pairs, there are initial question-answer pairs in the form of one question and one answer and initial question-answer pairs in the form of one question and multiple answers.

[0151] In one example, before S504, coreference resolution can be performed on the multiple questions, and / or intent recognition can be performed on the multiple questions, and questions with intent recognition results that do not meet the requirements are deleted from the multiple questions. Thereby, the quality of the multiple questions is improved, and the quality of the multiple initial question-answer pairs is improved.

[0152] Among them, performing coreference resolution on the multiple questions means unifying different pronouns corresponding to the same entity in the multiple questions into the same pronoun. Deleting questions with intent recognition results that do not meet the requirements may include identifying questions with fuzzy intent recognition results. Coreference resolution and intent recognition can be implemented through a large language model, and the specific implementation process is not limited here.

[0153] It should be noted that in addition to coreference resolution and intent recognition, the quality of the questions can also be improved by other means. For example, other screening rules can be set.

[0154] S505, preprocess the multiple initial question-answer pairs to obtain a first question-answer data set and a second question-answer data set. The first question-answer data set is a data set composed of first question-answer pairs, and the second question-answer data set is a data set composed of second question-answer pairs.

[0155] In this embodiment, by preprocessing the multiple initial question-answer pairs, the quality of the multiple initial question-answer pairs is optimized to obtain a first question-answer data set with the question-answer form of one question and one answer and a second question-answer data set with the question-answer form of one question and multiple answers for the question-answer pairs.

[0156] In a possible implementation, preprocessing is performed on multiple initial question-answer pairs, including: searching for relevant information of multiple initial question-answer pairs on the network, and performing data augmentation on the answers in multiple initial question-answer pairs according to the relevant information of multiple initial question-answer pairs. Thus, by obtaining network data to supplement the answers in multiple initial question-answer pairs, the answer quality is improved.

[0157] Among them, the relevant information of multiple initial question-answer pairs can be searched on the network by means of a large language model calling a network interface. Data augmentation is performed on the answers in multiple initial question-answer pairs. For example, the answers searched on the network are supplemented as new answers in the initial question-answer pairs, or the answers searched on the network are merged with the existing answers in the initial question-answer pairs.

[0158] In another possible implementation, preprocessing is performed on multiple initial question-answer pairs, including: scoring multiple initial question-answer pairs through a data quality inspection large model to obtain quality scores respectively corresponding to multiple initial question-answer pairs; deleting the initial question-answer pairs with quality scores less than the score threshold from multiple initial question-answer pairs according to the quality scores respectively corresponding to multiple initial question-answer pairs. Thus, by deleting the initial question-answer pairs with unqualified quality scores, the quality of the initial question-answer pairs is improved.

[0159] Among them, the data quality inspection large model can score the initial question-answer pairs from the dimension of instruction following. Instruction following means whether the generated answer meets the instruction requirements (which can be understood as whether it meets the requirements in the thought chain instruction template), so as to delete the initial question-answer pairs that do not meet the instruction requirements.

[0160] Optionally, as Figure 5 shown, after preprocessing multiple initial question-answer pairs to obtain a first question-answer data set and a second question-answer data set, it further includes: S506, for each answer in the second question-answer pair, obtaining the step scores respectively corresponding to the multiple thinking steps included in each answer and annotating them in the second question-answer pair. The step scores corresponding to the multiple thinking steps can play a role in training a question-answer reward model, and for specific details, reference can be made to the description of the foregoing embodiments, which will not be elaborated here.

[0161] In this optional manner, artificial preference scoring can be performed on the multiple thinking steps included in each answer in the second question-answer pair to obtain the step scores respectively corresponding to the multiple thinking steps included in each answer. Among them, scoring can be performed on the multiple thinking steps by integrating one or more dimensions of instruction requirements, safety requirements, content requirements, and format requirements, and the scoring of the thinking steps can be performed manually.

[0162] Further, before obtaining the step scores corresponding to the multiple thinking steps included in each answer, question-and-answer pair sampling can be performed in the second question-and-answer pair, and the multiple thinking steps included in each answer in the sampled question-and-answer pair are scored. Thus, by sampling a part of the question-and-answer pairs, the workload of scoring the thinking steps of the answers in the question-and-answer pairs is reduced, and the efficiency is improved.

[0163] As an example, the multiple initial question-and-answer pairs generated by the answer generation large model include 20,000 one-question-one-answer corpora (i.e., question-and-answer pairs in the form of one question and one answer) and 10,000 one-question-multiple-answer corpora (i.e., question-and-answer pairs in the form of one question and multiple answers); the 20,000 one-question-one-answer corpora are preprocessed, and more than 10,000 selected one-question-one-answer corpora are left, that is, the first question-and-answer pair; the 10,000 one-question-multiple-answer corpora are preprocessed, and then 5,000 one-question-multiple-answer corpora are randomly sampled, that is, the second question-and-answer pair; the thinking steps included in the answers in the 5,000 one-question-multiple-answer corpora are scored.

[0164] In the embodiments of the present application, the generation processes of the first question-and-answer pair and the second question-and-answer pair are provided: based on the segmented text and the extracted keywords, question generation is performed, and then a large model is used to generate answers for the questions, and question-and-answer pairs are obtained through a series of processes. Compared with the traditional method of constructing question-and-answer pairs, this method reduces the manual construction cost, improves the generation efficiency of question-and-answer pairs, and has a higher corpus availability rate. In this method, the answers can also be scored according to the steps of the thinking chain, providing a data basis for the knowledge question-and-answer model in reinforcement learning to perceive the complete thinking process and user preferences.

[0165] Figure 6 It is a schematic diagram of the generation process of the first question-and-answer data set and the second question-and-answer data set provided according to the embodiments of the present application. As Figure 6 shown, the pre-trained corpus is segmented and keywords are extracted by a large model, and then based on the segmented text and the extracted keywords, questions are generated by the large model; then, the questions are quality-checked by the large model (for example, anaphora resolution and intention recognition are performed); for the questions that pass the quality check, one-question-one-answer data generation and one-question-multiple-answer data generation are performed by the large model to obtain one-question-one-answer corpora and one-question-multiple-answer corpora; for the one-question-one-answer corpora, data quality check is performed by the large model (that is, scoring is performed by the data quality check large model in the foregoing embodiments), and finally the first question-and-answer data set is obtained; for the one-question-multiple-answer corpora, data quality check is performed by the large model (that is, scoring is performed by the data quality check large model in the foregoing embodiments), and then random sampling is performed, and manual preference scoring is performed on the multiple thinking steps included in the answers in the randomly sampled one-question-multiple-answer corpora, and finally the second question-and-answer data set with the step scores of the thinking steps marked is obtained.

[0166] Figure 7It is a flow example diagram for generating from question-answer pairs to question-answer model training according to an embodiment of the present application. As Figure 7 shown, first, based on the pre-trained corpus, a first question-answer dataset (including first question-answer pairs) and a second question-answer dataset (including second question-answer pairs) are generated. This process can refer to the description of the foregoing embodiments; based on the pre-trained corpus and the first question-answer dataset, an initial question-answer model can be trained; based on the second question-answer dataset, a reward model for step-by-step scoring of the thought chain (i.e., the second reward model) and a reward model for global scoring (i.e., the first reward model) can be trained. Combining the initial question-answer model, the reward model for step-by-step scoring of the thought chain, and the reward model for global scoring, reinforcement learning training is performed to finally obtain a knowledge question-answer model for the target industry.

[0167] An embodiment of the present application also provides an industry knowledge question-answer method. The industry knowledge question-answer method includes: obtaining a question to be answered in the target industry; inputting the question to be answered into the knowledge question-answer model of the target industry, and generating an answer to the question to be answered through the knowledge question-answer model, where the knowledge question-answer model is trained according to the question-answer model training method provided in any of the foregoing embodiments; outputting the answer to the question to be answered. Thus, the knowledge question-answer model is used for knowledge answering in the target industry, and by providing an answer that conforms to the user's preference and is described in the way of a thought chain, the user's question-answer experience is improved.

[0168] Exemplary device

[0169] Correspondingly, an embodiment of the present application also provides a question-answer model training device.

[0170] Please refer to Figure 8 , in an exemplary embodiment, a question-answer model training device 800 is provided. The question-answer model training device 800 includes: a model acquisition unit 801, configured to acquire an initial question-answer model of the target industry. The initial question-answer model is trained based on the first question-answer pairs, the question-answer form of the first question-answer pairs is one question and one answer, and the answers in the first question-answer pairs are described in the way of a thought chain; acquire a question-answer reward model of the target industry. The question-answer reward model is trained based on the second question-answer pairs, the question-answer form of the second question-answer pairs is one question and multiple answers, and the answers in the second question-answer pairs are described in the way of a thought chain; a model training unit 802, configured to perform reinforcement learning training on the initial question-answer model based on the question-answer reward model and a question sample of the target industry to obtain a knowledge question-answer model of the target industry.

[0171] In some embodiments, the training process of the question-answer reward model includes: determining an expected answer to the question in the second question-answer pair and a non-expected answer to the question in the second question-answer pair among the multiple answers included in the second question-answer pair; training the question-answer reward model according to the question, the expected answer, and the non-expected answer in the second question-answer pair.

[0172] In some embodiments, among multiple answers, each answer includes multiple thinking steps forming a chain of thought and step scores respectively corresponding to the multiple thinking steps; in the multiple answers included in the second question-and-answer pair, determining the expected answer to the question in the second question-and-answer pair and the non-expected answer to the question in the second question-and-answer pair includes: determining the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers; and determining the expected answer and the non-expected answer among the multiple answers according to the answer scores respectively corresponding to the multiple answers.

[0173] In some embodiments, determining the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers includes: in the multiple thinking steps included in the first answer, if the step score of the last thinking step is zero, determining that the answer score of the first answer is zero; if the step score of the last thinking step is not zero, calculating the mean value of the step scores respectively corresponding to the multiple thinking steps included in the first answer, and determining that the answer score of the first answer is the mean value; wherein, the first answer is any one of the multiple answers.

[0174] In some embodiments, determining the expected answer and the non-expected answer among the multiple answers according to the answer scores respectively corresponding to the multiple answers includes: sorting the multiple answers according to the answer scores; performing answer sampling among the sorted multiple answers; among the sampled answers, determining the answer with the highest answer score as the expected answer and the answer with the lowest answer score as the non-expected answer.

[0175] In some embodiments, the question-and-answer reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; training the question-and-answer reward model according to the question, the expected answer and the non-expected answer in the second question-and-answer pair includes: training the first reward model according to the question, the expected answer and the non-expected answer in the second question-and-answer pair; and training the second reward model according to the question, the individual thinking steps of the expected answer and the individual thinking steps of the non-expected answer in the second question-and-answer pair.

[0176] In some embodiments, training the first reward model according to the question, the expected answer, and the unexpected answer in the second question-answer pair includes: inputting the question and the expected answer in the second question-answer pair into the first reward model, predicting the score of the expected answer in the first reward model, and obtaining the first predicted score output by the first reward model; inputting the question and the unexpected answer in the second question-answer pair into the first reward model, predicting the score of the unexpected answer in the first reward model, and obtaining the second predicted score output by the first reward model; inputting the question and the expected answer in the second question-answer pair into the first prediction model, predicting the probability that the answer to the question in the second question-answer pair is the expected answer in the first prediction model, and obtaining the first predicted probability output by the first prediction model; inputting the question and the unexpected answer in the second question-answer pair into the first prediction model, predicting the probability that the answer to the question in the second question-answer pair is the unexpected answer in the first prediction model, and obtaining the second predicted probability output by the first prediction model; determining a first loss value according to the first predicted score, the second predicted score, the first predicted probability, and the second predicted probability; and adjusting the parameters of the backbone network of the first reward model according to the first loss value, where the first prediction model and the first reward model share the same backbone network.

[0177] In some embodiments, training the second reward model according to the question, the single thinking step of the expected answer, and the single thinking step of the unexpected answer in the second question-answer pair includes: inputting the question and the single thinking step of the expected answer in the second question-answer pair into the second reward model, predicting the score of the single thinking step of the expected answer in the second reward model, and obtaining the third predicted score output by the second reward model; inputting the question and the single thinking step of the unexpected answer in the second question-answer pair into the second reward model, predicting the score of the single thinking step of the unexpected answer in the second reward model, and obtaining the fourth predicted score output by the second reward model; inputting the question and the single thinking step of the expected answer in the second question-answer pair into the second prediction model, predicting the probability that the answer step to the question in the second question-answer pair is the single thinking step of the expected answer in the second prediction model, and obtaining the third predicted probability output by the second prediction model; inputting the question and the single thinking step of the unexpected answer in the second question-answer pair into the second prediction model, predicting the probability that the answer step to the question in the second question-answer pair is the single thinking step of the unexpected answer in the second prediction model, and obtaining the fourth predicted probability output by the second prediction model; determining a second loss value according to the third predicted score, the fourth predicted score, the third predicted probability, and the fourth predicted probability; and adjusting the parameters of the backbone network of the second reward model according to the second loss value, where the second prediction model and the second reward model share the same backbone network.

[0178] In some embodiments, the algorithm adopted in reinforcement learning training is the Proximal Policy Optimization (PPO) algorithm. The model training unit 802 is specifically configured to: initialize the actor model and the reference model in the training framework of the PPO algorithm as the initial question-answering model respectively; initialize the critic model and the reward model in the training framework as the question-answering reward model respectively; in the training framework, freeze the parameters of the reference model and the parameters of the reward model, and train the actor model and the critic model based on the loss function of the actor model, the loss function of the critic model, and the question samples; determine the trained actor model as the knowledge question-answering model.

[0179] In some embodiments, the question-answering reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; the model training unit is specifically configured to: initialize the critic model as the first reward model, and initialize the reward model as the first reward model and the second reward model.

[0180] In some embodiments, the loss function of the actor model includes an advantage variable corresponding to the predicted token output by the actor model. The process of determining the value of the advantage variable includes: predicting the immediate benefit generated by the predicted token from a global perspective through the first reward model to obtain the first predicted benefit of the predicted token; predicting the immediate benefit generated by the predicted token from the perspective of individual thinking steps through the second reward model to obtain the second predicted benefit of the predicted token; determining the advantage variable according to the first predicted benefit and the second predicted benefit.

[0181] In some embodiments, the generation process of the first question-answer pair and the second question-answer pair includes: obtaining the pre-trained corpus of the target industry; performing text segmentation and keyword extraction on the pre-trained corpus to obtain the extracted text and the keywords in the pre-trained corpus; generating multiple questions through the question generation model according to the extracted text and the keywords; generating answers corresponding to the multiple questions through the answer generation model set with the thought chain instruction template to obtain multiple initial question-answer pairs; preprocessing the multiple initial question-answer pairs to obtain the first question-answer data set and the second question-answer data set, where the first question-answer data set is the data set composed of the first question-answer pairs, and the second question-answer data set is the data set composed of the second question-answer pairs; among them, the question-answer forms of the multiple initial question-answer pairs include one question with one answer and one question with multiple answers.

[0182] In some embodiments, generating multiple questions through the question generation model according to the extracted text and the keywords includes: if the extracted text contains keywords, input the extracted text and the keywords that appear in the extracted text into the question generation model, otherwise input the extracted text into the question generation model; perform question generation in the question generation model to obtain multiple questions.

[0183] In some embodiments, before generating answers corresponding to multiple questions through an answer generation model with a chain of thought instruction template and obtaining multiple initial question-answer pairs, it further includes: coreference resolution for multiple questions; and / or, intention recognition for multiple questions, and deleting questions whose intention recognition results do not meet the requirements from multiple questions.

[0184] In some embodiments, after preprocessing multiple initial question-answer pairs to obtain a first question-answer dataset and a second question-answer dataset, it further includes: for each answer in the second question-answer pair, obtaining the step scores corresponding to multiple thinking steps included in each answer and annotating them in the second question-answer pair.

[0185] The question-answer model training device 800 provided in this embodiment belongs to the same inventive concept as the question-answer model training method provided in the above embodiments of the present application, can execute the question-answer model training method provided in any of the above embodiments of the present application, and has corresponding functional modules and beneficial effects for executing the question-answer model training method. For technical details not described in detail in this embodiment, reference can be made to the specific processing content of the question-answer model training method provided in the above embodiments of the present application, which will not be elaborated here.

[0186] Correspondingly, an industry knowledge question-answer device is further provided in an embodiment of the present application.

[0187] Please refer to Figure 9 , in an exemplary embodiment, an industry knowledge question-answer device 900 is provided. The industry knowledge question-answer device 900 includes: a question acquisition unit 901, configured to acquire questions to be answered in a target industry; an answer generation unit 902, configured to input the questions to be answered into a knowledge question-answer model in the target industry, and generate answers to the questions to be answered through the knowledge question-answer model, where the knowledge question-answer model is trained according to the question-answer model training method provided in any of the foregoing embodiments; and an output unit 903, configured to output answers to the questions to be answered.

[0188] The industry knowledge question-answer device 900 provided in this embodiment belongs to the same inventive concept as the industry knowledge question-answer method provided in the above embodiments of the present application, can execute the industry knowledge question-answer method provided in any of the above embodiments of the present application, and has corresponding functional modules and beneficial effects for executing the industry knowledge question-answer method. For technical details not described in detail in this embodiment, reference can be made to the specific processing content of the industry knowledge question-answer method provided in the above embodiments of the present application, which will not be elaborated here.

[0189] The functions implemented by each unit in the above devices (question-answer model training device, industry knowledge question-answer device) can be respectively implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0190] It should be understood that the units in the above device can be implemented in the form of a processor invoking software. For example, the device includes a processor, the processor is connected to a memory, instructions are stored in the memory, and the processor invokes the instructions stored in the memory to implement any of the above methods or the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory inside the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of a hardware circuit, and the functions of some or all of the units can be realized through the design of the hardware circuit. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized through the design of the logical relationship of the components in the circuit. Again, for example, in another implementation, the hardware circuit can be implemented through a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured through a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be all implemented in the form of a processor invoking software, or all implemented in the form of a hardware circuit, or some implemented in the form of a processor invoking software, and the remaining part implemented in the form of a hardware circuit.

[0191] In the embodiments of the present application, the processor is a circuit with the ability to process signals. In one implementation, the processor can be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can realize certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconstructed. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to realize the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a kind of ASIC, such as an NPU, a TPU, a DPU, etc.

[0192] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0193] In addition, all or part of the units in the above device can be integrated together or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of an SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units of the device. The types of the at least one processor may be different. For example, it includes a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0194] Exemplary electronic device

[0195] Another embodiment of the present application also proposes an electronic device. Refer to Figure 10 As shown, the electronic device may include: a memory 1000 and a processor 1010; wherein, the memory 1000 is connected to the processor 1010 and is used to store programs; the processor 1010 is used to implement the question-and-answer model training method or the industry knowledge question-and-answer method disclosed in any of the above embodiments by running the programs stored in the memory 1000.

[0196] Specifically, the above electronic device may further include: a bus, a communication interface 1020, an input device 1030, and an output device 1040.

[0197] The processor 1010, the memory 1000, the communication interface 1020, the input device 1030, and the output device 1040 are interconnected through the bus. Among them:

[0198] The bus may include a path for transmitting information between various components of the computer system.

[0199] The processor 1010 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or may be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the solution of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] The processor 1010 may include a main processor and may also include a baseband chip, a modem, etc.

[0201] The program for implementing the technical solution of the present application is stored in the memory 1000, and the operating system and other key services may also be stored. Specifically, the program may include program code, and the program code includes computer operation instructions. More specifically, the memory 800 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash memory, and so on.

[0202] The input device 1030 may include devices for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0203] The output device 1040 may include devices for allowing information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0204] The communication interface 1020 may include devices of any transceiver type for communicating with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0205] The processor 1010 executes the program stored in the memory 1000 and calls other devices, and can be used to implement each step of any one of the question-and-answer model training methods or any one of the industry knowledge question-and-answer methods provided in the above embodiments of the present application.

[0206] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads and runs the program stored on the memory through the data interface to execute any one of the question-and-answer model training methods or any one of the industry knowledge question-and-answer methods provided in the above embodiments. For the specific processing process and its beneficial effects, reference may be made to the embodiments of the above question-and-answer model training method or industry knowledge question-and-answer method.

[0207] Exemplary computer program product and storage medium

[0208] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the question-and-answer model training method or the industry knowledge question-and-answer method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0209] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0210] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to perform the steps in the question-and-answer model training method or the industry knowledge question-and-answer method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0211] For the foregoing method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0212] It should be noted that the embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0213] The steps in the methods of the embodiments of the present application can be adjusted, combined, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0214] The modules and sub-modules in the devices and terminals in the embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0215] In several embodiments provided by the present application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or modules can be in the form of electricity, machinery, or other forms.

[0216] The modules or sub-modules described as separate components may or may not be physically separated. The components as modules or sub-modules may or may not be physical modules or sub-modules, that is, they can be located in one place, or they can be distributed to multiple network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional module or sub-module in various embodiments of the present application can be integrated in a processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The above-mentioned integrated modules or sub-modules can be implemented in the form of hardware or in the form of software functional modules or sub-modules.

[0218] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0219] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software units executed by a processor, or a combination of the two. The software units can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0220] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0221] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a question-answering model, characterized in that, Including: Obtain an initial question-and-answer model for a target industry, where the initial question-and-answer model is trained based on first question-and-answer pairs. The question-and-answer form of the first question-and-answer pairs is one question and one answer, and the answers in the first question-and-answer pairs are described in the form of a thought chain; Obtain a question-and-answer reward model for the target industry, where the question-and-answer reward model is trained based on second question-and-answer pairs. The question-and-answer form of the second question-and-answer pairs is one question and multiple answers, and the answers in the second question-and-answer pairs are described in the form of a thought chain; Based on the question-and-answer reward model and the question samples of the target industry, perform reinforcement learning training on the initial question-and-answer model to obtain the knowledge question-and-answer model of the target industry.

2. The Q&A model training method according to claim 1, wherein The training process of the question-and-answer reward model includes: Among the multiple answers included in the second question-and-answer pair, determine the expected answer to the question in the second question-and-answer pair and the non-expected answers to the question in the second question-and-answer pair; Train the question-and-answer reward model according to the question in the second question-and-answer pair, the expected answer, and the non-expected answers.

3. The Q&A model training method according to claim 2, wherein Among the multiple answers, each answer includes multiple thinking steps forming a thought chain and step scores respectively corresponding to the multiple thinking steps; The determining the expected answer to the question in the second question-and-answer pair and the non-expected answers to the question in the second question-and-answer pair among the multiple answers included in the second question-and-answer pair includes: Determine the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers; Determine the expected answer and the non-expected answers among the multiple answers according to the answer scores respectively corresponding to the multiple answers.

4. The method for training a question-and-answer model according to claim 3, wherein The determining the answer scores respectively corresponding to the multiple answers according to the step scores respectively corresponding to the multiple thinking steps in the multiple answers includes: Among the multiple thinking steps included in the first answer, if the step score of the last thinking step is zero, then determine that the answer score of the first answer is zero; If the step score of the last thinking step is not zero, then calculate the mean of the step scores respectively corresponding to the multiple thinking steps included in the first answer, and determine that the answer score of the first answer is the mean; Wherein, the first answer is any answer among the multiple answers.

5. The method for training a question-and-answer model according to claim 3, wherein The determining the expected answer and the non-expected answers among the multiple answers according to the answer scores respectively corresponding to the multiple answers includes: Sort the multiple answers according to the answer scores; Perform answer sampling among the sorted multiple answers; Among the sampled answers, determine the answer with the highest answer score as the expected answer, and determine the answer with the lowest answer score as the non-expected answer.

6. The Q&A model training method according to claim 2, wherein The question-and-answer reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; The training the question-and-answer reward model according to the question in the second question-and-answer pair, the expected answer, and the non-expected answers includes: Train the first reward model according to the question in the second question-and-answer pair, the expected answer, and the non-expected answers; Train the second reward model according to the question in the second question-answer pair, the single thinking step of the expected answer, and the single thinking step of the unexpected answer.

7. The method for training a question and answer model according to claim 6, wherein The training of the first reward model according to the question in the second question-answer pair, the expected answer, and the unexpected answer includes: Input the question and the expected answer in the second question-answer pair into the first reward model, predict the score of the expected answer in the first reward model, and obtain the first predicted score output by the first reward model; Input the question and the unexpected answer in the second question-answer pair into the first reward model, predict the score of the unexpected answer in the first reward model, and obtain the second predicted score output by the first reward model; Input the question and the expected answer in the second question-answer pair into the first prediction model, predict the probability that the answer to the question in the second question-answer pair is the expected answer in the first prediction model, and obtain the first predicted probability output by the first prediction model; Input the question and the unexpected answer in the second question-answer pair into the first prediction model, predict the probability that the answer to the question in the second question-answer pair is the unexpected answer in the first prediction model, and obtain the second predicted probability output by the first prediction model; Determine the first loss value according to the first predicted score, the second predicted score, the first predicted probability, and the second predicted probability; Adjust the parameters of the backbone network of the first reward model according to the first loss value. The first prediction model and the first reward model share the same backbone network.

8. The method for training a question-and-answer model according to claim 6, wherein The training of the second reward model according to the question in the second question-answer pair, the single thinking step of the expected answer, and the single thinking step of the unexpected answer includes: Input the question and the single thinking step of the expected answer in the second question-answer pair into the second reward model, predict the score of the single thinking step of the expected answer in the second reward model, and obtain the third predicted score output by the second reward model; Input the question and the single thinking step of the unexpected answer in the second question-answer pair into the second reward model, predict the score of the single thinking step of the unexpected answer in the second reward model, and obtain the fourth predicted score output by the second reward model; Input the question and the single thinking step of the expected answer in the second question-answer pair into the second prediction model, predict the probability that the answer step to the question in the second question-answer pair is the single thinking step of the expected answer in the second prediction model, and obtain the third predicted probability output by the second prediction model; Input the question and the single thinking step of the unexpected answer in the second question-answer pair into the second prediction model, predict the probability that the answer step to the question in the second question-answer pair is the single thinking step of the unexpected answer in the second prediction model, and obtain the fourth predicted probability output by the second prediction model; Determine a second loss value according to the third prediction score, the fourth prediction score, the third prediction probability, and the fourth prediction probability; Adjust the parameters of the backbone network of the second reward model according to the second loss value. The second prediction model and the second reward model share the same backbone network.

9. The method for training a question-and-answer model according to any one of claims 1 to 8, characterized in that The algorithm used for the reinforcement learning training is the Proximal Policy Optimization algorithm. The initial Q&A model is trained by reinforcement learning based on the Q&A reward model and the question samples in the target industry to obtain the knowledge Q&A model in the target industry, including: Initialize the actor model and the reference model in the training framework of the Proximal Policy Optimization algorithm as the initial Q&A model respectively; Initialize the critic model and the reward model in the training framework as the Q&A reward model respectively; In the training framework, freeze the parameters of the reference model and the reward model, and train the actor model and the critic model based on the loss function of the actor model, the loss function of the critic model, and the question samples; Determine the trained actor model as the knowledge Q&A model.

10. The method for training a question and answer model according to claim 9, wherein The Q&A reward model includes a first reward model for the overall thinking process of the answer and a second reward model for individual thinking steps in the answer; The step of initializing the critic model and the reward model in the training framework as the Q&A reward model respectively includes: Initialize the critic model as the first reward model, and initialize the reward model as the first reward model and the second reward model.

11. The Q&A model training method according to claim 10, wherein The loss function of the actor model includes an advantage variable corresponding to the predicted token output by the actor model. The process of determining the value of the advantage variable includes: Predict the immediate reward generated by the predicted token from a global perspective through the first reward model to obtain the first predicted reward of the predicted token; Predict the immediate reward generated by the predicted token from the perspective of individual thinking steps through the second reward model to obtain the second predicted reward of the predicted token; Determine the advantage variable according to the first predicted reward and the second predicted reward.

12. The Q&A model training method according to any one of claims 1 to 8, characterized in that The generation process of the first Q&A pair and the second Q&A pair includes: Obtain the pre-trained corpus of the target industry; Perform text segmentation and keyword extraction on the pre-trained corpus to obtain the extracted text in the pre-trained corpus and the keywords in the pre-trained corpus; Generate multiple questions through a question generation model according to the extracted text and the keywords; Generate answers corresponding to the multiple questions through an answer generation model with a thought chain instruction template to obtain multiple initial Q&A pairs; Preprocess the multiple initial Q&A pairs to obtain a first Q&A data set and a second Q&A data set. The first Q&A data set is the data set composed of the first Q&A pairs, and the second Q&A data set is the data set composed of the second Q&A pairs; Among them, the Q&A forms of the multiple initial Q&A pairs include one question and one answer, and one question and multiple answers.

13. The Q&A model training method according to claim 12, wherein According to the extracted text and the keyword, a plurality of questions are generated through a question generation model, including: If the keyword appears in the extracted text, the extracted text and the keyword that appears in the extracted text are input into the question generation model; otherwise, the extracted text is input into the question generation model. Question generation is performed in the question generation model to obtain the plurality of questions.

14. The method for training a question-and-answer model according to claim 12, wherein After preprocessing the plurality of initial question-answer pairs to obtain a first question-answer dataset and a second question-answer dataset, it further includes: For each answer in the second question-answer pair, step scores corresponding to the multiple thinking steps included in each answer are obtained and marked in the second question-answer pair.

15. An industry knowledge Q&A method, characterized in that, It includes: Obtain the question to be answered in the target industry; Input the question to be answered into the knowledge question-answer model of the target industry, and generate an answer to the question to be answered through the knowledge question-answer model, where the knowledge question-answer model is trained according to any one of claims 1 to 14. Output the answer to the question to be answered.

16. An electronic device, characterized in that, It includes a memory and a processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the question-answer model training method according to any one of claims 1 to 14 or the industry knowledge question-answer method according to claim 15 by running the program in the memory.

17. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by a processor, it implements the question-answer model training method according to any one of claims 1 to 14 or the industry knowledge question-answer method according to claim 15.

Images