Model training method and device, task execution method and device, electronic equipment and storage medium

Abstract

According to the model training method and device, the task execution method and device, the electronic equipment and the storage medium, in the method, the prompt content, the reply content output by the target model for the prompt content and the process data for generating the reply content can be firstly obtained, and then the prompt content, the reply content and the process data are input into the reward system; therefore, the reward value of each reasoning step in the process data is obtained, and finally, the target model is iteratively trained based on the reward value of each reasoning step. The reward system of the method no longer generates the reward value for the token of the sample, but generates the reward value for each reasoning step in the process data, so that the target model can pay attention to the integrity and logicality of the reply content in the training process, and then the performance and stability of the target model in a complex task and the robustness of the model are improved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a model training, task execution method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, an increasing number of network models are emerging. These models are feature-rich and bring great convenience to users in various aspects of production, daily life, and learning. For example, generative models can output answers based on user-input questions, thereby assisting users in specific fields and tasks and improving work efficiency.

[0003] The quality of the responses generated by a generative model depends to some extent on the quality of its training. In related technologies, training generative models often involves tuning the model parameters based on the reward values ​​of tokens in the samples. This approach leads the generative model to focus on the quality of each token and the total number of tokens, neglecting the overall coherence and logical coherence of the response, which are more important. Therefore, the quality of the responses generated by these models needs improvement. Summary of the Invention

[0004] In view of the above, this specification provides a product recommendation method and apparatus, electronic device and storage medium through one or more embodiments.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a model training method is proposed, the method comprising:

[0007] The system obtains the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process;

[0008] The prompt content, the response content, and the process data are input into the reward system to obtain the reward value for each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to generating the response content;

[0009] The target model is iteratively trained based on the reward value of each inference step in the process data.

[0010] According to a second aspect of one or more embodiments of this specification, a task execution method is provided, the method comprising:

[0011] The system obtains the prompt content and inputs it into a pre-trained target model, so that the target model performs a first chain of reasoning based on the prompt content to obtain alternative responses and the alternative response process data; the target model is trained using the above-described model training method.

[0012] The prompt content, the alternative response content, and the alternative process data are input into the reward system to obtain the reward value for each reasoning step in the alternative process data;

[0013] Determine whether the reward value exceeds a preset reward threshold;

[0014] If so, the alternative response content will be used as the response content corresponding to the prompt content;

[0015] If not, then a second chain of reasoning is performed on the prompt content using the target model to obtain alternative response content and the alternative process data for generating the alternative response content; the second chain of reasoning differs from at least some of the reasoning steps of the first chain of reasoning.

[0016] According to a third aspect of one or more embodiments of this specification, a model training apparatus is provided, the apparatus comprising:

[0017] The acquisition module is used to acquire the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process;

[0018] The reward module is used to input the prompt content, the response content, and the process data into the reward system to obtain the reward value for each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to generating the response content;

[0019] The training module is used to iteratively train the target model based on the reward value of each inference step in the process data.

[0020] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the executable instructions to implement the steps of the request allocation method as described above.

[0021] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the request allocation method as described above.

[0022] According to a sixth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the request allocation method as described above.

[0023] As can be seen from the above embodiments, this specification first obtains the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content. The process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process. Then, the prompt content, response content, and process data can be input into the reward system to obtain the reward value for each reasoning step in the process data. The reward value is used to characterize the contribution of the reasoning step to the generation of the response content. Finally, the target model can be iteratively trained based on the reward value of each reasoning step in the process data.

[0024] In this method, the prompt content, the response output by the target model to the prompt content, and the process data for generating the response content are first obtained. Then, the prompt content, response content, and process data are input into the reward system to obtain the reward value for each inference step in the process data. Finally, the target model is iteratively trained based on the reward value of each inference step. This method's reward system no longer generates reward values ​​for the tokens of the samples, but rather for each inference step in the process data. This allows the target model to focus on the integrity and logic of the response content during training, thereby improving the performance, stability, and robustness of the target model in complex tasks. Attached Figure Description

[0025] Figure 1 This is a flowchart of model training provided in an exemplary embodiment.

[0026] Figure 2 This is a schematic diagram of a method for obtaining sample process data provided in an exemplary embodiment.

[0027] Figure 3 This is a schematic diagram of an intermediate result of a reasoning step provided in an exemplary embodiment.

[0028] Figure 4 This is a schematic diagram of a task execution method provided in an exemplary embodiment.

[0029] Figure 5 An exemplary embodiment provides a schematic diagram of the structure of a device.

[0030] Figure 6 A schematic diagram of a model training apparatus provided in an exemplary embodiment.

[0031] Figure 7A schematic diagram of a model training apparatus provided in an exemplary embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0033] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0034] Based on the aforementioned technical problems mentioned in the background art, at least one embodiment of this specification provides a model training method. This method determines the reward value for each inference step in the process data of generating response content, thereby enabling the generative model to focus on the integrity and logic of the response content during training and improving the quality of the response content generated by the generative model.

[0035] This method can be executed by a computing device, which can be a terminal device such as a desktop computer, laptop computer, tablet computer, laptop computer, or smartphone; or it can be a server, such as a single server, multiple servers, a server cluster, or a cloud computing platform. This specification does not limit the type of computing device. For ease of description, the following section uses a server as the execution subject to provide a detailed explanation of the model training method provided in this specification.

[0036] Please refer to the appendix. Figure 1 The flowchart of the model training method is shown as an example, including the following steps:

[0037] S100: Obtain the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process;

[0038] In this specification, the server can respond to a task request sent by a user, determine the inference task to be executed based on the received task request, and then execute the aforementioned inference task through the target model, returning the execution result to the user. The recommendation task to be executed here can be, for example, intelligent customer service tasks, mathematical problem-solving tasks, report writing tasks, data analysis tasks, etc.

[0039] Prior to this, the target model can be deployed to the server after being trained. When the server trains the target model, it can obtain the prompt content, the response content output by the target model in response to the prompt content, and the process data of generating the response content. Then, the target model can be trained based on the prompt content input to the target model, the response content output by the target model in response to the prompt content, and the process data of generating the response content.

[0040] The server can use the user's input data for the reasoning task to be performed as the prompt content. Alternatively, it can obtain a chain-like reasoning template that matches the task type, and then generate the prompt content based on the matched chain-like reasoning template and the user's input data for the reasoning task.

[0041] The input data mentioned above can be determined according to the actual application scenario. For example, for a reasoning task to be performed, such as report writing, the user's input data could be "Please help me write a report, with the following requirements:...".

[0042] The chain-like reasoning templates described above can be customized to meet specific needs. For example, for a reasoning task like report writing, the template could be: "Outline - First Draft - Verification". Similarly, for a reasoning task like mathematical problem solving, the template could be: "Clarify the Problem's Conditions - Select a Formula - Substitute the Formula for Calculation - Verify the Result".

[0043] As can be seen from the above, each thinking link in chain-like reasoning can form a reasoning step. The chain-like reasoning template mentioned above can be composed of reference reasoning steps corresponding to each thinking link in chain-like reasoning. In other words, the chain-like reasoning template mentioned above can include at least one set of reference reasoning steps required to perform the reasoning task to be performed. Then, prompts can be generated based on the chain-like reasoning template, so that the target model can generate different chain-like reasoning processes based on the chain-like reasoning template.

[0044] The target model mentioned above can be a large language model (LLM). When generating response content based on the input prompts, the target model can perform chain-like reasoning, where each thought process forms a reasoning step.

[0045] It should be noted that during the chain-like reasoning process, the target model can use the intermediate results of each preceding reasoning step as context information for each reasoning step. Based on this context information, the current reasoning step is executed to output its intermediate result, and finally, the response content is generated. The intermediate results of each reasoning step collectively constitute the process data for the target model to output a response to the input prompts.

[0046] S102: Input the prompt content, the response content, and the process data into the reward system to obtain the reward value for each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to generating the response content.

[0047] As can be seen from the above, the quality of the reasoning steps formed in each thinking link of chain-like thinking and their contribution to the answer content will affect the quality of the answer content. Therefore, the server can train the target model based on the prompts input to the target model, as well as the response content output by the target model in response to the prompts and the process data of generating the response content.

[0048] Specifically, the server can input the prompts into the target model, the response output by the target model in response to the prompts, and the process data for generating the response into the reward system, thereby obtaining the reward value for each reasoning step in the process data.

[0049] The reward value mentioned above is used to characterize the contribution of the reasoning step to the generated response, as well as the quality of the reasoning step.

[0050] The aforementioned reward system can be built based on a language model. Here, the language model is used to perform chain-like reasoning on each reasoning step in the process data of generating response content for the target model, so as to generate the thinking content corresponding to the reasoning step, and output the reasoning reward value corresponding to the reasoning step based on the thinking content.

[0051] For example, if the reasoning step is an "outline," the process of chain reasoning for this step by the language model in the reward system could be: "Analyze whether the reasoning step covers the core topic - analyze whether it provides a clear direction of argumentation - comprehensively judge and generate the reward value for the reasoning step." Of course, the process of chain reasoning for this step by the language model in the reward system could also be: "Analyze the rationality of the reasoning step - judge the relevance of the reasoning step to the context - evaluate the necessity and sufficiency of the reasoning step - consider the improvability of the step - comprehensively judge and generate the reward value for the reasoning step."

[0052] As can be seen from the above, the server can use the language model in the reward system to generate response content based on the target model, and perform chain reasoning on the rationality of each reasoning step in the chain reasoning process of the target model in response to the input prompts, so as to obtain the reasoning reward value corresponding to the reasoning step.

[0053] Furthermore, the server can use the language model to perform chain-like reasoning on each step of the target model based on the input prompts, and combine the general knowledge of the language model to perform chain-like reasoning on the reasoning step to obtain the reasoning reward value of the reasoning step, and then iteratively train the target model based on the reasoning reward value of the reasoning step.

[0054] The language model mentioned above can be COT-PRM (Chain-of-thought Process Reward Model), and the language model can be pre-trained to convergence based on sample process data.

[0055] Specifically, the aforementioned sample training data includes sample prompts, sample process data, and sample responses. The sample training data is labeled with the thought content label and inference reward value label for each inference step corresponding to the sample prompts. The sample process data, sample prompts, and sample responses can then be input into the language model to be trained. For each inference step contained in the sample process data, the language model can obtain the thought content and inference reward value for that inference step. Based on the differences between the thought content and the thought content label output by the language model for that inference step, and the differences between the inference reward value and the inference reward value label output by the language model, a first objective loss can be determined. The language model is then iteratively trained with the goal of minimizing this first objective loss.

[0056] Among them, the smaller the difference between the thought content and the thought content label of the reasoning step output by the language model to be trained, and the smaller the difference between the reasoning reward value and the reasoning reward value label of the reasoning step output by the language model to be trained, the smaller the determined first target loss.

[0057] It should be noted that the method for obtaining the sample training data used in the above content may be to collect the user's historical input data and input the collected user historical input data into the pre-trained target model, so as to generate at least one sample response content and sample process data corresponding to each sample response content through the target model.

[0058] Building upon this, to enhance the diversity and accuracy of the training data, the server can also add prompts to guide the target model in self-reflection, thereby acquiring rich process data as sample process data. Specifically, for example... Figure 2 As shown.

[0059] Figure 2 This is a schematic diagram of a method for obtaining sample process data provided in an exemplary embodiment.

[0060] Combination Figure 2 It can be seen that the server can re-input the preset self-test prompts, sample prompts, sample responses, and sample process data into the target model, so that the target model can verify at least some of the reasoning steps in the chain-like reasoning process based on the sample prompts. If the target model determines that there are errors in at least some of the above reasoning steps, it can regenerate the sample process data according to the new thinking direction.

[0061] The self-test prompts described above are used to guide the target model to resample and reflect on the process data of chain-like reasoning based on the sample prompts. These prompts could be, for example: "For each reasoning step in the sample process data, please determine if this step is reasonable? Are there any other possible solutions?" Or, for example: "Please review each reasoning step in the sample process data, and what aspects do you think can be improved?"

[0062] As can be seen from the above, the server can build a reward system based on a language model. This allows the server to process each step of the process data through chain-like reasoning using the language model. Furthermore, the reward value for each reasoning step can be determined based on the thought content obtained from chain-like reasoning. This can improve the reward system's comprehensive evaluation ability of the target model's output, effectively improve the accuracy of the reward value, and thus enhance the quality of the trained target model.

[0063] Based on this, when a language model performs chain-like reasoning on a reasoning step to obtain its reasoning reward value, the core task of the language model is to model the distribution of real natural language. Natural language has the characteristics of being fuzzy, diverse, and context-dependent. Therefore, when obtaining the reasoning reward value of a reasoning step through the language model, it often considers the contribution of the reasoning step to the entire chain-like reasoning process of the target model from a global perspective, as well as the complex dependency relationship between the reasoning step and the context, and gives a reward value with lower discrimination accuracy. This reward value with lower discrimination accuracy reflects the real uncertainty of natural language text and helps to avoid misjudgment.

[0064] For example, for this reasoning step, the reward value obtained by the language model through chain reasoning can be: 0.95 for "very good", 0.65 for "good", "bad" (0.35) and "very bad" (0.15). Among them, the actual rewards of 0.96, 0.97, 0.94 and 0.93 are all regarded as the reward value of 0.95. That is to say, the reward value of 0.95 given by the language model can include a variety of uncertain cases such as 0.96, 0.97, 0.94 and 0.93. Similarly, 0.35 also includes a variety of uncertain cases such as 0.33, 0.30 and 0.40. This uncertainty precisely reflects the uncertainty in real natural language text, such as: "plan" can refer to a noun or a verb, "walk" can refer to walking or a queue, etc.

[0065] However, when training the target model using the reward value obtained from chain-like reasoning based on the language model for this reasoning step, the low precision of the reward value discrimination may lead to unstable results in iterative training of the target model. For example, reinforcement learning training of the target model requires high-precision reward signals.

[0066] Based on this, the aforementioned reward system can also be constructed based on language models and dense reward models. The dense reward model is used to generate fine-grained reward values ​​corresponding to each inference step in the process data, and then the target model can be iteratively trained based on the fine-grained reward values.

[0067] Among them, the discrimination accuracy of the fine-grained reward value is higher than that of the inference reward value generated by the language model. For example, the fine-grained reward value output by the dense reward model can be any one of 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1, while the reward value output by the language model can be any one of 1, 0.5, and 1.

[0068] The aforementioned dense reward model can be a dense-PRM (Process Reward Model) model. This dense reward model is pre-trained to convergence using dense training data. This dense training data includes sample prompts, sample process data, and sample responses. The dense training data is labeled with the inference reward value label for each inference step in the sample process data. The sample prompts, sample process data, and sample responses can then be input into the dense reward model to be trained. This yields the fine-grained reward value output by the dense reward model for each inference step of the sample process data. Based on the difference between the fine-grained reward value output by the dense reward model for that inference step and the inference reward value label for that inference step, a second objective loss is determined. The dense reward model is then iteratively trained with the goal of minimizing this second objective loss.

[0069] The smaller the difference between the fine-grained reward value output by the dense reward model to be trained for this inference step and the inference reward value label for this inference step, the smaller the determined second target loss.

[0070] It should be noted that the aforementioned dense training data can be obtained through manual annotation. Alternatively, the aforementioned dense training data can also be generated based on the language model trained to convergence. In the case where the aforementioned dense training data is generated based on the language model trained to convergence, the server can distill the language model into a dense reward model.

[0071] Furthermore, the aforementioned reward system can be used to weightedly fuse the inference reward value output by the language model for each inference step with the fine-grained reward value output by the dense reward model for that inference step, to obtain the reward value for that inference step. See the following formula for details:

[0072] R = alpha * GRM + beta * PRM

[0073] In the above formula, R is the reward value for the inference step, GRM is the inference reward value output by the language model for the inference step, PRM is the fine-grained reward value output by the dense reward model for the inference step, alpha is the preset weight value of the language model, and beta is the preset weight value of the dense reward model.

[0074] In practical applications, the aforementioned reward system can also be used to post-process the inference reward value output by the language model and the fine-grained reward value output by the dense reward model, and then weighted and summed the post-processed results of the two reward values ​​to obtain the reward value for that inference step. See the following formula for details:

[0075] R = alpha * f(GRM) + beta * g(PRM)

[0076] In the above formula, R is the reward value for this inference step, GRM is the inference reward value output by the language model for this inference step, PRM is the fine-grained reward value output by the dense reward model for this inference step, alpha is the preset weight value of the language model, beta is the preset weight value of the dense reward model, f is the processing function used for post-processing the inference reward value output by the language model, which can be an activation function, a truncation function, etc., and g is the processing function used for post-processing the fine-grained reward value output by the dense reward model, which can be an activation function, a truncation function, etc.

[0077] As can be seen from the above, on the one hand, the server can use a language model to perform chain-like reasoning for each step in the process data, and determine the reasoning reward value based on the thinking content of the chain-like reasoning, thereby improving the comprehensive evaluation ability of the reward system for the output of the target model. On the other hand, the server can output more granular reward values ​​through a dense reward model, and thus comprehensively consider the reasoning reward value and the granular reward value, thereby effectively improving the accuracy of the reward value determined for each reasoning step, and thereby improving the quality of the trained target model.

[0078] S104: Based on the reward value of each inference step in the process data, iteratively train the target model.

[0079] It should be noted that the intermediate results of each reasoning step include multiple tokens, specifically as follows: Figure 3 As shown.

[0080] Figure 3 This is a schematic diagram of an intermediate result of a reasoning step provided in an exemplary embodiment.

[0081] Combination Figure 2 It can be seen that for each inference step, the target model can generate a corresponding intermediate result, which consists of at least one token.

[0082] Based on this, the server can iteratively train the target model based on the log probability of each inference step and the reward value of each inference step in the process data. For each inference step in the process data, the log probability of that inference step can be determined based on the probability of each token that makes up the intermediate results of that inference step.

[0083] The method for determining the logarithmic probability mentioned above can be referenced by the following formula:

[0084]

[0085] In the above formula, Inference step A is generated within context C. t The probability, P(a) i |a1,……a i-1 ) is in context C and the generated token a1, ... a i-1 Generate tokens under the following conditions i The probability. Context C contains the hints and intermediate results of the generated reasoning steps.

[0086] Furthermore, the server can adjust the network parameters of the target model according to the PPO (Proximal Policy Optimization) optimization strategy, based on the log probability of each inference step and the reward value of each inference step in the process data mentioned above, in order to iteratively train the target model.

[0087] Specifically, the server can use KL divergence to constrain the difference in probabilities before and after the inference step optimization. Based on this, the network loss, advantage function, etc. are determined according to the reward value of each inference step to determine the optimization direction of the network parameters and adjust the network parameters of the target model, thereby iteratively training the target model.

[0088] As can be seen from the above, the server can calculate the logarithmic probability of each inference step as a whole, thus enabling KL divergence constraints and optimization direction determination at the step level. This allows the target model to focus on the logic between inference steps during training, improving the logic and coherence of the generated responses. Compared to token-level policy optimization, step-level PPO here focuses more on the integrity and coherence of the chain-like reasoning process, enabling the trained target model to exhibit higher stability and robustness when dealing with more complex and diverse tasks.

[0089] In addition, after training the target model using the above model training methods to obtain the trained target model, the model parameters of the trained target model can be frozen (i.e., the model parameters of the target model are restricted from being updated when the task is executed), and then the task can be executed through the target model.

[0090] To facilitate understanding, the following provides a detailed explanation of the process of executing a task using the target model trained through the above model training method. Specifically... Figure 4 As shown.

[0091] Figure 4This is a schematic diagram of a task execution method provided in an exemplary embodiment, including the following steps:

[0092] S400: Obtain the prompt content and input the prompt content into the pre-trained target model so that the target model performs a first chain of reasoning based on the prompt content to obtain alternative response content and alternative process data for generating the alternative response content; the target model is trained by the above model training method.

[0093] S402: Input the prompt content, the alternative response content, and the alternative process data into the reward system to obtain the reward value for each reasoning step in the alternative process data.

[0094] S404: Determine whether the reward value exceeds a preset reward threshold.

[0095] S406: If so, the alternative response content shall be used as the response content corresponding to the prompt content.

[0096] S408: If not, then a second chain of reasoning is performed on the prompt content through the target model to obtain alternative response content and alternative process data for generating the alternative response content; the second chain of reasoning differs from at least some of the reasoning steps of the first chain of reasoning.

[0097] As can be seen from the above, the server can perform a first-chain reasoning process using the target model, generating alternative responses and their process data. Then, a reward system calculates a reward value for each reasoning step to assess the quality of the reasoning process. If the reward value exceeds a reward threshold, the alternative response is directly adopted as the final response. If it does not exceed the threshold, the model is triggered to perform a second-chain reasoning process, making adjustments to at least some reasoning steps and regenerating the alternative responses and their process data. This ensures that the model's output responses are not only accurate but also that the reasoning process is reasonable and reliable. Furthermore, through continuous iterative optimization, the model's reasoning ability and adaptability are improved, enhancing its overall intelligence and task execution performance.

[0098] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0099] Please refer to Figure 6 Model training devices can be applied to, for example Figure 5 The technical solution described herein is implemented on the device shown. The model training apparatus may include:

[0100] The acquisition module 601 is used to acquire the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process.

[0101] The reward module 602 is used to input the prompt content, the response content and the process data into the reward system to obtain the reward value of each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to the generation of the response content;

[0102] The training module 603 is used to iteratively train the target model based on the reward value of each inference step in the process data.

[0103] Optionally, the acquisition module 601 is specifically used to acquire a chain-like reasoning template; the chain-like reasoning template includes at least one set of reference reasoning steps required to perform the reasoning task to be performed; generate prompt content according to the chain-like reasoning template; input the prompt content into the target model to obtain the response content output by the target model in response to the prompt content and the process data for generating the response content.

[0104] Optionally, the reward system is built based on a language model, which is used to perform chain-like reasoning on each reasoning step in the process data to generate the thinking content corresponding to the reasoning step, and output the reasoning reward value corresponding to the reasoning step based on the thinking content; the thinking content is used to reflect the rationality of the reasoning step in the chain-like reasoning process.

[0105] Optionally, the language model is pre-trained to convergence based on sample training data, wherein the sample training data includes sample prompts, sample process data, and sample responses, and the sample training data is labeled with the thinking content label and reasoning reward value label for each reasoning step in the sample process data.

[0106] Optionally, the reward system is constructed based on the language model and the dense reward model, wherein the dense reward model is used to generate fine-grained reward values ​​corresponding to each inference step in the process data, and the discrimination accuracy of the fine-grained reward values ​​is higher than the discrimination accuracy of the inference reward values ​​generated by the language model.

[0107] Optionally, the dense reward model is pre-trained to convergence based on dense training data; the dense training data is generated based on the language model trained to convergence.

[0108] Optionally, the reward system is used to perform a weighted fusion of the reasoning reward value output by the language model for each reasoning step and the fine-grained reward value output by the dense reward model for the same reasoning step, so as to obtain the reward value for that reasoning step.

[0109] Optionally, the training module 603 is specifically used to adjust the network parameters of the target model based on the log probability of each inference step and the reward value of each inference step in the process data; wherein the log probability of the inference step is determined based on the probability of each token in the intermediate results.

[0110] Please refer to Figure 7 Task execution devices can be applied to, for example Figure 5 The device shown is used to implement the technical solution of this specification. The task execution device may include:

[0111] The first reasoning module 701 is used to acquire the prompt content and input the prompt content into a pre-trained target model, so that the target model performs a first chain of reasoning based on the prompt content to obtain alternative response content and alternative process data for generating the alternative response content; the target model is trained by a model training method.

[0112] Evaluation module 702 is used to input the prompt content, the alternative response content and the alternative process data into the reward system to obtain the reward value for each reasoning step in the alternative process data;

[0113] The judgment module 703 is used to determine whether the reward value exceeds a preset reward threshold;

[0114] The determination module 704 is used to, if yes, use the alternative response content as the response content corresponding to the prompt content;

[0115] The second reasoning module 705 is used to, if not, perform a second chain-like reasoning on the prompt content through the target model to obtain alternative response content and generate alternative process data for the alternative response content; the second chain-like reasoning differs from at least some of the reasoning steps of the first chain-like reasoning.

[0116] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0117] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0118] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A model training method, the method comprising: The system obtains the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process; The prompt content, the response content, and the process data are input into the reward system to obtain the reward value for each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to generating the response content; The target model is iteratively trained based on the reward value of each inference step in the process data.

2. The model training method as described in claim 1, wherein obtaining the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content specifically includes: Obtain a chain-thinking reasoning template; The chain-based reasoning template includes at least one set of reference reasoning steps required to perform the reasoning task to be performed. Based on the chain-like reasoning template, generate prompts; The prompt is input into the target model to obtain the response output by the target model in response to the prompt and the process data for generating the response.

3. The model training method as described in claim 1, wherein the reward system is constructed based on a language model, the language model is used to perform chain-like reasoning on each reasoning step in the process data to generate the thinking content corresponding to the reasoning step, and output the reasoning reward value corresponding to the reasoning step based on the thinking content; the thinking content is used to reflect the rationality of the reasoning step in the chain-like reasoning process.

4. The model training method as described in claim 3, wherein the language model is pre-trained to convergence based on sample training data, wherein, The sample training data includes sample prompts, sample process data, and sample responses. The sample training data is labeled with the thinking content tag and reasoning reward value tag for each reasoning step in the sample process data.

5. The model training method according to any one of claims 1-4, wherein the reward system is constructed based on a language model and a dense reward model, wherein, The dense reward model is used to generate fine-grained reward values ​​corresponding to each inference step in the process data. The discrimination accuracy of the fine-grained reward values ​​is higher than that of the inference reward values ​​generated by the language model.

6. The model training method as described in claim 5, wherein the dense reward model is pre-trained to convergence based on dense training data; the dense training data is generated based on the language model trained to convergence.

7. The model training method as described in claim 5, wherein the reward system is used to perform a weighted fusion of the inference reward value output by the language model for each inference step and the fine-grained reward value output by the dense reward model for the inference step, so as to obtain the reward value for the inference step.

8. The model training method as described in claim 1, wherein the target model is iteratively trained based on the reward value of each inference step in the process data, specifically including: The network parameters of the target model are adjusted based on the log probability of each inference step and the reward value of each inference step in the process data; wherein the log probability of the inference step is determined based on the probability of each token in the intermediate results.

9. A task execution method, the method comprising: Obtain the prompt content and input the prompt content into the pre-trained target model so that the target model performs a first chain of reasoning based on the prompt content to obtain alternative response content and alternative process data for generating the alternative response content; The target model is obtained by training the method described in any one of claims 1 to 8 above; The prompt content, the alternative response content, and the alternative process data are input into the reward system to obtain the reward value for each reasoning step in the alternative process data; Determine whether the reward value exceeds a preset reward threshold; If so, the alternative response content will be used as the response content corresponding to the prompt content; If not, then the target model is used to perform a second chain of reasoning on the prompt content to obtain alternative response content and generate alternative response content data. The second chain-like reasoning differs from at least some of the reasoning steps in the first chain-like reasoning.

10. A model training apparatus, the apparatus comprising: The acquisition module is used to acquire the prompt content, the response content output by the target model in response to the prompt content, and the process data for generating the response content, wherein the process data includes the intermediate results output by the target model at each reasoning step during the chain-like reasoning process; The reward module is used to input the prompt content, the response content, and the process data into the reward system to obtain the reward value for each reasoning step in the process data, wherein the reward value is used to characterize the contribution of the reasoning step to generating the response content; The training module is used to iteratively train the target model based on the reward value of each inference step in the process data.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

Images