Model training method and text generation method

Abstract

This specification provides a model training method and a text generation method. The model training method includes: acquiring multiple sample task processing flows and the processing results of each sample task. Each sample task processing flow includes at least one task processing step, and each task processing step includes a sample task action and a sample task state. The sample task action is determined by an initial first policy model based on the sample task state. Based on the processing results of each sample task, first parameter tuning information corresponding to each sample task processing flow is determined. Based on a target second policy model, second parameter tuning information corresponding to each task processing step is acquired. Based on the first and second parameter tuning information, the initial first policy model is trained to obtain the target first policy model. This method can improve model training efficiency and the accuracy of the trained model.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to model training methods and text generation methods. Background Technology

[0002] Reinforcement learning can be used to train agents, enabling them to learn how to take actions to maximize long-term cumulative rewards through interaction with the environment. The policy model is a core component of the agent, used to determine the appropriate action given a particular environmental state. This action then applies to the given environmental state, achieving a state transition and obtaining the next environmental state.

[0003] Generally, in multi-round interactive tasks or multi-step reasoning tasks, the policy model needs to make multiple decisions to obtain the final task result. Therefore, when training the policy model, in addition to the reward information corresponding to the final task result, the reward information corresponding to each decision can also be used to train the policy model. If only the reward information corresponding to the final task result is considered as the supervision signal for training the policy model, the trained policy model will be inaccurate. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a model training method. One or more embodiments of this specification simultaneously relate to a model training method applied to a cloud-side device, a model training method applied to a text interaction task, a text generation method, a text generation method applied to a cloud-side device, a model training device, a model training device applied to a cloud-side device, a model training device applied to a text generation task, a text generation device, a text generation device applied to a cloud-side device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising:

[0006] Obtain multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0007] Based on the processing results of each sample task, determine the first parameter tuning information corresponding to the processing flow of each sample task;

[0008] Based on the target second strategy model, obtain the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow;

[0009] Based on the first and second parameter tuning information, the initial first policy model is trained to obtain the target first policy model.

[0010] According to a second aspect of the embodiments of this specification, a model training method is provided, applied to a cloud-side device, comprising:

[0011] The receiving end device sends multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0012] Based on the processing results of each sample task, determine the first parameter tuning information corresponding to the processing flow of each sample task;

[0013] Based on the target second strategy model, obtain the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow;

[0014] Based on the first and second parameter tuning information, the initial first policy model is trained to obtain the target first policy model.

[0015] Obtain the model parameters of the target first strategy model and send the model parameters to the end-side device.

[0016] According to a third aspect of the embodiments of this specification, a model training method is provided for a text interaction task, comprising:

[0017] Multiple sample text interaction processes and corresponding sample text interaction results are obtained. The sample text interaction process includes at least one round of sample text interaction. Each round of sample text interaction includes a sample text interaction action and a sample text interaction state. The sample text interaction state includes the first input text input to the initial first strategy model. The sample text interaction action includes the first response text output by the initial first strategy model. The sample text interaction action is determined by the initial first strategy model based on the sample text interaction state.

[0018] Based on the text interaction results of each sample, determine the first parameter tuning information corresponding to the text interaction process of each sample;

[0019] Based on the target second strategy model, obtain the second parameter tuning information corresponding to each round of sample text interaction in the sample text interaction processing flow;

[0020] Based on the first and second parameter tuning information, train the initial first policy model to obtain the target first policy model.

[0021] According to a fourth aspect of the embodiments of this specification, a text generation method is provided, comprising:

[0022] Get the second input text;

[0023] The second input text is input into the target first policy model to obtain the second response text corresponding to the second input text, wherein the target first policy model is trained by the above-described model training method.

[0024] According to a fifth aspect of the embodiments of this specification, a text generation method is provided, applied to a cloud-side device, comprising:

[0025] The second input text sent by the receiving device;

[0026] The second input text is input into the target first policy model to obtain the second response text corresponding to the second input text, wherein the target first policy model is trained by the above-described model training method;

[0027] The second response text is sent to the end device.

[0028] According to a sixth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0029] Memory and processor;

[0030] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0031] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising:

[0032] Memory and processor;

[0033] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0034] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0035] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0036] One embodiment of this specification can obtain multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. Each sample task processing flow includes at least one task processing step, and each task processing step includes a sample task action and a sample task state. The sample task action is determined by an initial first policy model based on the sample task state. First parameter tuning information corresponding to each sample task processing flow is determined based on the sample task processing results. Second parameter tuning information corresponding to each sample task processing step in each sample task processing flow is obtained based on a target second policy model. The initial first policy model is trained based on the first and second parameter tuning information to obtain the target first policy model.

[0037] By using the above method, we can obtain the first parameter tuning information for each sample task processing flow and the second parameter tuning information for each sample task processing step obtained through the target second policy model. By combining the two parameter tuning information to train the initial first policy model, we can consider both the parameter tuning trend reflected in each sample task processing flow and the parameter tuning trend reflected in each task processing step in each sample task processing flow. This can solve the problem of credit allocation difficulty in multi-round interactive tasks and multi-step inference tasks, thereby improving the rationality of credit allocation in multi-round interactive tasks and multi-step inference tasks, improving training efficiency, and improving the accuracy of the trained target first policy model. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a model training method provided in one embodiment of this specification;

[0039] Figure 2 This is a schematic diagram of data flow for training an initial first strategy model, provided in one embodiment of this specification.

[0040] Figure 3 This is a schematic diagram of data flow for training an initial first strategy model, provided in one embodiment of this specification.

[0041] Figure 4 This is a flowchart illustrating an initial second strategy model training method provided in one embodiment of this specification;

[0042] Figure 5 This is a schematic diagram of data flow for training an initial second strategy model, provided in one embodiment of this specification.

[0043] Figure 6 This is a flowchart illustrating a model training method provided in one embodiment of this specification;

[0044] Figure 7 This is a flowchart illustrating a model training method provided in one embodiment of this specification;

[0045] Figure 8 This is a flowchart illustrating a text generation method provided in one embodiment of this specification;

[0046] Figure 9 This is a flowchart illustrating a text generation method provided in one embodiment of this specification;

[0047] Figure 10 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification;

[0048] Figure 11 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification;

[0049] Figure 12 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification;

[0050] Figure 13 This is a schematic diagram of the structure of a text generation device provided in one embodiment of this specification;

[0051] Figure 14 This is a schematic diagram of the structure of a text generation device provided in one embodiment of this specification;

[0052] Figure 15 This is an architecture diagram of a model training system provided in one embodiment of this specification;

[0053] Figure 16 This is a structural block diagram of an electronic device provided in one embodiment of this specification;

[0054] Figure 17 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0055] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0056] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0057] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0058] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0059] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0060] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0061] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0062] Multi-turn interaction tasks: These can be multi-turn reasoning tasks, referring to the process of continuous, context-based information exchange between two intelligent agents (such as human and machine, or machine and machine). For example, a multi-turn interaction task can be a text interaction task, where the information is text. In each round of text interaction, either party can refer to the text interaction content of all previous rounds and / or the input text of the current round to understand and generate response text.

[0063] Multi-step reasoning tasks: Tasks that require multiple logical deductions, thinking, or information processing steps. For example, each mathematical calculation in a mathematical reasoning task can be considered a reasoning step, or in a text interaction task, the generation of the response text corresponding to each round of interaction can also be considered a reasoning step. Furthermore, any single round of text interaction in a multi-round text interaction task can be considered a multi-step reasoning task. Specifically, in the process of generating the response text corresponding to any round of text interaction, the generation of a token, word, or sentence in the response text can be considered an information processing step, i.e., a reasoning step.

[0064] Reinforcement learning is a machine learning paradigm in which an agent learns how to take actions to maximize cumulative rewards by interacting with its environment. The core concepts of reinforcement learning include state, action, reward, and policy. The agent continuously improves its policy through trial and error to achieve better performance on a given task; this learning method performs well in multi-step interactive tasks.

[0065] Policy Model: In reinforcement learning, a policy model is a model that makes decisions about actions based on states. A policy refers to the mapping from states to actions. Policy models can be applied to different tasks; different tasks require different actions and states. For example, in a text interaction task, the policy model can be a large language model. The input text, context, or prompts can be considered states, and the next output, i.e., the response text such as a token, sentence, or reply, can be considered an action. Text interaction tasks include, but are not limited to, question-and-answer tasks and intelligent customer service. In one or more embodiments of this specification, for any given task, the corresponding state and action can be set based on actual needs.

[0066] Reward Model: The reward model is used to evaluate the quality of the output of the policy model. That is, the input to the reward model includes the output of the policy model, and the output includes an evaluation value or reward value. This reward value characterizes the quality of the policy model's output, or in other words, it characterizes the contribution of the policy model's output to achieving the target result corresponding to the task. In one or more embodiments of this specification, in a multi-round interactive task, the output of the policy model input to the reward model can be the result corresponding to each round of the interactive task, or it can be the result corresponding to each step of reasoning in each round of the interactive task. In other words, the reward model evaluates each round of the interactive task, or it can evaluate each step of reasoning in each round of the interactive task. The reward model evaluating each round of the interactive task can be a result-based reward model, and the reward model evaluating each step of reasoning in each round of the interactive task can be a process-based reward model. It should be understood that the process-based reward model can also be used to evaluate each round of the interactive task.

[0067] Credit allocation: In multi-turn interactive tasks or multi-step inference tasks, credit allocation refers to the reasonable distribution of the final reward or punishment signal to the various actions or steps that lead to the outcome. Reasonable and effective credit allocation enables the policy model to learn policies that positively impact the outcome and reduces negative policies, thereby accelerating the convergence of the policy model and improving its training efficiency.

[0068] In multi-round interactive tasks (such as text interaction tasks, page navigation tasks, etc.), policy models can be trained not only based on the reward information corresponding to the final task result of each round of the interactive task, but also based on the reward information corresponding to each inference step in each round of the interactive task. In practical applications, the evaluation criteria for the reward information corresponding to the final task result and the reward information corresponding to each inference step are different, that is, the reward information corresponding to the final task result and the reward information corresponding to each inference step have different impacts on the first policy model. Therefore, in the process of training the policy model, how to effectively allocate credits, improve the accuracy of the trained policy model, and improve the training efficiency of the policy model are key issues.

[0069] Based on this, one embodiment of this specification can obtain multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. Each sample task processing flow includes at least one task processing step, and each task processing step includes a sample task action and a sample task state. The sample task action is determined by an initial first policy model based on the sample task state. According to the processing results of each sample task, first parameter tuning information corresponding to each sample task processing flow is determined. According to the target second policy model, second parameter tuning information corresponding to each task processing step in each sample task processing flow is obtained. Then, based on the first and second parameter tuning information, the initial first policy model can be trained to obtain the target first policy model.

[0070] By using the above method, we can obtain the first parameter tuning information of each sample task processing flow and the second parameter tuning information corresponding to each sample task processing step obtained through the second strategy model related to the sample task. By combining the two parameter tuning information to train the initial first strategy model, we can consider both the parameter tuning trend reflected by each sample task processing flow and the parameter tuning trend reflected by each task processing step in each sample task processing flow. This can solve the problem of credit allocation difficulty in multi-round interactive tasks and multi-step inference tasks, thereby improving the rationality of credit allocation in multi-round interactive tasks and multi-step inference tasks, improving training efficiency, and improving the accuracy of the trained target first strategy model.

[0071] This specification provides a model training method. It also relates to a model training method for cloud-based devices, a model training method for text interaction tasks, a text generation method, a text generation method for cloud-based devices, a model training device, a model training device for cloud-based devices, a model training device for text generation tasks, a text generation device, a text generation device for cloud-based devices, an electronic device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0072] See Figure 1 , Figure 1 A flowchart illustrating a model training method according to an embodiment of this specification is shown, specifically including the following steps.

[0073] Step 102: Obtain multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0074] In one or more embodiments of this specification, the execution subject of the model training method can be any computing device with computing capabilities, such as a terminal or server, and this specification does not impose any specific restrictions on it.

[0075] In this specification, the computing device can acquire multiple sample task processing flows and acquire the sample task processing results corresponding to each sample task processing flow. Each sample task processing flow includes multiple task processing steps, and each task processing step includes a sample task action and the sample task state corresponding to that action.

[0076] It should be noted that the model training method provided in this specification can be applied to multi-turn interaction tasks or multi-step reasoning tasks, and the sample tasks can be multi-turn interaction tasks or multi-step reasoning tasks. In one or more embodiments of this specification, multi-turn interaction tasks can be text interaction tasks or page navigation tasks. Text interaction tasks include, but are not limited to, intelligent customer service, question-and-answer tasks, etc. Page navigation tasks include, but are not limited to, page form filling tasks and user interface testing tasks. Multi-step reasoning tasks can be mathematical reasoning tasks, text generation tasks, etc.

[0077] It should be understood that the model training methods provided in this manual are also applicable to single-round interactive tasks and single-step inference tasks.

[0078] In practical applications, different tasks require different state and action spaces, which can be set according to actual needs. For example, for text interaction tasks, the state space may include user input text, historical context, or prompt text, while the action space may include tokens, sentences, and replies in the response text generated based on the state space. For page navigation tasks, the state space may include the layout of elements on the page, text content, button positions, and operable areas, while the action space may include actions such as clicking, searching, entering keywords, and scrolling to the bottom of the page. For mathematical reasoning tasks, such as solving algebraic equations, geometric proofs, or logical deductions, the state space may include the initial mathematical problem, the intermediate answers or conclusions already deduced, the current reasoning context, and prompt text, while the action space may include executing each logical calculation and reasoning step, such as squaring both sides, extracting common factors, and multiplying large numbers.

[0079] In one or more embodiments of this specification, as described above, the sample task can be a text interaction task, a page navigation task, a mathematical reasoning task, etc., and the sample task processing flow is the process of executing the sample task, consisting of at least one ordered sample task processing step. For example, the sample task is a text interaction task, which refers to the task of generating a response text corresponding to the received input text. Executing the sample task includes: starting from the first receipt of input text, generating response text, and then receiving the next round of input text, generating response text again, until the task termination condition is reached. The task termination condition can be that the sample task processing time reaches a preset time threshold, or that no input text can be obtained, etc. Thus, during the execution of the sample task, sample task processing steps consisting of at least one input text and response text can be obtained, and the sample task processing steps constitute the sample task processing flow in chronological order. Simply put, the sample task processing flow is a task processing trajectory consisting of several sample task states and the sample task actions corresponding to the sample task states.

[0080] It should be noted that the sample task processing steps include the sample task state and the sample task action corresponding to the sample task state. The sample task action corresponding to the sample task state is the sample task action generated by the initial first strategy model based on the sample task state. As mentioned above, different tasks have different state spaces and action spaces, which can be set according to actual needs.

[0081] In one or more embodiments of this specification, the sample task processing flow can be obtained from an existing sample database or obtained by sampling through an initial first policy model. This specification aims to train a high-performance target first policy model that can decide the sample task action corresponding to the sample task state based on the sample task state. Therefore, in this specification, obtaining the sample task processing flow through the initial first policy model can achieve a closed loop between the model training method and the model inference method, thereby improving the performance of the trained target first policy model.

[0082] In the initial first strategy model, when sampling the sample task processing flow, the initial first strategy model can output the sample task action corresponding to the current sample task state based on the current sample task state, and continuously interact with the environment to obtain the next sample task state corresponding to the current sample task state through the environment, and then determine the next sample task action, gradually completing the sample task and obtaining the sample task processing flow.

[0083] It should be noted that the initial first policy model can both perform action selection (decision-making) and participate in the state transition process. In this specification, the initial first policy model is configured to follow an interaction paradigm of first perceiving the current state, then deciding on the current action based on the current state, and finally influencing the environment to obtain the next state. It should also be noted that the next sample task state is usually determined by the environment or system simulator. The specific process for determining the next sample task state is a well-known technique in the field of reinforcement learning, and will not be elaborated upon in this specification.

[0084] It should be understood that the sample task processing flow includes both the flow for successfully executed sample tasks and the flow for unsuccessfully executed sample tasks. Furthermore, the processing steps within the sample task processing flow are arranged in chronological order.

[0085] The sample task processing result may include reward information obtained by evaluating the sample task processing flow. In this specification, the reward information obtained by evaluating the sample task processing flow is denoted as the first reward information, which may specifically include a first reward value. In practical applications, this first reward information can be obtained manually or through an artificial intelligence model such as a large language model, based on different evaluation criteria. For example, the sample task processing flow and / or the final sample task state corresponding to the sample task processing flow can be input into an artificial intelligence model to obtain the first reward value corresponding to the sample task processing flow. The different evaluation criteria include, but are not limited to, the completion rate of the sample task, the accuracy of the sample task execution, etc.

[0086] Of course, the sample task processing results may also include the sample task processing flow itself and the final sample task status obtained when the sample task is completed.

[0087] It should be noted that the model training method provided in this manual is a one-round training. Therefore, after performing one round of training, the target first policy model can be used as the initial first policy model again, and the model training method provided in this manual can be used to train the newly obtained initial first policy model until the model training stopping condition is met, and the target first policy model corresponding to the final training round is obtained.

[0088] Therefore, in one or more embodiments of this specification, where the sample task processing flow is obtained by sampling through the initial first policy model, for each training round, the computing device can first determine the initial state of the sample task corresponding to that training round, sample multiple sample task processing flows based on the initial state of the sample task using the initial first policy model, and obtain the sample task processing results corresponding to each sample task processing flow. Then, based on the sample task processing results corresponding to the sample task processing flows, the initial first policy model is trained to obtain the target first policy model, and the target first policy model is used again as the initial first policy model. Based on the newly obtained initial first policy model, multiple sample task processing flows are resampled to perform the next round of training on the newly obtained initial first policy model, until the model training stopping condition is reached, and the target first policy model corresponding to the final training round is obtained.

[0089] It should be noted that the conditions for stopping model training include, but are not limited to, the number of iterations reaching a preset number, the calculated loss being less than a preset loss, and the number of training samples used reaching a preset number, etc.

[0090] Step 104: Based on the processing results of each sample task, determine the first parameter tuning information corresponding to the processing flow of each sample task.

[0091] The first parameter tuning information is used to evaluate the sample task processing flow, and it may include evaluation information for evaluating the sample task processing flow. The first parameter tuning information can reflect the overall quality of the sample task processing flow.

[0092] As mentioned above, the sample task processing result may include first reward information, which may specifically be a reward value. As previously mentioned, this first reward information is used to evaluate the sample task processing flow. Therefore, in one or more embodiments of this specification, first parameter tuning information corresponding to each sample task processing flow can be obtained based on each first reward information.

[0093] Specifically, when obtaining the first parameter tuning information corresponding to each sample task processing flow based on the first reward information, the first reward information corresponding to each sample task processing flow can be used as the first parameter tuning information corresponding to that sample task processing flow. Alternatively, for each sample task processing flow, the difference between the first reward information corresponding to that sample task processing flow and the first reward information of the preset reward information can be calculated, and then the difference in the first reward information can be used as the first parameter tuning information corresponding to that sample task processing flow. This specification does not impose specific limitations here.

[0094] More specifically, the first reward information includes a first reward value. Therefore, the first parameter tuning information may include the first reward value. That is, for each sample task processing flow, the first reward value corresponding to that sample task processing flow can be added to the first parameter tuning information corresponding to that sample task processing flow. The first parameter tuning information may also include a first reward difference. That is, for each sample task processing flow, the first reward difference between the first reward value corresponding to that sample task processing flow and the preset reward value is calculated, and then the first reward difference is added to the first parameter tuning information corresponding to that sample task processing flow.

[0095] Based on the sample task processing results corresponding to each sample task processing flow, the specific method for determining the first parameter tuning information corresponding to each sample task processing flow is not specifically limited in this manual. The first parameter tuning information only needs to reflect the quality and performance of the sample task processing flow.

[0096] In one or more embodiments of this specification, the first parameter tuning information may include a reward value or advantage value to reflect the performance and completion effect of the sample task, thereby guiding the optimization training of the initial first policy model. Specifically, in reinforcement learning, the first parameter tuning information may be used as a trajectory-level reward signal to train the parameters of the initial first policy model, so as to improve the performance of the trained target first policy model in future interactions.

[0097] In one or more embodiments of this specification, a standard sample task processing flow and a standard sample task processing result corresponding to the sample task can also be set. Thus, the first parameter tuning information characterizes the difference between the sample task processing result corresponding to the sample task processing flow and the standard sample task processing result; or, in other words, the first parameter tuning information characterizes the difference between the sample task processing flow and the standard sample task processing flow. The smaller the difference between the sample task processing result corresponding to the sample task processing flow and the standard sample task processing result, and the smaller the difference between the sample task processing flow and the standard sample task processing flow, the higher the overall quality of the sample task processing flow.

[0098] It should be noted that the standard sample task processing results and standard sample task processing flow can be obtained manually or through a large language model. For example, for a text interaction task, the initial state of the sample task is input text 1. After the text interaction task is completed, the sample task processing flow A is obtained: input text 1 and response text 1, input text 2 and response text 2. The sample task processing result A may include response text 2 and / or sample task processing flow A. Therefore, for this text interaction task, the standard sample task processing flow B corresponding to this text interaction task can be obtained through a large language model: input text 1, response text 3, input text 2, response text. Thus, the standard task processing result B corresponding to this text interaction task may include standard sample task processing flow B and / or response text 2.

[0099] Step 106: Based on the target second strategy model, obtain the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow.

[0100] It should be noted that the objective second policy model is related to the sample task. The objective second policy model can be used to evaluate the quality of the output of the initial first policy model. Specifically, the objective second policy model can be used to evaluate each sample task action output by the initial first policy model, or in other words, to evaluate each sample task step obtained through the initial first policy model, and to obtain the reward information corresponding to the sample task step. That is to say, the second policy model in this specification can be a process reward model. In this specification, the reward information corresponding to the sample task step is denoted as the second reward information.

[0101] The second parameter tuning information is used to evaluate the sample task processing steps or the sample task actions corresponding to the sample task processing steps. The second parameter tuning information can reflect the local quality of the sample task processing steps or sample task processing actions.

[0102] In one or more embodiments of this specification, when obtaining the second parameter tuning information corresponding to each sample task processing step in each sample task processing flow according to the target second strategy model, for each sample task processing step in any sample task processing flow, the second reward information corresponding to that sample task processing step can be obtained through the target second strategy model. For example, the sample task state corresponding to that sample task processing step can be input into the target second strategy model, or the sample task state corresponding to that sample task processing step and the sample task processing steps preceding that sample task processing step can be jointly input into the target second strategy model to obtain the second reward information corresponding to that sample task processing step. The second reward information corresponding to that sample task processing step can be used as the second parameter tuning information corresponding to that sample task processing step, or the difference between the second reward information corresponding to that sample task processing step and the preset reward information can be calculated, and then the difference in the second reward information can be used as the second parameter tuning information corresponding to that sample task processing flow. This specification does not impose specific limitations here.

[0103] In one or more embodiments of this specification, the second reward information may include a second reward value. Therefore, the first parameter tuning information may include the second reward value; that is, for each sample task processing step, the second reward value corresponding to that sample task processing step may be added to the second parameter tuning information corresponding to that sample task processing step. The second parameter tuning information may also include a second reward difference; that is, for each sample task processing flow, the second reward difference between the second reward value corresponding to that sample task processing step and a preset reward value is calculated, and then the second reward difference is added to the second parameter tuning information corresponding to that sample task processing step.

[0104] It should be noted that the target second policy model is a model related to the sample task, and it is a reward model that evaluates each sample task processing step in the sample task processing flow. For example, the target second policy model can be obtained by training an initial second policy model based on the sample task processing flow and the corresponding sample task processing results. The sample training data used to train the initial second policy model can be the same as or different from the sample training data used to train the initial first policy model, as long as it is sample training data from that sample task.

[0105] It is worth noting that in one or more embodiments of this specification, the target policy model may be a pre-trained model or a non-pre-trained model. Thus, during the training of the initial first policy model, the initial second policy model may also be trained according to the sample task processing flow and the sample task processing results corresponding to the sample task processing flow.

[0106] It should be noted that, in one or more embodiments of this specification, step-level reward information, i.e., second reward information, can be obtained based on the output probability of the target second strategy model. The second reward information level includes a second reward value. Specifically, the output probability of the target second strategy model and the second reward value are positively correlated; the higher the output probability, the higher the second reward value, and vice versa. In other words, the output probability of the target second strategy model can characterize the second reward value corresponding to the sample task action and / or sample task processing flow. Therefore, in essence, this target second strategy model can be referred to as a process reward model that evaluates each task processing step in the sample task processing flow.

[0107] In other words, in this specification, the target second policy model is a process reward model. Based on the process reward model, step-level reward information corresponding to any task processing step in any sample task processing flow can be obtained. The aforementioned sample task processing results include trajectory-level reward information for evaluating any sample task processing flow. Specifically, this trajectory-level reward information can be obtained based on artificial intelligence models such as large language models or manually. In subsequent steps, the initial first policy model is trained based on the trajectory-level reward information and the step-level reward information, thereby improving the accuracy of obtaining the target first policy model and increasing training efficiency.

[0108] In one or more embodiments of this specification, the second parameter tuning information may include a reward value or an advantage value to reflect the execution status and completion effect of each sample task processing step in the sample task processing flow, thereby guiding the optimization training of the initial first policy model. Specifically, in reinforcement learning, the second parameter tuning information can be used as a step-level reward signal to train the parameters of the initial first policy model in order to improve the performance of the trained target first policy model in future interactions.

[0109] Step 108: Train the initial first policy model based on the first parameter tuning information and the second parameter tuning information to obtain the target first policy model.

[0110] In one or more embodiments of this specification, a loss value can be determined based on first and second parameter tuning information, thereby allowing the initial second policy model to be trained based on the loss value to obtain a target first policy model.

[0111] In one or more embodiments of this specification, when determining the loss value based on the first and second parameter tuning information, for each sample task processing step, the second parameter tuning information corresponding to that sample task processing step and the first parameter tuning information corresponding to the sample task processing flow in which that sample task processing step is located can be obtained. Thus, the target parameter tuning information corresponding to that sample task processing step can be obtained based on the second and first parameter tuning information, and the target loss value can be determined based on the target parameter tuning information corresponding to each sample task processing step. Specifically, for each sample task processing step, the sum of the second parameter tuning information corresponding to that sample task processing step and the first parameter tuning information corresponding to the sample task processing flow in which that sample task processing step is located can be used as the target parameter tuning information corresponding to that sample task processing step.

[0112] In one or more embodiments of this specification, the first parameter tuning information includes a first parameter tuning value, which may be a first reward value; the second parameter tuning information includes a second parameter tuning value; and the target parameter tuning information includes a target parameter tuning value. Therefore, the sum or weighted sum of the first parameter tuning value and the second parameter tuning value can be used to calculate the target parameter tuning value.

[0113] In one or more embodiments of this specification, the initial second strategy model can be trained with the goal of increasing the probability that the output of the initial first strategy model corresponds to the sample task processing step corresponding to the first target parameter tuning value and decreasing the probability that the output of the initial first strategy model corresponds to the sample task processing step corresponding to the second target parameter tuning value. The first target parameter tuning value is greater than the second target parameter tuning value, or in other words, the first target parameter tuning value is greater than a preset parameter tuning value, and the second target parameter tuning value is not greater than the preset parameter tuning value. The preset parameter tuning value can be set according to actual needs.

[0114] In practical applications, the target parameter tuning value provides direction for training the initial first-policy model. A loss function can be constructed based on this. The principle behind constructing the loss function is that a larger target parameter tuning value encourages the initial first-policy model to output the sample task processing step corresponding to that target parameter tuning value, while a smaller target parameter tuning value inhibits the initial first-policy model from outputting the sample task processing step corresponding to that target parameter tuning value. Therefore, by minimizing the loss value of the pre-constructed loss function, the probability of the initial first-policy model deciding on sample task processing steps with larger target parameter tuning values ​​can be increased, while the probability of the initial first-policy model deciding on sample task processing steps with smaller target parameter tuning values ​​can be reduced. It should be noted that this specification does not restrict the specific structure of the pre-constructed loss function; it can be set according to actual needs, as long as the aforementioned principles are met.

[0115] In one or more embodiments of this specification, training the initial first policy model to obtain the target first policy model based on the target parameter tuning information corresponding to each sample task processing step includes: for each sample task processing step, inputting the sample task state corresponding to that sample task processing step and a second sample task processing step into the initial first policy model to obtain a second probability output by the initial first policy model, wherein the second sample task processing step is a sample task processing step preceding the current sample task processing step in the sample task processing flow, and the second probability is the probability that the result output by the initial first policy model is the sample task action corresponding to that sample task processing step. The initial first policy model is trained based on the second probability corresponding to each sample task processing step and the target parameter tuning information to obtain the target first policy model.

[0116] In one or more embodiments of this specification, training the initial first policy model to obtain the target first policy model based on the second probability corresponding to each sample task processing step and the target parameter tuning information includes: for each sample task processing step, calculating the local loss value corresponding to that sample task processing step based on the second probability corresponding to that sample task processing step and the target parameter tuning information; obtaining the global loss value based on the local loss value corresponding to each sample task processing step; and training the initial first policy model based on the global loss value to obtain the target first policy model.

[0117] Specifically, in one or more embodiments of this specification, for each sample task processing step, the product of the second probability corresponding to that sample task processing step and the target parameter tuning value can be calculated to obtain the local loss value corresponding to that sample task processing step. Finally, the global loss value can be obtained based on the local loss values ​​corresponding to each sample task processing step. Specifically, when calculating the global loss value based on the calculated local loss values ​​corresponding to each task processing step in each sample task processing flow, a weighted average of the local loss values ​​corresponding to each task processing step in each sample task processing flow can be performed to obtain the global loss value. Of course, other methods can also be used to calculate the global loss value based on the local loss values, and this specification does not impose specific limitations.

[0118] It should be noted that the local loss value in this specification is the loss value corresponding to each task processing step. This local loss value is jointly determined by the second probability corresponding to each sample task processing step and the target parameter tuning information, and then the global loss value can be obtained. More refined supervision signals can be used to improve the accuracy of credit allocation and improve the accuracy of the target first policy model.

[0119] As mentioned earlier, the above model training method is a single-round training. Therefore, after one round of training, the target first policy model can be used again as the initial first policy model, and the model training method provided in this description can be used to train the newly obtained initial first policy model until the model training stopping condition is met, thus obtaining the target first policy model corresponding to the final training round. The specific model training stopping condition will not be elaborated here. Figure 2 As shown, Figure 2 This document provides a schematic diagram of the data flow for training an initial first strategy model.

[0120] By using the above model training method, we can obtain the first parameter tuning information of each sample task processing flow and the second parameter tuning information corresponding to each sample task processing step obtained by the second strategy model related to the sample task. By combining the two parameter tuning information to train the initial first strategy model, we can consider both the parameter tuning trend reflected by each sample task processing flow and the parameter tuning trend reflected by each task processing step in each sample task processing flow. This can solve the problem of credit allocation difficulty in multi-round interactive tasks and multi-step inference tasks, thereby improving the rationality of credit allocation in multi-round interactive tasks and multi-step inference tasks, improving training efficiency, and improving the accuracy of the trained target first strategy model.

[0121] Furthermore, it eliminates the need for complex intermediate reward function design, allowing training solely based on the sample task processing results and the target second policy model for each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the structure of the first policy model and avoiding training instability issues caused by value function bias.

[0122] In practical applications, the above model training method can be used to train policy models for performing different tasks. The task states and actions vary depending on the task. For example, a sample task could be a text generation task, which refers to generating a response text corresponding to the received user input text. This text generation task could specifically be a human-computer interaction task such as an intelligent customer service system or a multi-turn question-and-answer system. The processing flow of this sample task would then be an interactive process of responding to the user's initial input text until conditions such as the user stopping text input are met. The sample task state could be the user input text, historical dialogue context, or prompt text, and the sample task action could be the text content output by the initial first policy model, which could be the next token, a sentence, or a reply. Furthermore, the text content output by the initial first-strategy model is added to the historical dialogue context to obtain a new historical dialogue context, or the user's input text for a new round of responses based on the text content output by the initial first-strategy model is added to the historical dialogue context to obtain a new historical dialogue context, thus obtaining the next sample task state. This process continues until conditions such as the user stopping input, the sample task objective being achieved, or the maximum number of interaction rounds being reached are met, ending the interaction and obtaining a sample task processing flow. The sample task processing result corresponding to the sample task processing flow can be obtained manually or through an artificial intelligence model such as a large language model. It should be noted that when the sample task is a human-computer interaction task scenario, the initial first-strategy model can be an autoregressive language model, such as a large language model based on the Transformer architecture. This initial first-strategy model can generate responses token by token based on the current context and form a closed-loop interaction with the user environment.

[0123] For example, a sample task could be a mathematical reasoning task, such as solving algebraic equations, geometric proofs, or logical deductions. The processing flow for this sample task would then be a process of performing logical calculations and reasoning based on an initial input mathematical problem until conditions such as obtaining the final answer to the initial mathematical problem are met. The sample task state can include the initial mathematical problem, intermediate answers or conclusions already derived, the current reasoning context, and preset prompt text. Sample task actions include each logical calculation and reasoning step executed by the initial first strategy model, such as squaring both sides, factoring out common factors, and multiplying large numbers. Furthermore, the result of the current logical calculation and reasoning step is updated in the reasoning context to obtain the next sample task state, until conditions such as obtaining the final answer, reasoning failure, or exceeding preset steps are met, at which point the logical calculation and reasoning ends, resulting in a sample task processing flow. The sample task processing result can be obtained manually or through an artificial intelligence model such as a large language model. It should be noted that when the sample task is a mathematical reasoning task scenario, the initial first strategy model can be a reasoning model with chain-of-thought (CoT) capability, and the text content output by this reasoning model is the logical calculation and reasoning steps.

[0124] For example, the sample task could be a page navigation task, such as automatically navigating to a target webpage, completing an online form, or executing a target UI operation. The sample task processing flow could then be a process of navigating from the initial page to the target page, or a process of completing a specified page operation based on the initial page. The sample task state could be the layout of elements on the page, text content, button positions, and operable areas, etc., and could be the operation instructions output by the initial first strategy model, such as clicking, searching, entering keywords, or scrolling to the bottom. Furthermore, executing the current operation instruction obtains the next page to update the current page state and obtain the next sample task state, until conditions such as navigation to the target page, navigation recognition, or navigation timeout are met, resulting in a sample task processing flow. The sample task processing result can be obtained manually or through an artificial intelligence model such as a large language model. It should be noted that in the scenario where the sample task is a page navigation task, the initial first strategy model can be a user interface agent (UI Agent), which can autonomously operate the graphical interface.

[0125] Furthermore, in reinforcement learning, advantage is a metric that measures the superiority of an action relative to the average expected reward. In other words, advantage is a core tool in reinforcement learning for measuring the relative benefit of an action; its core idea is to quantify the additional benefit gained by a specific action compared to the average behavior. Therefore, to further improve the accuracy of the trained first policy model, advantage can be introduced. Specifically, in one or more embodiments of this specification, the first parameter tuning information includes a first parameter tuning value that can be a first advantage value.

[0126] In one or more embodiments of this specification, determining the first parameter tuning information corresponding to each sample task processing flow based on the processing results of each sample task includes: determining a first sample task processing flow and a second sample task processing flow, wherein the first sample task processing flow is any one of the plurality of sample task processing flows, and the second sample task processing flow is a sample task processing flow other than the first sample task processing flow among the plurality of sample task processing flows. Based on the sample task processing results corresponding to the second sample task processing flow, determining the first benchmark information corresponding to the first sample task processing flow. Based on the first benchmark information and the sample task processing results corresponding to the first sample task processing flow, determining the first parameter tuning information corresponding to the first sample task processing flow.

[0127] In one or more embodiments of this specification, the first benchmark information represents the average reward information of the first reward information corresponding to the second sample task processing flow, and is referred to as the first average reward information in this specification. Specifically, the first average reward information may be the average reward value of the first reward value corresponding to the second sample task processing flow, that is, the first average reward information includes the average reward value of the first reward value corresponding to the second sample task processing flow.

[0128] In one or more embodiments of this specification, the first benchmark information includes first average reward information of the first reward information corresponding to the second sample task processing flow. Determining the first parameter tuning information corresponding to the first sample task processing flow based on the first benchmark information and the sample task processing results corresponding to the first sample task processing flow includes: determining the first reward information difference corresponding to the first sample task processing flow based on the first average reward information and the first reward information corresponding to the first sample task processing flow; and determining the first parameter tuning information corresponding to the first sample task processing flow based on the first reward information difference.

[0129] As mentioned above, the sample task processing result includes first reward information corresponding to the sample task processing flow, and the first reward information includes a first reward value. In practical applications, the advantage value needs to be determined based on the reward value. Therefore, in one or more embodiments of this specification, the first benchmark information includes the first average reward information of the first reward information corresponding to the second sample task processing flow. Specifically, the first average reward value of the first reward value corresponding to each second sample task processing flow can be calculated, and the difference between the first average reward value and the first reward value corresponding to the first sample task processing flow can be calculated to obtain the first parameter tuning value corresponding to the first sample task processing flow.

[0130] In other words, for each sample task processing flow, first benchmark information can be determined based on the sample task processing results corresponding to each sample task processing flow other than the current sample task processing flow. The first benchmark information includes a first average reward value. Based on the first average reward value and the sample task processing result corresponding to the current sample task processing flow, first parameter tuning information corresponding to that sample task processing result is determined. Specifically, based on the first reward value corresponding to each sample task processing flow other than the current sample task processing flow, a first average reward value is determined. Based on the first average reward value and the first reward value corresponding to the current sample task processing flow, a first parameter tuning value corresponding to the current sample task processing flow is determined. In one or more embodiments of this specification, for each sample task processing flow, the average reward value of the first reward values ​​corresponding to each sample task processing flow other than the current sample task processing flow can be calculated and used as the first average reward value. The difference between the first average reward value and the first reward value corresponding to the current sample task processing flow can be calculated and used as the first parameter tuning value corresponding to the current sample task processing flow.

[0131] In one or more embodiments of this specification, Formula 1 can be used to determine the first parameter tuning value. Formula 1 is shown below:

[0132] Formula 1

[0133] in, Let be the first reward value corresponding to the i-th sample task processing flow, i.e., the first reward value corresponding to the first sample task processing flow. The first sample task processing flow can be any one of the multiple sample task processing flows. Let $\frac{i}{i}$ be the first reward value corresponding to any sample task processing flow other than the $i$-th sample task processing flow, i.e., the first reward value corresponding to the second sample task processing flow. The second sample task processing flow is any sample task processing flow other than the first sample task processing flow among the multiple sample task processing flows. This refers to the first parameter tuning value corresponding to the i-th sample task processing flow (the first sample task processing flow), where G is the total number of sample task processing flows. The first baseline information is represented by the first average reward value corresponding to the first reward value of all second sample task processing flows.

[0134] Based on the above method, for each sample task processing flow, the first parameter tuning information corresponding to the sample task processing flow can be obtained according to the first reward information corresponding to other sample task processing flows. Advantage values ​​are introduced so that the initial first policy model can be trained based on each first advantage value in subsequent steps. This can accelerate model convergence, improve training efficiency, and improve the accuracy of the trained target first policy model.

[0135] It should be noted that, for the sake of clarity in the formulas of this scheme, it will be consistently stated here that r represents the second reward value, and the initial second strategy model adopts... This indicates that the reward value determined based on the probability output by the initial second policy model is used... This indicates that the second-strategy model for objective purposes adopts... This indicates that the reward value determined based on the probability output by the objective second policy model is used... The first reward value is denoted by R, the sample task processing flow is denoted by e, the total number of sample task processing flows is G, the task processing steps are denoted by y, the actions corresponding to the task processing steps are denoted by a, and the states corresponding to the task processing steps are denoted by s. The baseline policy model adopts... The first parameter tuning value, also known as the first dominant value, is indicated. This indicates that the second parameter tuning information, also known as the second dominant value, is adopted. express.

[0136] As mentioned above, the advantage value can also be introduced when obtaining the second parameter tuning information corresponding to each sample task processing step through the second strategy model. Specifically, in one or more embodiments of this specification, the second parameter tuning information may include a second advantage value.

[0137] In one or more embodiments of this specification, obtaining the second parameter tuning information corresponding to each sample task processing step in each sample task processing flow according to the target second strategy model includes: for each sample task processing step in any sample task processing flow, obtaining the second reward information corresponding to that sample task processing step through the target second strategy model; and obtaining the second parameter tuning information corresponding to that sample task processing step based on the second reward information corresponding to each sample task processing step.

[0138] In one or more embodiments of this specification, obtaining the second reward information corresponding to the sample task processing step through the target second strategy model includes: obtaining a first sample task processing step preceding the sample task processing step; inputting the sample task state corresponding to the sample task processing step and the first sample task processing step into the target second strategy model to obtain the second reward information corresponding to the sample task processing step, wherein the second reward information is determined according to a first probability, the first probability being the probability that the result output by the target second strategy model is the sample task action corresponding to the sample task processing step.

[0139] In other words, for each sample task processing flow, and for each sample task processing step within that flow, the sample task processing step preceding the current sample task processing step is designated as the first task processing step. Then, the sample task state corresponding to this first sample task processing step and the first sample task processing step are input into the target second policy model to obtain the second reward information corresponding to that step. This second reward information is determined by a first probability, which is the probability that the output of the target second policy model is the sample task action corresponding to that step. Thus, the second reward information corresponding to each sample task processing step can be obtained. Next, based on the second reward information corresponding to each first sample task processing step, the second baseline information corresponding to that step can be obtained. Based on the second reward information and the second baseline information, the second parameter tuning information corresponding to that step can be obtained. Thus, the second parameter tuning information corresponding to each sample task processing step in each sample task processing flow can be obtained.

[0140] As mentioned above, in one or more embodiments of this specification, the reward information includes a reward value that is positively correlated with the output probability of the target second strategy model. This specification does not impose specific restrictions on the method of determining the reward value based on the probability. For example, the probability can be directly used as the reward value, or the logarithm of the probability can be taken to obtain the reward value, or the probability can be normalized and then the logarithm can be taken to obtain the reward value, etc.

[0141] In one or more embodiments of this specification, obtaining second parameter tuning information corresponding to a sample task processing step based on the second reward information corresponding to each sample task processing step includes: determining second baseline information corresponding to the sample task processing step based on the second reward information corresponding to sample task processing steps other than the sample task processing step; and obtaining the second parameter tuning information corresponding to the sample task processing step based on the second baseline information and the second reward information corresponding to the sample task processing step.

[0142] The second average reward information includes a second average reward value. This second average reward information may include the average reward value of the second reward values ​​corresponding to the sample task processing steps excluding the sample task step in the sample task processing flow; in this specification, it is denoted as the second average reward value.

[0143] Therefore, in one or more embodiments of this specification, for each sample task processing flow and for each sample task processing step in the sample task processing flow, the second benchmark information corresponding to the sample task processing step in the sample task processing flow represents the average reward information of the first reward information corresponding to the sample task processing steps other than the sample task step in the sample task processing flow, and is referred to as the second average reward information in this specification.

[0144] In one or more embodiments of this specification, obtaining second parameter tuning information corresponding to a sample task processing step based on second benchmark information and second reward information includes: determining the difference in second reward information based on the second average reward information and the second reward information corresponding to the sample task processing step; and obtaining the second parameter tuning information corresponding to the sample task processing step based on the difference in second reward information. Specifically, for each sample task processing flow, and for each sample task processing step in the sample task processing flow, the difference between the second reward value and the second average reward value corresponding to the sample task processing step in the sample task processing flow can be calculated to obtain the second parameter tuning value corresponding to the sample task processing step in the sample task processing flow.

[0145] In other words, for each sample task processing flow, and for each sample task processing step within that flow, the difference between the second average reward value and the second reward value corresponding to that sample task processing step is calculated. This difference is then used as the second parameter tuning value for that sample task processing flow. Thus, the second parameter tuning values ​​for each sample task processing step within each sample task processing flow can be obtained.

[0146] In one or more embodiments of this specification, Formula 2 can be used to obtain the second advantage value corresponding to any sample task processing step in any sample task processing flow. Formula 2 is shown below:

[0147] Formula 2

[0148] in, The second reward value represents the processing step of the k-th sample task in the processing flow of the i-th sample task, and G is the total number of sample task processing flows. This refers to the second parameter tuning value corresponding to the k-th sample task processing step in the i-th sample task processing flow. The i-th sample task processing flow can be any one of multiple sample task processing flows. W is the set of all sample task processing steps in the i-th sample task processing flow, excluding the k-th sample task processing step. Let be the second reward value corresponding to the m-th sample task processing step in set W, which is also the second reward value corresponding to any sample task processing step in set W. Let n be the total number of sample task processing steps in the i-th sample task processing flow, excluding the k-th sample task processing step, which is also the total number of sample task processing steps contained in set W. This represents the second average reward value.

[0149] The second reward value is determined based on the second probability output by the target second policy model. In one or more embodiments of this specification, .in, Let represent the second reward value corresponding to the k-th sample task processing step in the i-th sample task processing flow, that is, the second reward value corresponding to any sample task processing step in any sample task processing flow. This indicates that the second reward value is determined based on the second probability of the sample task action corresponding to the k-th sample task processing step in the i-th sample task processing flow, as output by the target second policy model. The output of the second strategy model is the second probability of the action of the sample task corresponding to the k-th sample task processing step in the i-th sample task processing flow. The output of the baseline strategy model is the probability of the sample task action corresponding to the k-th sample task processing step in the i-th sample task processing flow, where β is a preset parameter.

[0150] It should be noted that the baseline policy model may not participate in the training process. The structure and parameters of this baseline policy model can be the same as those of the initial first policy model before training begins. This baseline policy model can also be a model pre-trained based on manually generated rules or sample datasets. Alternatively, the baseline policy model can participate in the training process, in which case a moving average algorithm, such as the exponential moving average algorithm, can be used to adjust the parameters of the baseline policy model based on the parameters of the target first policy model for each training round. This specification does not impose specific restrictions on this; settings can be made based on actual needs.

[0151] Based on the above method, an advantage value is introduced when obtaining the second parameter tuning information. This advantage value is then used to train the initial first policy model in subsequent steps, accelerating model convergence, improving training efficiency, and increasing the accuracy of the trained target first policy model. Based on the determination of the first and second parameter tuning values, the first and second advantage values ​​are fused to obtain the target parameter tuning value, or target advantage value, which is then used to train the initial first policy model, thereby improving the accuracy of the trained target first policy model.

[0152] In practical applications, a baseline policy model is introduced for regularization to prevent behavioral breakdowns during policy updates, thereby improving the rationality and consistency of the generated results. Therefore, in step 108 above, a baseline policy model can also be introduced to constrain the training of the initial first policy model, improving training stability and the accuracy of the trained model.

[0153] In one or more embodiments of this specification, calculating the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step and the target parameter tuning information includes: obtaining a first probability distribution output by the initial first policy model, wherein the first probability distribution is the sampling probability of each sample task action of the initial first policy model under the sample task state corresponding to the sample task processing step; obtaining a benchmark policy model, and inputting the sample task state corresponding to the sample task processing step and the second sample task processing step into the benchmark policy model to obtain a second probability distribution, wherein the second probability distribution is the sampling probability of each sample task action of the benchmark policy model under the sample task state corresponding to the sample task processing step, and the second sample task processing step is the sample task processing step preceding the sample task processing step in the sample task processing flow; calculating the difference between the first probability distribution and the second probability distribution to obtain the distribution difference value corresponding to the sample task processing step; and calculating the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step, the target parameter tuning information, and the distribution difference value.

[0154] It should be noted that the baseline policy model may not participate in the training process. The structure and parameters of this baseline policy model can be the same as those of the initial first policy model before training begins. This baseline policy model can also be a model pre-trained based on manually generated rules or sample datasets. Alternatively, the baseline policy model can participate in the training process, in which case a moving average algorithm, such as the exponential moving average algorithm, can be used to adjust the parameters of the baseline policy model based on the parameters of the target first policy model for each training round. This specification does not impose specific restrictions on this; settings can be made based on actual needs.

[0155] It should be understood that by inputting the sample task state into the initial first policy model, one can obtain each selectable action in the pre-constructed action space output by the initial first policy model and its corresponding probability, i.e., the first probability distribution. Of course, one can also obtain the probability of a certain sample task action from the output of the initial first policy model. This applies to all policy models in this specification; that is, based on the policy model, one can obtain each possible outcome output by the policy model and its corresponding probability (i.e., probability distribution), and also obtain the probability of a specified outcome output by the policy model.

[0156] In one or more embodiments of this specification, the local loss value corresponding to the sample task processing step is calculated based on the second probability corresponding to the sample task processing step, the target parameter tuning information, and the distribution difference value. This includes: for each sample task processing step, calculating the product of the second probability corresponding to the sample task processing step and the target parameter tuning value, and calculating the first difference between the product and the distribution difference value corresponding to the sample task processing step to obtain the local loss value corresponding to the sample task processing step.

[0157] In reinforcement learning, Kullback-Leibler divergence (KL divergence) is often used to compare the similarity or difference between policy distributions. In practical applications, the difference between the first probability distribution and the second probability distribution can be considered the KL divergence. It should be noted that existing algorithms can be used to calculate the difference between the two distributions, and these will not be elaborated upon here.

[0158] Based on the foregoing, this baseline policy model provides a baseline comparison for training the initial first policy model, keeping the model parameters of the initial first policy model within a credible range. This avoids catastrophic deviations and prevents the forgetting of historical knowledge, thereby improving training stability and the accuracy of the trained target first policy model. Specifically, when determining the local loss, the difference between the first probability distribution and the second probability distribution is added, thus preventing the new policy from differing too much from the old policy during policy updates and improving training stability.

[0159] In one or more embodiments of this specification, data from historical training rounds may be introduced to constrain the current training round to improve training stability. Therefore, based on the second probability corresponding to the sample task processing step and the target parameter tuning information, the local loss value corresponding to the sample task processing step is calculated, including: obtaining the third probability corresponding to the sample task processing step, wherein the third probability is the historical second probability from the historical training rounds corresponding to the sample task processing step. Based on the second probability, the third probability, and the target parameter tuning information corresponding to the sample task processing step, the local loss value corresponding to the sample task processing step is calculated.

[0160] Specifically, the third probability is the probability of the sample task action corresponding to the sample task processing step in the historical training rounds, after the sample task state and the second sample task processing step are input into the initial first policy model corresponding to the historical training rounds. This output of the initial first policy model is the probability of the sample task action corresponding to the sample task processing step, which is the historical second probability. In one or more embodiments of this specification, to avoid excessive differences in model parameters that could affect training stability, the historical training rounds can be the previous training round before the current training round, using the second probability from the previous training round. This constrains the current training round while ensuring training stability.

[0161] In one or more embodiments of this specification, the local loss value corresponding to the sample task processing step is calculated based on the second probability, the third probability, and the target parameter tuning information. This includes: calculating a first ratio between the second probability and the third probability corresponding to the sample task processing step, and calculating the product of the first ratio and the target parameter tuning value to obtain the local loss value corresponding to the sample task processing step.

[0162] Based on the second probability corresponding to each task processing step, and the product of the historical second probability and the target advantage value, policy gradient updates are performed. This provides more refined supervision signals for the training of the initial first policy model, thereby improving the accuracy of credit allocation and the accuracy of the resulting target first policy model. Furthermore, using data from previous training rounds—that is, the historical second probabilities—to constrain the current round of training can improve training stability.

[0163] Furthermore, based on the foregoing, a benchmark policy model can be introduced for regularization to avoid behavioral breakdowns during policy updates, thereby improving the rationality and consistency of the generated results. In one or more embodiments of this specification, to further improve the refinement of the policy update supervision signal, when calculating the local loss value corresponding to each sample task processing step based on the second probability, third probability, and target parameter tuning information, the local loss value can be calculated based on the distribution difference value, second probability, third probability, and target parameter tuning information corresponding to the sample task processing step. Specifically, the first ratio of the second probability and the third probability can be calculated, as well as the product of the first ratio and the target parameter tuning value, and the local loss value can be calculated based on the product and the distribution difference value.

[0164] As mentioned above, in one or more embodiments of this specification, when calculating the global loss value based on the calculated local loss values ​​corresponding to each task processing step in each sample task processing flow, the local loss values ​​corresponding to each task processing step in each sample task processing flow can be averaged or weighted to obtain the global loss value. Of course, other methods can also be used to calculate the global loss value, and this specification does not impose specific limitations.

[0165] In one or more embodiments of this specification, Formula 3 can be used to determine the global loss value. Formula 3 is shown below:

[0166]

[0167] Where G represents the total number of sample task processing steps. This represents the total number of task processing steps in the processing flow of the i-th sample task. This refers to the k-th task processing step in the processing flow of the i-th sample task. This refers to the task processing steps preceding the k-th task processing step in the i-th sample task processing flow. The output of the initial first policy model in the current training round is The probability, The output of the initial first policy model in the previous training round is The probability of. and All parameters are preset. The difference in probability distribution between the initial first policy model and the baseline policy model is called the KL divergence. The first advantage value corresponding to the k-th task processing step in the i-th sample task processing flow. The second advantage value is the value corresponding to the i-th sample task processing flow where the k-th task processing step is located. Let be a function that calculates the target advantage value based on the first advantage value and the second advantage value, i.e., the target advantage value corresponding to the k-th task processing step in the i-th sample task processing flow. The function for calculating the target advantage value can be: That is, the weighted advantage value of the first advantage value and the second advantage value can be used as the target advantage value, as mentioned above. The weight is , The weight is 1. The settings can be configured according to actual needs. Alternatively, the sum of the first and second advantage values ​​can be directly used as the target advantage value. In other words, the function for calculating the target advantage value is... This manual does not impose specific limitations on this.

[0168] Based on the model training method described above, which incorporates historical training data, a baseline strategy model, and KL divergence, this manual provides a data flow diagram. (See also...) Figure 3 , Figure 3 This is a schematic diagram of data flow for training an initial first policy model, provided in one embodiment of this specification. The model training method, which incorporates historical training data, a baseline policy model, and KL divergence, can determine local and global loss values ​​through distribution difference values, second probabilities, third probabilities, and target parameter tuning information. This further improves the refinement of the supervision signal, enhances the accuracy of credit allocation, and ultimately improves the accuracy of the resulting target first policy model.

[0169] In addition, as mentioned above, the target second policy model is used to provide step-level reward information, and the target second policy model can be a pre-trained model related to the sample task, or it can be a model trained based on the sample training data used by the first policy model.

[0170] It should be noted that in this specification, in each round of training, the first policy model before parameter adjustment is the initial first policy model, and the first policy model after adjustment is the target policy model. Correspondingly, the second policy model before parameter adjustment is the initial second policy model, and the second policy model after adjustment is the target policy model.

[0171] In practical applications, it is difficult to obtain training samples for the second strategy model, resulting in limited generalization ability and insufficient compatibility with the first strategy model. Therefore, in one or more embodiments of this specification, the first strategy model and the second strategy model can be trained in a coordinated manner.

[0172] In one or more embodiments of this specification, a method for obtaining a target second policy model is also provided. See also Figure 4 , Figure 4 This is a flowchart illustrating an initial second policy model training method according to an embodiment of this specification. The target second policy model is obtained by training using the following method:

[0173] Step 402: Based on the processing results of each sample task, divide the sample task processing flow to obtain the third sample task processing flow and the fourth sample task processing flow.

[0174] Step 404: Obtain the initial state of the sample task, input the initial state of the sample task into the initial second strategy model, and train the initial second strategy model according to the preset training objective to obtain the target second strategy model. The preset training objective includes increasing the probability that the output result of the initial second strategy model is the third sample task processing flow and decreasing the probability that the output result of the initial second strategy model is the fourth sample task processing flow.

[0175] In one or more embodiments of this specification, the sample task processing flow can be divided into a third sample task processing flow and a fourth sample task processing flow based on the sample task processing results. Specifically, the sample task processing flow can be divided into two types based on a preset reward threshold, or it can be divided into two types manually.

[0176] As mentioned earlier, the sample task processing result includes first reward information, which includes a first reward value. In practical applications, the sample task processing result can be a binary classification result; for example, the reward value can be 0 or 1, where 0 indicates a failed processing flow and 1 indicates a successful processing flow. The sample task processing result can also be a non-binary classification result; for example, the reward value can be any value between 0 and 100, with a higher reward value indicating more accurate and efficient execution of the sample task. Regardless of the sample task processing result, it can be divided into two types based on actual needs and circumstances, resulting in two sample task processing flows.

[0177] For example, the processing result of a sample task with a first reward value greater than a preset reward threshold can be determined as the first sample task processing result, the sample task processing flow corresponding to the first sample task processing result can be used as the third sample task processing flow, the processing result of a sample task with a first reward value not greater than the preset reward threshold can be determined as the second sample task processing result, and the sample task processing flow corresponding to the second sample task processing result can be used as the fourth sample task processing flow. In this way, the first sample task processing result and the second sample task processing result, as well as the third sample task processing flow and the fourth sample task processing flow can be obtained.

[0178] For example, the sample task processing flow includes flow A, flow B, and flow C. Assuming that the sample task processing results obtained by the evaluation model for flow A, flow B, and flow C are 30 points, 60 points, and 90 points respectively, and the preset reward threshold is 60 points, then it can be determined that the first sample task processing result is the sample task processing result corresponding to 90 points, the first sample task processing result is the sample task processing result corresponding to 30 points, and the first sample task processing result is the sample task processing result corresponding to 60 points. Therefore, the first sample task processing flow is flow C, and the second sample task processing flow is flow A and flow B.

[0179] It should be understood that the third sample task processing flow is one or more of the multiple sample task processing flows, and the fourth sample task processing flow is one or more of the multiple sample task processing flows.

[0180] In one or more embodiments of this specification, the sample task processing result corresponding to the third sample task processing flow is better than the sample task processing result corresponding to the fourth sample task processing flow. Specifically, the first reward value corresponding to the third sample task processing flow is greater than a preset reward threshold, while the first reward value corresponding to the fourth sample task processing flow is not greater than the preset reward threshold. Alternatively, the first reward value corresponding to the third sample task processing flow is greater than the first reward value corresponding to the fourth sample task processing flow. Or, the sample task processing result corresponding to the third sample task processing flow can be a sample task processing flow with a higher reward value, while the sample task processing result corresponding to the fourth sample task processing flow can be a sample task processing flow with a lower reward value; that is, the sample task processing result corresponding to the third sample task processing flow is better than the sample task processing result corresponding to the fourth sample task processing flow.

[0181] Furthermore, the initial state of the sample task can be obtained. It should be noted that this initial state is the current sample task state, used to enable the policy model to decide the action to be performed in the current sample task state. Of course, the initial state of the sample task can be preset at the beginning of the sample task. During the execution of the sample task, the initial state can be the final sample task state corresponding to the previous training round; that is, for each training round, the initial state of the sample task corresponding to that round can be the sample task state corresponding to the previous training round.

[0182] Specifically, the preset training objectives include increasing the likelihood probability of the initial second-policy model generating the third sample task processing flow and decreasing the likelihood probability of the initial second-policy model generating the fourth sample task processing flow. Since the third sample task processing flow is superior to the fourth sample task processing flow—that is, the third sample task processing flow is considered a positive sample flow that conforms to the expected behavior, and the fourth sample task processing flow is considered a negative sample flow that does not conform to the expected behavior—the initial second-policy model can be trained by minimizing the positive and negative sample task processing flows.

[0183] In practical applications, a contrastive loss function corresponding to the positive and negative sample task processing flow can be pre-constructed, making the trained objective second policy model more inclined to generate high-quality sample task execution paths.

[0184] As mentioned above, the above model training method is a one-round training method. Therefore, after one round of training, the target first policy model and the target second policy model can be used as the initial first policy model and the initial second policy model, respectively. The newly obtained initial first policy model and initial second policy model can be trained again according to the above model training method until the model training stopping condition is reached, and the final target first policy model and target second policy model can be obtained. The specific details are as described in step 102 above, and will not be repeated here.

[0185] It should be noted that when co-training the initial first policy model and the initial second policy model, the model training stopping conditions for the initial first policy model and the initial second policy model may differ. These stopping conditions include, but are not limited to, reaching a preset number of iterations, the calculated loss being less than a preset loss, and the number of training samples used reaching a preset number. Furthermore, the stopping time for the initial first policy model and the initial second policy model may differ. In this specification, the initial second policy model is used to assist in training the initial first policy model; that is, this specification aims to train the initial first policy model. Therefore, the entire model training process can be stopped when the initial first policy model reaches the first model training stopping condition. When the initial second policy model reaches the second model training stopping condition, training of the initial second policy model can be discontinued, and only training of the initial first policy model needs to continue. The specific method can be determined based on the actual situation.

[0186] Based on the above training method for the initial second policy model, the initial second policy model can be used to assist in training the initial first policy model. Furthermore, the initial first policy model and the initial second policy model can be trained collaboratively based on the same training data, which improves the compatibility between the initial first policy model and the initial second policy model. Since the initial first policy model is used to assist in training the initial second policy model, the accuracy of the trained target first policy model can be improved.

[0187] Furthermore, it eliminates the need for complex intermediate reward function designs, allowing training to be completed solely based on the processing results of each sample task, significantly reducing engineering implementation complexity. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the model structure and avoiding training instability issues caused by value function bias.

[0188] Furthermore, the first parameter tuning information is obtained based on the entire sample task processing flow, and the initial second policy model is used to obtain the second parameter tuning information based on the task processing steps. This combines the two types of parameter tuning information to train the initial first policy model. That is, it considers both the parameter tuning trends reflected in each sample task processing flow and the parameter tuning trends reflected in each task processing step within each sample task processing flow. This addresses the difficulty of credit allocation in multi-step inference tasks, thereby improving the rationality of credit allocation in multi-step inference tasks, increasing training efficiency, and further improving the accuracy of the trained target first policy model. The model training method provided in this specification, by introducing trajectory-level and step-level result supervision signals, can achieve effective training of the initial first policy model without the need for an explicit reward function.

[0189] Furthermore, in the first or more embodiments of this specification, when training the initial second policy model, the Exploration-Based Trajectory Optimization (ETO) algorithm is selected to train the initial second policy model, wherein ETO is an extension of the Direct Preference Optimization (DPO) algorithm, or the cross-entropy loss function is selected to train the initial second policy model, and this specification does not impose specific limitations.

[0190] In one or more embodiments of this specification, when the sample task processing result is a binary classification result, the ETO algorithm can be used to train the initial second policy model based on the processing flow of each sample task and the corresponding sample task processing result. Alternatively, the cross-entropy loss function can be selected to train the initial second policy model. When the sample task processing result is not a binary classification result, the ETO algorithm can be used to train the initial second policy model based on the processing flow of each sample task and the corresponding sample task processing result. In short, the ETO algorithm is applicable to all task types, including open-domain tasks, and the training method corresponding to the cross-entropy loss function is suitable for cases where the sample task processing result is a binary classification result.

[0191] In one or more embodiments of this specification, inputting the initial state of the sample task into the initial second policy model, and training the initial second policy model according to a preset training objective to obtain a target second policy model includes: inputting the initial state of the sample task into the initial second policy model to obtain a fourth probability that the output of the initial second policy model is the third sample task processing flow, and obtaining a fifth probability that the output of the initial second policy model is the fourth sample task processing flow. A second ratio of the fourth probability to the fifth probability is calculated, and the initial second policy model is trained according to the second ratio to obtain the target second policy model.

[0192] In practical applications, the third loss value can be determined based on the second ratio, and the initial second policy model can be trained based on the third loss value to obtain the target second policy model.

[0193] In one or more embodiments of this specification, the third loss value can be determined based on the second ratio using Formula 4. Formula 4 is shown below:

[0194] Formula 4

[0195] Where L1 represents the fourth loss value, Characterizes the initial state of the sample task. Characterizing the third sample task processing flow, Characterizing the fourth sample task processing flow, The fourth probability, representing the output of the second initial policy model, The fifth probability, representing the output of the second initial policy model, This is a hyperparameter.

[0196] It should be noted that the fourth probability in Formula 4 can be the probability that the output of the second initial strategy model is the third sample task processing flow, or the probability that the output of the second initial strategy model is any sample task processing step in the third sample task processing flow. The fifth probability can be the probability that the output of the second initial strategy model is the fourth sample task processing flow, or the probability that the output of the second initial strategy model is any sample task processing step in the fourth sample task processing flow.

[0197] By employing the ETO algorithm described above, training the initial second policy model can improve both the training efficiency and accuracy of the second initial policy model. When training the initial first policy model, the initial second policy model is used as an aid, and both models are trained collaboratively using the same training data, improving their compatibility. Since the initial first policy model is used to assist in training the initial second policy model, the accuracy of the trained target first policy model can be improved.

[0198] Furthermore, it eliminates the need for complex intermediate reward function designs, allowing training to be completed solely based on the processing results of each sample task, significantly reducing engineering implementation complexity. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the model structure and avoiding training instability issues caused by value function bias.

[0199] Furthermore, the first parameter tuning information is obtained based on the entire sample task processing flow, and the initial second policy model is used to obtain the second parameter tuning information based on the task processing steps. This combines the two types of parameter tuning information to train the initial first policy model. That is, it considers both the parameter tuning trends reflected in each sample task processing flow and the parameter tuning trends reflected in each task processing step within each sample task processing flow. This addresses the difficulty of credit allocation in multi-step inference tasks, thereby improving the rationality of credit allocation in multi-step inference tasks, increasing training efficiency, and further improving the accuracy of the trained target first policy model. The model training method provided in this specification, by introducing trajectory-level and step-level result supervision signals, can achieve effective training of the initial first policy model without the need for an explicit reward function.

[0200] Furthermore, when training the initial second policy model according to the second ratio to obtain the target second policy model, a benchmark policy model may also be introduced. In one or more embodiments of this specification, training the initial second policy model according to the second ratio to obtain the target second policy model includes: obtaining a benchmark policy model, inputting the initial state of the sample task into the benchmark policy model, obtaining that the output result of the benchmark policy model is the sixth probability of the third sample task processing flow, and obtaining that the output result of the benchmark policy model is the seventh probability of the fourth sample task processing flow. A third ratio of the sixth probability to the seventh probability is calculated, and a second difference between the third ratio and the second ratio is calculated. The initial second policy model is then trained based on the second difference to obtain the target second policy model.

[0201] It should be noted that, as mentioned earlier, the baseline policy model may not participate in the training process. The structure and parameters of this baseline policy model can be the same as those of the initial first policy model before training begins. This baseline policy model can also be a model pre-trained based on manually generated rules or sample datasets. Alternatively, the baseline policy model can participate in the training process, in which case a moving average algorithm, such as the exponential moving average algorithm, can be used to adjust the parameters of the baseline policy model based on the model parameters of the target first policy model for each training round. This specification does not impose specific restrictions on this; settings can be made based on actual needs.

[0202] In practical applications, the fourth loss value can be determined based on the second difference, and the initial second policy model can be trained based on the fourth loss value to obtain the target second policy model.

[0203] In one or more embodiments of this specification, the third loss value can be determined based on the second difference using Formula 5. Formula 5 is shown below:

[0204] Formula 5

[0205] Where L2 represents the fourth loss value, Characterizes the initial state of the sample task. Characterizing the third sample task processing flow, Characterizing the fourth sample task processing flow, The fourth probability, representing the output of the second initial policy model, The fifth probability, representing the output of the second initial policy model, The sixth probability, representing the output of the baseline policy model. The seventh probability, representing the output of the baseline policy model, This is a hyperparameter.

[0206] It should be noted that, corresponding to Formula 4 above, the sixth probability in Formula 5 can be the probability that the output result of the benchmark strategy model is the third sample task processing flow, or the probability that the output result of the benchmark strategy model is any task processing step in the third sample task processing flow. The seventh probability in Formula 5 can be the probability that the output result of the benchmark strategy model is the third sample task processing flow, or the probability that the output result of the benchmark strategy model is any task processing step in the third sample task processing flow.

[0207] Based on the above training method, a baseline policy model is introduced. This baseline policy model is used to provide a baseline comparison for the training of the initial second policy model, and to control the adjustment of the model parameters of the initial second policy model within a credible parameter range, so as to avoid catastrophic deviation and avoid forgetting historical knowledge, thereby improving training stability and improving the accuracy of the trained target second policy model.

[0208] By using probability ratios to encourage the initial second policy model to generate high-quality trajectories for the first sample task processing flow or similar third sample task processing flow, the goal is to increase the probability that the initial second policy model outputs a third sample task processing flow and decrease the probability that the initial second policy model outputs a fourth sample task processing flow. This training of the initial second policy model enables it to learn high-quality policies, thereby providing more accurate step reward information to the initial first policy model in subsequent steps and improving the accuracy of the trained target first policy model.

[0209] In one or more embodiments of this specification, when the sample task processing result is a binary classification result, a cross-entropy loss function can also be selected to train an initial second policy model, so as to improve the accuracy of the target second policy model that can be obtained through training efficiency.

[0210] In one or more embodiments of this specification, obtaining the initial state of the sample task, inputting the initial state of the sample task into the initial second policy model, and training the initial second policy model according to a preset training objective includes:

[0211] For each sample task processing flow, if the sample task processing flow is the third sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the eighth probability of the sample task processing flow. Based on the eighth probability, the first loss value corresponding to the sample task processing flow is obtained.

[0212] When the sample task processing flow is the fourth sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the ninth probability of the sample task processing flow. Based on the ninth probability, the second loss value corresponding to the sample task processing flow is obtained.

[0213] Based on the first loss value and the second loss value, the initial second policy model is trained to obtain the target second policy model.

[0214] In one or more embodiments of this specification, when the sample task processing flow is a third sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the eighth probability of the sample task processing flow, including:

[0215] When the sample task processing flow is the third sample task processing flow, the third sample task processing step is determined, wherein the third sample task processing step is any sample task processing step in the sample task processing flow.

[0216] The sample task state corresponding to the third sample task processing step and the sample task processing steps preceding the third sample task processing step are input into the initial second strategy model, and the result output by the initial second strategy model is the tenth probability of the sample task action corresponding to the third sample task processing step.

[0217] Based on the tenth probability corresponding to each sample task processing step in the sample task processing flow, the eighth probability corresponding to the sample task processing flow is obtained.

[0218] Accordingly, when the sample task processing flow is the fourth sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the ninth probability of the sample task processing flow, including:

[0219] When the sample task processing flow is the fourth sample task processing flow, the fourth sample task processing step is determined, wherein the fourth sample task processing step is any sample task processing step in the sample task processing flow.

[0220] The sample task state corresponding to the fourth sample task processing step and the sample task processing steps preceding the fourth sample task processing step are input into the initial second strategy model. The result output by the initial second strategy model is the eleventh probability of the sample task action corresponding to the fourth sample task processing step.

[0221] Based on the eleventh probability corresponding to each sample task processing step in the sample task processing flow, the ninth probability corresponding to the sample task processing flow is obtained.

[0222] In one or more embodiments of this specification, Formula Six may be used to calculate the first loss value. Formula Six is ​​shown below:

[0223] Formula Six

[0224] Where L3 represents the first loss value, Characterizes the processing flow of the i-th third sample task. This represents the processing step of the k-th sample task in the processing flow of the i-th third sample task. This represents the tenth probability corresponding to the k-th sample task processing step in the i-th third sample task processing flow. Wherein, , Characterizing the output of the second initial policy model The tenth probability.

[0225] Furthermore, in practical applications, a benchmark policy model can be introduced. Therefore, the sample task state corresponding to the third sample task processing step and the sample task processing steps preceding the third sample task processing step can be input into the benchmark policy model. The output of the benchmark policy model is the twelfth probability of the sample task action corresponding to the third sample task processing step. The third loss value is then calculated based on the twelfth and tenth probabilities. Thus, in Formula Six above… ,in, Characterizing the output of the baseline strategy model The twelfth probability, Characterizing the output of the second initial policy model The tenth probability.

[0226] In one or more embodiments of this specification, Formula 7 can be used to calculate the second loss value. Formula 7 is shown below:

[0227] Formula 7

[0228] Where L4 represents the second loss value, Characterizes the processing flow of the i-th fourth sample task. This represents the processing step of the k-th sample task in the processing flow of the i-th fourth sample task. This represents the eleventh probability corresponding to the processing step of the k-th sample task in the processing flow of the i-th fourth sample task. Wherein, , Characterizing the output of the second initial policy model The eleventh probability.

[0229] Furthermore, in practical applications, a benchmark policy model can be introduced. Therefore, the sample task state corresponding to the fourth sample task processing step and the sample task processing steps preceding the fourth sample task processing step can be input into the benchmark policy model to obtain the thirteenth probability of the sample task action corresponding to the fourth sample task processing step as the output of the benchmark policy model. Then, in Formula 7 above... ,in, Characterizing the output of the baseline strategy model The thirteenth probability, Characterizing the output of the second initial policy model The thirteenth probability.

[0230] Based on the training method of the initial second policy model described above, the initial second policy model can be trained using the same sample data during the training of the initial first policy model. This can improve the compatibility between the initial first policy model and the initial second policy model. Furthermore, since the initial second policy model is used to provide step reward information for the training of the initial first policy model, the accuracy of the trained target first policy model can be improved.

[0231] Based on the above training method, training the initial second policy model using the cross-entropy loss function can improve training stability and the performance of the trained target second policy model.

[0232] In addition, a baseline policy model is introduced to provide a baseline comparison for the training of the initial second policy model. This ensures that the model parameters of the initial second policy model are adjusted and controlled within a credible range, avoiding catastrophic deviations and preventing the forgetting of historical knowledge, thereby improving training stability and the accuracy of the trained target second policy model. See also Figure 5 , Figure 5 This is a schematic diagram of data flow for training an initial second strategy model, provided in one embodiment of this specification.

[0233] This specification also provides an embodiment of a model training method applied to cloud-based devices. Figure 6 A schematic flowchart illustrating a model training method provided in one embodiment of this specification is shown. Figure 6 As shown, this model training method, applied to cloud-side devices, includes:

[0234] Step 602: The receiving end device sends multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0235] Step 604: Based on the processing results of each sample task, determine the first parameter tuning information corresponding to the processing flow of each sample task.

[0236] Step 606: Based on the target second strategy model, obtain the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow.

[0237] Step 608: Train the initial first policy model based on the first parameter tuning information and the second parameter tuning information to obtain the target first policy model.

[0238] Step 610: Obtain the model parameters of the target first strategy model and send the model parameters to the end-side device.

[0239] It should be noted that in this embodiment, the process by which the cloud-side device trains the target first policy model based on the multiple sample task processing flows sent by the edge device and the sample task processing results corresponding to each sample task processing flow is consistent with the model training method described above, and will not be elaborated further here. After training and obtaining the target first policy model, the model parameters of the target first policy model can be obtained, and the model parameters can be sent to the edge device.

[0240] Based on the model training method applied to cloud-side devices, the cloud-side devices can obtain the first parameter tuning information of each sample task processing flow sent by the end-side devices, and the second parameter tuning information corresponding to each sample task processing step obtained through the second policy model related to the sample task. Thus, the initial first policy model can be trained by combining the two parameter tuning information, taking into account both the parameter tuning trend reflected by each sample task processing flow and the parameter tuning trend reflected by each task processing step in each sample task processing flow. This can solve the problem of credit allocation difficulty in multi-round interactive tasks and multi-step inference tasks, thereby improving the rationality of credit allocation in multi-round interactive tasks and multi-step inference tasks, improving training efficiency, and improving the accuracy of the trained target first policy model.

[0241] Furthermore, it eliminates the need for complex intermediate reward function design, allowing training solely based on the sample task processing results and the target second policy model for each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the structure of the first policy model and avoiding training instability issues caused by value function bias.

[0242] As mentioned earlier, a text interaction task refers to the task of generating response text corresponding to the user's input text based on the received input text. This text interaction task can specifically be a conversational human-computer interaction task such as an intelligent customer service system or a multi-turn question-and-answer system. The sample task state can be the input text, historical dialogue context, or prompt text, etc., and the sample task action can be the response text content output by the initial first strategy model. This response text content can be the next token, a sentence, a reply, etc. Furthermore, the text content output by the initial first strategy model is added to the historical dialogue context to obtain a new historical dialogue context, or the user's input text for a new round of response based on the text content output by the initial first strategy model is added to the historical dialogue context to obtain a new historical dialogue context, thus obtaining the next sample task state. This continues until conditions such as the user stopping input, the sample task objective being achieved, or the maximum number of interaction rounds being reached are met, ending the interaction and obtaining a sample task processing flow. The sample task processing result corresponding to the sample task processing flow can be obtained manually or through an artificial intelligence model such as a large language model. It should be noted that when the sample task is applied in a human-computer interaction scenario, the initial first strategy model can be an autoregressive language model, such as a large language model based on the Transformer architecture. This initial first strategy model can generate a response token by token according to the current context and form a closed-loop interaction with the user environment.

[0243] The following is in conjunction with the appendix Figure 7 Taking the application of the model training method provided in this specification in a text interaction task as an example, the model training method will be further explained. Figure 7 The diagram illustrates a model training method according to an embodiment of this specification, applied to a text interaction task, and specifically includes the following steps.

[0244] Step 702: Obtain multiple sample text interaction processes and the sample text interaction results corresponding to each sample text interaction process. The sample text interaction process includes at least one round of sample text interaction. Each round of sample text interaction includes sample text interaction actions and sample text interaction states. The sample text interaction state includes the first input text input to the initial first strategy model. The sample text interaction action includes the first response text output by the initial first strategy model. The sample text interaction action is determined by the initial first strategy model based on the sample text interaction state.

[0245] Step 704: Based on the text interaction results of each sample, determine the first parameter tuning information corresponding to the text interaction process of each sample.

[0246] Step 706: Based on the target second strategy model, obtain the second parameter tuning information corresponding to each round of sample text interaction in the sample text interaction processing flow.

[0247] Step 708: Based on the first parameter tuning information and the second parameter tuning information, train the initial first policy model to obtain the target first policy model.

[0248] In practical applications, a sample text interaction process can be a single sample dialogue session, which can be a complete sample text interaction task process consisting of multiple rounds of sample text interactions. For example, a single sample dialogue session might involve a user asking a question, an initial first policy model responding, the user asking follow-up questions, and the initial first policy model responding again. Thus, the sample interaction process includes at least one round of sample text interaction, each round being a dialogue within a sample dialogue session. Each dialogue includes a first sample input text and a first sample response text. Based on reinforcement learning, each round of sample text interaction includes a sample text interaction action and a sample text interaction state. The sample text interaction state includes at least the input text, and the sample text interaction action includes at least the response text reply. Furthermore, for each round of sample text interaction, the corresponding sample text interaction action is determined by the initial first policy model based on the corresponding sample text interaction state.

[0249] It should be noted that the sample text interaction result can be success or failure as described above, or it can be an evaluation value obtained based on other evaluation models or manual methods. Further details will not be elaborated here.

[0250] Furthermore, it should be noted that the specific process of training the initial first policy model and obtaining the target first policy model based on the text interaction results of each sample is consistent with the model training method described above, and will not be repeated in this embodiment. This embodiment only uses a text interaction task as an example; that is, the sample task processing flow is the sample text interaction flow, the sample task processing result is the sample text interaction result, each sample task processing step is each round of sample text interaction, the sample task action is the sample text interaction action, and the sample task state is the sample text interaction state.

[0251] Furthermore, in one or more embodiments of this specification, when the sample task is a text interaction task specifically a text generation task, the generation of each response text can be token-by-token. Therefore, when training the initial first policy model, as described in step 108, when calculating the global loss value based on the calculated local loss value corresponding to each task processing step in the processing flow of each sample task, the global loss value when the sample task is a text generation task can be calculated using Formula 8. Formula 8 is shown as follows:

[0252]

[0253] Where G represents the total number of sample task processing steps. This represents the total number of task processing steps in the processing flow of the i-th sample task. This refers to the k-th task processing step in the processing flow of the i-th sample task. This refers to the task processing steps preceding the k-th task processing step in the i-th sample task processing flow. Let t be the t-th token in task processing step j within the sample task processing flow i. This refers to the token that precedes the t-th token in the k-th task processing step within the i-th sample task processing flow. The output of the initial first policy model in the current training round is The probability, The result output by the initial first policy model in the previous training round for the current round is The probability of. and All parameters are preset. The difference in probability distribution between the initial first policy model and the baseline policy model is called the KL divergence. Let be the first advantage value corresponding to the k-th task processing step in the i-th sample task processing flow. This is the second advantage value corresponding to the sample task processing flow where the k-th task processing step is located, which is the second advantage value corresponding to the i-th sample task processing flow. The function is used to calculate the target advantage value based on the first advantage value and the second advantage value, which is the target advantage value corresponding to the k-th task processing step in the i-th sample task processing flow. The function for calculating the target advantage value is the same as the function for calculating the target advantage value in Formula 3 above, and will not be repeated here.

[0254] The model training method provided in this embodiment for text interaction tasks can improve the accuracy of the first strategy model generated by the final target model, enhance the user experience, and improve the training efficiency and stability of the model.

[0255] During the training of the initial first-policy model, there is no need to rely on complex intermediate reward function design. Training can be completed solely based on the sample task processing results corresponding to each sample task processing flow, significantly reducing the engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, this method does not require a value network for advantage estimation, thereby simplifying the model structure and avoiding training instability issues caused by value function bias.

[0256] Furthermore, when training the initial first policy model, the initial second policy model is used as an aid, and the initial first policy model and the initial second policy model are trained collaboratively based on the same training data, which improves the compatibility between the initial first policy model and the initial second policy model. Since the initial first policy model is used to assist in training the initial second policy model, the accuracy of the trained target first policy model can be improved.

[0257] In addition, the first parameter tuning information is obtained based on the entire sample task processing flow, and the initial second strategy model is used to obtain the second parameter tuning information based on the task processing steps. Thus, the initial first strategy model is trained by combining the two types of parameter tuning information. That is, it considers both the parameter tuning trend reflected in the processing flow of each sample task and the parameter tuning trend reflected in each task processing step in the processing flow of each sample task. This solves the problem of credit allocation difficulty in multi-step reasoning tasks, which improves the rationality of credit allocation in multi-step reasoning tasks, improves training efficiency, and further improves the accuracy of the trained target first strategy model.

[0258] See Figure 8 , Figure 8 A flowchart illustrating a text generation method according to an embodiment of this specification is shown, specifically including the following steps:

[0259] Step 802: Obtain the second input text.

[0260] Step 804: Input the second input text into the target first policy model to obtain the second response text corresponding to the second input text, wherein the target first policy model is trained by the above-described model training method.

[0261] In practical applications, after training the target first policy model based on the above model training method, the second input text can be obtained. Then, the second input text is input into the target first policy model to obtain the corresponding text output by the first policy model.

[0262] Based on the above text generation method, since the target first policy model is obtained based on the above model training method, as mentioned above, the above model training method can effectively improve the accuracy of the trained target first policy model. Therefore, this text generation method can improve the accuracy of the generated target text and improve the user experience.

[0263] See Figure 9 , Figure 9 The diagram illustrates a text generation method according to an embodiment of this specification, applied to a cloud-side device, and specifically includes the following steps:

[0264] Step 902: The second input text sent by the receiving end device.

[0265] Step 904: Input the second input text into the target first policy model to obtain the second response text corresponding to the second input text, wherein the target first policy model is trained by the above-described model training method.

[0266] Step 906: Send the second response text to the end device.

[0267] In practical applications, the target-first strategy model can be deployed on cloud-side devices.

[0268] Based on the above text generation method, since the target first policy model is obtained based on the above model training method, as mentioned above, the above model training method can effectively improve the accuracy of the trained target first policy model. Therefore, this text generation method can improve the accuracy of the generated target text and improve the user experience.

[0269] Corresponding to the above method embodiments, this specification also provides an embodiment of a model training device. Figure 10 A schematic diagram of a model training apparatus provided in one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0270] The first acquisition module 1002 is configured to acquire multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step, and each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0271] The first determining module 1004 is configured to determine the first parameter tuning information corresponding to the processing flow of each sample task based on the processing results of each sample task.

[0272] The second acquisition module 1006 is configured to acquire the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow according to the target second strategy model.

[0273] The first training module 1008 is configured to train the initial first policy model based on each first parameter tuning information and each second parameter tuning information to obtain the target first policy model.

[0274] Optionally, the first determining module 1004 is further configured to: determine a first sample task processing flow and a second sample task processing flow, wherein the first sample task processing flow is any one of the plurality of sample task processing flows, and the second sample task processing flow is a sample task processing flow other than the first sample task processing flow among the plurality of sample task processing flows; determine first benchmark information corresponding to the first sample task processing flow based on the sample task processing result corresponding to the second sample task processing flow; and determine first parameter tuning information corresponding to the first sample task processing flow based on the first benchmark information and the sample task processing result corresponding to the first sample task processing flow.

[0275] Optionally, the sample task processing result includes first reward information corresponding to the sample task processing flow, and the first benchmark information includes first average reward information of the first reward information corresponding to the second sample task processing flow.

[0276] The first determining module 1004 is further configured to: determine the difference in first reward information corresponding to the first sample task processing flow based on the first average reward information and the first reward information corresponding to the first sample task processing flow; and determine the first parameter tuning information corresponding to the first sample task processing flow based on the difference in first reward information.

[0277] Optionally, the second acquisition module 1006 is further configured to: for each sample task processing step in any sample task processing flow, obtain the second reward information corresponding to the sample task processing step through the target second strategy model; and obtain the second parameter tuning information corresponding to the sample task processing step based on the second reward information corresponding to each sample task processing step.

[0278] Optionally, the second acquisition module 1006 is further configured to: acquire a first sample task processing step preceding the sample task processing step; input the sample task state corresponding to the sample task processing step and the first sample task processing step into the target second strategy model to obtain second reward information corresponding to the sample task processing step, wherein the second reward information is determined according to a first probability, and the first probability is the probability that the result output by the target second strategy model is the sample task action corresponding to the sample task processing step.

[0279] Optionally, the second acquisition module 1006 is further configured to: determine the second benchmark information corresponding to the sample task processing step based on the second reward information corresponding to the sample task processing step other than the sample task processing step; and obtain the second parameter tuning information corresponding to the sample task processing step based on the second benchmark information and the second reward information.

[0280] Optionally, the second benchmark information includes the second average reward information corresponding to the second reward information of sample task processing steps other than the sample task processing step.

[0281] The second acquisition module 1006 is further configured to: determine the difference in second reward information based on the second average reward information and the second reward information corresponding to the sample task processing step; and obtain the second parameter tuning information corresponding to the sample task processing step based on the difference in second reward information.

[0282] Optionally, the first training module 1008 is further configured to: for each sample task processing step, obtain the second parameter tuning information corresponding to the sample task processing step, and obtain the first parameter tuning information corresponding to the sample task processing flow in which the sample task processing step is located; obtain the target parameter tuning information corresponding to the sample task processing step based on the second parameter tuning information and the first parameter tuning information; and train the initial first policy model based on the target parameter tuning information corresponding to each sample task processing step to obtain the target first policy model.

[0283] Optionally, the first training module 1008 is further configured to: for each sample task processing step, input the sample task state corresponding to the sample task processing step and the second sample task processing step into the initial first policy model to obtain the second probability output by the initial first policy model, wherein the second sample task processing step is the sample task processing step located before the current sample task processing step in the sample task processing flow, and the second probability is the probability of the sample task action corresponding to the current sample task processing step as output by the initial first policy model; train the initial first policy model according to the second probability corresponding to each sample task processing step and the target parameter tuning information to obtain the target first policy model.

[0284] Optionally, the first training module 1008 is further configured to: for each sample task processing step, calculate the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step and the target parameter tuning information; obtain the global loss value based on the local loss value corresponding to each sample task processing step; and train the initial first policy model based on the global loss value to obtain the target first policy model.

[0285] Optionally, the first training module 1008 is further configured to: obtain a first probability distribution output by the initial first policy model, wherein the first probability distribution is the sampling probability of each sample task action of the initial first policy model under the sample task state corresponding to the sample task processing step; obtain a benchmark policy model, and input the sample task state corresponding to the sample task processing step and the second sample task processing step into the benchmark policy model to obtain a second probability distribution, wherein the second probability distribution is the sampling probability of each sample task action of the benchmark policy model under the sample task state corresponding to the sample task processing step; calculate the difference between the first probability distribution and the second probability distribution to obtain the distribution difference value corresponding to the sample task processing step; and calculate the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step, the target parameter tuning information, and the distribution difference value.

[0286] Optionally, the target parameter tuning information includes the target parameter tuning value;

[0287] The first training module 1008 is further configured to: calculate the product of the second probability corresponding to the sample task processing step and the target parameter tuning value; calculate the first difference between the product and the distribution difference value corresponding to the sample task processing step, and obtain the local loss value corresponding to the sample task processing step.

[0288] Optionally, the first training module 1008 is further configured to: obtain the third probability corresponding to the sample task processing step, wherein the third probability is the historical second probability in the historical training rounds corresponding to the sample task processing step; and calculate the local loss value corresponding to the sample task processing step based on the second probability, the third probability, and the target parameter tuning information.

[0289] Optionally, the target parameter tuning information includes the target parameter tuning value;

[0290] The first training module 1008 is further configured to: calculate the first ratio of the second probability to the third probability corresponding to the sample task processing step; calculate the product of the first ratio and the target parameter tuning value to obtain the local loss value corresponding to the sample task processing step.

[0291] Optionally, the device further includes a second strategy model training module, configured to: divide the sample task processing flow according to the processing results of each sample task to obtain a third sample task processing flow and a fourth sample task processing flow; obtain the initial state of the sample task, input the initial state of the sample task into an initial second strategy model, and train the initial second strategy model according to a preset training objective to obtain a target second strategy model, wherein the preset training objective includes increasing the probability that the output result of the initial second strategy model is the third sample task processing flow and decreasing the probability that the output result of the initial second strategy model is the fourth sample task processing flow.

[0292] Optionally, the second strategy model training module is further configured to: input the initial state of the sample task into an initial second strategy model to obtain a fourth probability that the output of the initial second strategy model is the third sample task processing flow, and to obtain a fifth probability that the output of the initial second strategy model is the fourth sample task processing flow; calculate a second ratio between the fourth probability and the fifth probability; and train the initial second strategy model according to the second ratio to obtain a target second strategy model.

[0293] Optionally, the second strategy model training module is further configured to: acquire a benchmark strategy model, input the initial state of the sample task into the benchmark strategy model, obtain the output result of the benchmark strategy model as the sixth probability of the third sample task processing flow, and obtain the output result of the benchmark strategy model as the seventh probability of the fourth sample task processing flow; calculate the third ratio of the sixth probability to the seventh probability, and calculate the second difference between the third ratio and the second ratio; train the initial second strategy model based on the second difference to obtain the target second strategy model.

[0294] Optionally, the second policy model training module is further configured to: for each sample task processing flow, if the sample task processing flow is the third sample task processing flow, input the initial state of the sample task into the initial second policy model to obtain the output result of the initial second policy model as the eighth probability of the sample task processing flow, and obtain the first loss value corresponding to the sample task processing flow based on the eighth probability; if the sample task processing flow is the fourth sample task processing flow, input the initial state of the sample task into the initial second policy model to obtain the output result of the initial second policy model as the ninth probability of the sample task processing flow, and obtain the second loss value corresponding to the sample task processing flow based on the ninth probability; train the initial second policy model based on the first loss value and the second loss value to obtain the target second policy model.

[0295] Optionally, the second strategy model training module is further configured to: determine a third sample task processing step when the sample task processing flow is a third sample task processing flow, wherein the third sample task processing step is any sample task processing step in the sample task processing flow; input the sample task state corresponding to the third sample task processing step and the sample task processing steps preceding the third sample task processing step into the initial second strategy model to obtain the output result of the initial second strategy model as the tenth probability of the sample task action corresponding to the third sample task processing step; and obtain the eighth probability corresponding to the sample task processing flow based on the tenth probability corresponding to each sample task processing step in the sample task processing flow.

[0296] When the sample task processing flow is the fourth sample task processing flow, a fourth sample task processing step is determined, wherein the fourth sample task processing step is any sample task processing step in the sample task processing flow; the sample task state corresponding to the fourth sample task processing step and the sample task processing steps preceding the fourth sample task processing step are input into the initial second strategy model, and the output of the initial second strategy model is the eleventh probability of the sample task action corresponding to the fourth sample task processing step; based on the eleventh probability corresponding to each sample task processing step in the sample task processing flow, the ninth probability corresponding to the sample task processing flow is obtained.

[0297] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0298] Based on the model training device provided in this embodiment, the first parameter tuning information is obtained according to the entire sample task processing flow. The initial second strategy model is used to obtain the second parameter tuning information based on the task processing steps. Thus, the initial first strategy model is trained by combining the two parameter tuning information. That is, it considers both the parameter tuning trend reflected in each sample task processing flow and the parameter tuning trend reflected in each task processing step in each sample task processing flow. This solves the problem of credit allocation difficulty in multi-step reasoning tasks, thereby improving the rationality of credit allocation in multi-step reasoning tasks, improving training efficiency, and further improving the accuracy of the trained target first strategy model.

[0299] Furthermore, it eliminates the need for complex intermediate reward function design, allowing training solely based on the sample task processing results and the target second policy model for each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the structure of the first policy model and avoiding training instability issues caused by value function bias.

[0300] Corresponding to the above-described model training method embodiments, this specification also provides an embodiment of a model training device located on a cloud-side device. Figure 11 A schematic diagram of a model training apparatus provided in one embodiment of this specification is shown. This apparatus is located on a cloud-side device. Figure 11 As shown, the device includes:

[0301] The first receiving module 1102 is configured to receive multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow sent by the end-side device. The sample task processing flow includes at least one sample task processing step, and each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0302] The second determining module 1104 is configured to determine the first parameter tuning information corresponding to the processing flow of each sample task based on the processing results of each sample task.

[0303] The third acquisition module 1106 is configured to acquire the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow according to the target second strategy model.

[0304] The second training module 1108 is configured to train the initial first policy model based on each first parameter tuning information and each second parameter tuning information to obtain the target first policy model.

[0305] The first sending module 1110 is configured to obtain the model parameters of the target first strategy model and send the model parameters to the end-side device.

[0306] The above is an illustrative scheme of a service request processing device according to this embodiment. It should be noted that the technical solution of this service request processing device and the technical solution of the service request processing method described above belong to the same concept. For details not described in detail in the technical solution of the service request processing device, please refer to the description of the technical solution of the service request processing method described above.

[0307] Based on the model training device provided in this embodiment, the first parameter tuning information is obtained according to the entire sample task processing flow. The initial second strategy model is used to obtain the second parameter tuning information based on the task processing steps. Thus, the initial first strategy model is trained by combining the two parameter tuning information. That is, it considers both the parameter tuning trend reflected in each sample task processing flow and the parameter tuning trend reflected in each task processing step in each sample task processing flow. This solves the problem of credit allocation difficulty in multi-step reasoning tasks, thereby improving the rationality of credit allocation in multi-step reasoning tasks, improving training efficiency, and further improving the accuracy of the trained target first strategy model.

[0308] This approach can improve the accuracy of text generated by the final target first-policy model obtained through training, enhancing user experience and improving training efficiency and stability. Specifically, during the initial training of the first-policy model, training is assisted by the target second-policy model. This eliminates the need for complex intermediate reward function design, allowing training to be completed solely based on the sample task processing results corresponding to each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, this approach simplifies the model structure and avoids training instability caused by value function bias.

[0309] Taking the application of the model training method provided in this specification in a text interaction task as an example, corresponding to the above method embodiments, this specification also provides an embodiment of a model training device applied to text interaction tasks. Figure 12 A schematic diagram of a model training apparatus provided in one embodiment of this specification is shown. Figure 12 As shown, the device is used for text interaction tasks, and the device includes:

[0310] The fourth acquisition module 1202 is configured to acquire multiple sample text interaction processes and sample text interaction results corresponding to each sample text interaction process. The sample text interaction process includes at least one round of sample text interaction. Each round of sample text interaction includes a sample text interaction action and a sample text interaction state. The sample text interaction state includes a first input text input to the initial first strategy model. The sample text interaction action includes a first response text output by the initial first strategy model. The sample text interaction action is determined by the initial first strategy model based on the sample text interaction state.

[0311] The third determining module 1204 is configured to determine the first parameter tuning information corresponding to the text interaction process of each sample based on the text interaction results of each sample.

[0312] The fifth acquisition module 1206 is configured to acquire the second parameter tuning information corresponding to each round of sample text interaction in the sample text interaction processing flow according to the target second strategy model;

[0313] The third training module 1208 is configured to train the initial first policy model based on the first and second parameter tuning information to obtain the target first policy model.

[0314] The above is an illustrative scheme of a model training device applied to a text interaction task according to this embodiment. It should be noted that the technical solution of this model training device belongs to the same concept as the technical solution of the model training method in the text generation task described above. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method applied to the text interaction task described above.

[0315] The model training device for text interaction tasks provided in this embodiment obtains first parameter tuning information based on the entire sample task processing flow. The initial second strategy model is used to obtain second parameter tuning information based on the task processing steps. Thus, the initial first strategy model is trained by combining the two types of parameter tuning information. That is, it considers both the parameter tuning trend reflected in each sample task processing flow and the parameter tuning trend reflected in each task processing step in each sample task processing flow. This solves the problem of credit allocation difficulty in multi-step reasoning tasks, thereby improving the rationality of credit allocation in multi-step reasoning tasks, improving training efficiency, and further improving the accuracy of the trained target first strategy model.

[0316] This approach can improve the accuracy of text generated by the final target first-policy model obtained through training, enhancing user experience and improving training efficiency and stability. Specifically, during the initial training of the first-policy model, training is assisted by the target second-policy model. This eliminates the need for complex intermediate reward function design, allowing training to be completed solely based on the sample task processing results corresponding to each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, this approach simplifies the model structure and avoids training instability caused by value function bias.

[0317] Corresponding to the above method embodiments, this specification also provides an embodiment of a text generation apparatus. Figure 13 A schematic diagram of a text generation apparatus according to one embodiment of this specification is shown. Figure 13 As shown, the device includes:

[0318] The sixth acquisition module 1302 is configured to acquire the second input text;

[0319] The first output module 1304 is configured to input the second input text into the target first policy model to obtain the second response text corresponding to the second input text, wherein the target first policy model is trained by the above-described model training method.

[0320] Based on the above text generation method, since the target first policy model is obtained based on the above model training method, as mentioned above, the above model training method can effectively improve the accuracy of the trained target first policy model. Therefore, this text generation method can improve the accuracy of the generated target text and improve the user experience.

[0321] Corresponding to the above method embodiments, this specification also provides an embodiment of a text generation device located on a cloud-side device. Figure 14 A schematic diagram of a text generation apparatus according to one embodiment of this specification is shown. Figure 14 As shown, the device is located on the cloud side and includes:

[0322] The second receiving module 1402 is configured to receive second input text sent by the receiving end-side device;

[0323] The second output module 1404 inputs the second input text into the target first strategy model to obtain the second response text corresponding to the second input text, wherein the target first strategy model is trained by the above-described model training method;

[0324] The second sending module 1406 is configured to send the second response text to the end-side device.

[0325] Based on the above text generation method, since the target first policy model is obtained based on the above model training method, as mentioned above, the above model training method can effectively improve the accuracy of the trained target first policy model. Therefore, this text generation method can improve the accuracy of the generated target text and improve the user experience.

[0326] See Figure 15 , Figure 15 This specification illustrates an architecture diagram of a model training system provided in one embodiment of the specification. The model training system may include a client 100 and a server 200.

[0327] Client 100 is used to send a model training request to server 200. The model training request carries multiple sample task processing procedures and sample task processing results corresponding to each sample task processing procedure. The sample task processing procedure includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state.

[0328] Server 200 is used to receive model training requests, determine the first parameter tuning information corresponding to the processing flow of each sample task based on the processing results of each sample task; obtain the second parameter tuning information corresponding to each sample task processing step in the processing flow of each sample task based on the target second strategy model; train the initial first strategy model based on each first parameter tuning information and each second parameter tuning information to obtain the target first strategy model; and send the model parameters of the final target second strategy model and the model parameters of the final target first strategy model corresponding to the completion of training to client 100.

[0329] Client 100 is also used to receive model parameters of the target first strategy model sent by server 200.

[0330] Using the scheme of the embodiments in this specification, the client can send a model training request to the server, carrying multiple sample task processing flows and the sample task processing results corresponding to each sample task processing flow. The server can receive the model training request, determine the first parameter tuning information corresponding to each sample task processing flow based on the processing results of each sample task; obtain the second parameter tuning information corresponding to each sample task processing step in each sample task processing flow based on the target second strategy model; train the initial first strategy model based on each first parameter tuning information and each second parameter tuning information to obtain the target first strategy model. This process is repeated iteratively until the model training stops, at which point the server can send the model parameters of the target first strategy model corresponding to the training completion to the client. The client can then receive the model parameters of the target first strategy model sent by the server.

[0331] Using the above method, the first parameter tuning information is obtained based on the entire sample task processing flow. The initial second strategy model is used to obtain the second parameter tuning information based on the task processing steps. Thus, the initial first strategy model is trained by combining the two types of parameter tuning information. This means that both the parameter tuning trend reflected in each sample task processing flow and the parameter tuning trend reflected in each task processing step in each sample task processing flow are considered. This solves the problem of credit allocation difficulty in multi-step reasoning tasks, thereby improving the rationality of credit allocation in multi-step reasoning tasks, increasing training efficiency, and further improving the accuracy of the trained target first strategy model.

[0332] Furthermore, it eliminates the need for complex intermediate reward function design, allowing training solely based on the sample task processing results and the target second policy model for each sample task, significantly reducing engineering implementation difficulty. Unlike current methods that rely on value networks for advantage estimation, it eliminates the need for such estimation, simplifying the structure of the first policy model and avoiding training instability issues caused by value function bias.

[0333] The model training system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the model training system scenario, server 200 is used to provide model training services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0334] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In a model training system scenario, users can publish data streams to server 200 through client 100, and server 200 can generate model parameters of the target first policy model based on the data stream, and push the model parameters of the target first policy model to other clients that have established communication.

[0335] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0336] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices may have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0337] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0338] It is worth noting that the model training methods, model training methods for text interaction tasks, and text generation methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the model training methods, model training methods for text interaction tasks, and text generation methods provided in the embodiments of this specification. In other embodiments, the model training methods, model training methods for text interaction tasks, and text generation methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0339] Figure 16 A structural block diagram of an electronic device 1600 according to an embodiment of this application is shown. The components of the electronic device 1600 include, but are not limited to, a memory 1610 and a processor 1620. The processor 1620 is connected to the memory 1610 via a bus 1630, and a database 1650 is used to store data.

[0340] Electronic device 1600 also includes access device 1640, which enables electronic device 1600 to communicate via one or more networks 1660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1640 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0341] In one embodiment of this application, the above-mentioned components of the electronic device 1600 and Figure 16 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 16 The illustrated electronic device block diagram is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0342] Electronic device 1600 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 1600 can also be a mobile or stationary server.

[0343] The processor 1620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method, model training method applied to text interaction task, and text generation method.

[0344] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the model training method, the model training method applied to text interaction tasks, and the text generation method described above. For details not described in detail in the technical solution of the electronic device, please refer to the descriptions of the technical solutions of the model training method, the model training method applied to text interaction tasks, and the text generation method described above.

[0345] Figure 17 A structural block diagram of a computing device 1700 provided according to one embodiment of this specification is shown.

[0346] The computing device 1700 includes:

[0347] Memory 1710 and processor 1720;

[0348] The memory 1710 is used to store computer programs / instructions, and the processor 1720 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1720, they implement the steps of the model training method, the model training method applied to the text interaction task, and the text generation method.

[0349] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0350] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0351] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0352] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the above-described model training method, model training method applied to text interaction task, and text generation method. For details not described in detail in the technical solution of the computing device, please refer to the descriptions of the technical solutions of the above-described model training method, model training method applied to text interaction task, and text generation method.

[0353] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method, the model training method applied to a text interaction task, and the text generation method.

[0354] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the model training method, the model training method applied to text interaction tasks, and the text generation method embodiments. Relevant details can be found in the descriptions of the model training method, the model training method applied to text interaction tasks, and the text generation method embodiments.

[0355] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method, the model training method applied to a text interaction task, and the text generation method.

[0356] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned model training method, model training method applied to text interaction task, and text generation method. For details not described in detail in the technical solution of the computer program product, please refer to the descriptions of the technical solutions of the above-mentioned model training method, model training method applied to text interaction task, and text generation method.

[0357] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0358] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0359] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0360] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0361] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, comprising: Obtain multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state. Based on the processing results of each sample task, the first parameter tuning information corresponding to the processing flow of each sample task is determined, wherein the first parameter tuning information reflects the overall quality of the sample task processing flow. Based on the target second strategy model, the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow is obtained, wherein the second parameter tuning information reflects the local quality of the sample task processing step. Based on the first and second parameter tuning information, the initial first policy model is trained to obtain the target first policy model. The target second strategy model is obtained by training using the following method: Based on the processing results of each sample task, the sample task processing flow is divided to obtain the third sample task processing flow and the fourth sample task processing flow. The sample task processing result corresponding to the third sample task processing flow is better than the sample task processing result corresponding to the fourth sample task processing flow. Obtain the initial state of the sample task, input the initial state of the sample task into the initial second strategy model, and train the initial second strategy model according to the preset training objective to obtain the target second strategy model. The preset training objective includes increasing the probability that the output result of the initial second strategy model is the third sample task processing flow and decreasing the probability that the output result of the initial second strategy model is the fourth sample task processing flow.

2. The method as described in claim 1, wherein determining the first parameter tuning information corresponding to the processing flow of each sample task based on the processing results of each sample task includes: A first sample task processing flow and a second sample task processing flow are determined, wherein the first sample task processing flow is any one of the plurality of sample task processing flows, and the second sample task processing flow is a sample task processing flow other than the first sample task processing flow among the plurality of sample task processing flows. Based on the sample task processing results corresponding to the second sample task processing flow, determine the first benchmark information corresponding to the first sample task processing flow; Based on the first benchmark information and the sample task processing results corresponding to the first sample task processing flow, the first parameter tuning information corresponding to the first sample task processing flow is determined.

3. The method as described in claim 2, wherein the sample task processing result includes first reward information corresponding to the sample task processing flow, and the first benchmark information includes first average reward information of the first reward information corresponding to the second sample task processing flow. Based on the first benchmark information and the sample task processing results corresponding to the first sample task processing flow, the first parameter tuning information corresponding to the first sample task processing flow is determined, including: Based on the first average reward information and the first reward information corresponding to the first sample task processing flow, the difference in the first reward information corresponding to the first sample task processing flow is determined. Based on the difference in the first reward information, the first parameter tuning information corresponding to the first sample task processing flow is determined.

4. The method as described in claim 1, wherein, based on the target second strategy model, the second parameter tuning information corresponding to each sample task processing step in the sample task processing flow is obtained, including: For each sample task processing step in any sample task processing flow, the second reward information corresponding to that sample task processing step is obtained through the target second strategy model. Based on the second reward information corresponding to each sample task processing step, the second parameter tuning information corresponding to that sample task processing step is obtained.

5. The method as described in claim 4, wherein obtaining the second reward information corresponding to the sample task processing step through the target second strategy model includes: Obtain the first sample task processing step that precedes this sample task processing step; The sample task status corresponding to the sample task processing step and the first sample task processing step are input into the target second strategy model to obtain the second reward information corresponding to the sample task processing step. The second reward information is determined according to a first probability, where the first probability is the probability of the sample task action corresponding to the sample task processing step as output by the target second strategy model.

6. The method as described in claim 4, wherein obtaining the second parameter tuning information corresponding to each sample task processing step based on the second reward information corresponding to each sample task processing step includes: Based on the second reward information corresponding to the sample task processing steps other than the sample task processing step, determine the second benchmark information corresponding to the sample task processing step. Based on the second baseline information and the second reward information corresponding to the sample task processing step, the second parameter tuning information corresponding to the sample task processing step is obtained.

7. The method of claim 6, wherein the second benchmark information includes the second average reward information corresponding to the second reward information of sample task processing steps other than the sample task processing step; Based on the second baseline information and second reward information corresponding to the sample task processing step, the second parameter tuning information corresponding to the sample task processing step is obtained, including: The difference in the second reward information is determined based on the second average reward information and the second reward information corresponding to the sample task processing steps; Based on the difference in the second reward information, the second parameter tuning information corresponding to the sample task processing step is obtained.

8. The method as described in any one of claims 1 to 7, wherein the initial first policy model is trained based on each first parameter tuning information and each second parameter tuning information to obtain a target first policy model, comprising: For each sample task processing step, obtain the second parameter tuning information corresponding to that sample task processing step, and obtain the first parameter tuning information corresponding to the sample task processing flow in which that sample task processing step is located. Based on the second parameter tuning information and the first parameter tuning information corresponding to the sample task processing step, the target parameter tuning information corresponding to the sample task processing step is obtained. Based on the target parameter tuning information corresponding to each sample task processing step, the initial first policy model is trained to obtain the target first policy model.

9. The method as described in claim 8, wherein the initial first policy model is trained based on the target parameter tuning information corresponding to each sample task processing step to obtain the target first policy model, comprising: For each sample task processing step, the sample task state corresponding to the sample task processing step and the second sample task processing step are input into the initial first strategy model to obtain the second probability output by the initial first strategy model. The second sample task processing step is the sample task processing step that is located before the sample task processing step in the sample task processing flow, and the second probability is the probability of the sample task action corresponding to the sample task processing step as output by the initial first strategy model. Based on the second probability corresponding to each sample task processing step and the target parameter tuning information, the initial first policy model is trained to obtain the target first policy model.

10. The method as described in claim 9, wherein the initial first policy model is trained based on the second probability corresponding to each sample task processing step and the target parameter tuning information to obtain the target first policy model, comprising: For each sample task processing step, the local loss value corresponding to that sample task processing step is calculated based on the second probability and target parameter tuning information. The global loss value is obtained based on the local loss value corresponding to each sample task processing step. Based on the global loss value, the initial first policy model is trained to obtain the target first policy model.

11. The method of claim 10, wherein calculating the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step and the target parameter tuning information includes: Obtain the first probability distribution output by the initial first strategy model, wherein the first probability distribution is the sampling probability of each sample task action of the initial first strategy model in the sample task state corresponding to the sample task processing step; Obtain a baseline strategy model, and input the sample task state corresponding to the sample task processing step and the second sample task processing step into the baseline strategy model to obtain a second probability distribution, wherein the second probability distribution is the sampling probability of each sample task action of the baseline strategy model under the sample task state corresponding to the sample task processing step. Calculate the difference between the first probability distribution and the second probability distribution to obtain the distribution difference value corresponding to the task processing steps of the sample. Based on the second probability, target parameter tuning information, and distribution difference value corresponding to the processing step of the sample task, calculate the local loss value corresponding to the processing step of the sample task.

12. The method of claim 11, wherein the target parameter tuning information includes the target parameter tuning value; Based on the second probability, target parameter tuning information, and distribution difference value corresponding to the processing step of the sample task, calculate the local loss value corresponding to the processing step of the sample task, including: Calculate the product of the second probability corresponding to the processing step of the sample task and the target parameter tuning value; Calculate the first difference between the product and the distribution difference value corresponding to the sample task processing step to obtain the local loss value corresponding to the sample task processing step.

13. The method of claim 10, wherein calculating the local loss value corresponding to the sample task processing step based on the second probability corresponding to the sample task processing step and the target parameter tuning information includes: Obtain the third probability corresponding to the sample task processing step, wherein the third probability is the historical second probability in the historical training rounds corresponding to the sample task processing step; Based on the second probability, third probability, and target parameter tuning information corresponding to the processing step of the sample task, calculate the local loss value corresponding to the processing step of the sample task.

14. The method of claim 13, wherein the target parameter tuning information includes the target parameter tuning value; Based on the second probability, third probability, and target parameter tuning information corresponding to the processing step of the sample task, calculate the local loss value corresponding to the processing step of the sample task, including: Calculate the ratio of the first probability to the second probability to the third probability corresponding to the processing step of the sample task; Calculate the product of the first ratio and the target parameter tuning value to obtain the local loss value corresponding to the sample task processing step.

15. The method of claim 1, wherein the initial state of the sample task is input into the initial second policy model, and the initial second policy model is trained according to a preset training objective to obtain a target second policy model, comprising: The initial state of the sample task is input into the initial second strategy model to obtain the fourth probability that the output of the initial second strategy model is the third sample task processing flow, and the fifth probability that the output of the initial second strategy model is the fourth sample task processing flow. Calculate the second ratio of the fourth probability to the fifth probability; Based on the second ratio, the initial second policy model is trained to obtain the target second policy model.

16. The method of claim 15, wherein training the initial second policy model according to the second ratio to obtain a target second policy model comprises: Obtain a baseline strategy model, input the initial state of the sample task into the baseline strategy model, and obtain the output result of the baseline strategy model as the sixth probability of the third sample task processing flow, and obtain the output result of the baseline strategy model as the seventh probability of the fourth sample task processing flow. Calculate the third ratio of the sixth probability to the seventh probability, and calculate the second difference between the third ratio and the second ratio; The initial second policy model is trained based on the second difference to obtain the target second policy model.

17. The method of claim 1, wherein the initial state of the sample task is input into the initial second policy model, and the initial second policy model is trained according to a preset training objective to obtain a target second policy model, comprising: For each sample task processing flow, if the sample task processing flow is the third sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the eighth probability of the sample task processing flow. Based on the eighth probability, the first loss value corresponding to the sample task processing flow is obtained. When the sample task processing flow is the fourth sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the ninth probability of the sample task processing flow. Based on the ninth probability, the second loss value corresponding to the sample task processing flow is obtained. Based on the first loss value and the second loss value, the initial second policy model is trained to obtain the target second policy model.

18. The method of claim 17, wherein when the sample task processing flow is a third sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the result output by the initial second policy model is the eighth probability of the sample task processing flow, comprising: When the sample task processing flow is the third sample task processing flow, the third sample task processing step is determined, wherein the third sample task processing step is any sample task processing step in the sample task processing flow. The sample task state corresponding to the third sample task processing step and the sample task processing step preceding the third sample task processing step are input into the initial second strategy model to obtain the output of the initial second strategy model as the tenth probability of the sample task action corresponding to the third sample task processing step. Based on the tenth probability corresponding to each sample task processing step in the sample task processing flow, the eighth probability corresponding to the sample task processing flow is obtained. Accordingly, when the sample task processing flow is the fourth sample task processing flow, the initial state of the sample task is input into the initial second policy model, and the output of the initial second policy model is the ninth probability of the sample task processing flow, including: When the sample task processing flow is the fourth sample task processing flow, the fourth sample task processing step is determined, wherein the fourth sample task processing step is any sample task processing step in the sample task processing flow. The sample task state corresponding to the fourth sample task processing step and the sample task processing steps preceding the fourth sample task processing step are input into the initial second strategy model to obtain the output of the initial second strategy model as the eleventh probability of the sample task action corresponding to the fourth sample task processing step. Based on the eleventh probability corresponding to each sample task processing step in the sample task processing flow, the ninth probability corresponding to the sample task processing flow is obtained.

19. A model training method applied to cloud-side devices, comprising: The receiving end device sends multiple sample task processing flows and sample task processing results corresponding to each sample task processing flow. The sample task processing flow includes at least one sample task processing step. Each sample task processing step includes a sample task action and a sample task state. The sample task action is determined by the initial first strategy model based on the sample task state. Based on the processing results of each sample task, the first parameter tuning information corresponding to the processing flow of each sample task is determined, wherein the first parameter tuning information reflects the overall quality of the sample task processing flow. Based on the target second strategy model, second parameter tuning information corresponding to each sample task processing step in each sample task processing flow is obtained, wherein the second parameter tuning information reflects the local quality of the sample task processing step; the target second strategy model is obtained by training using the following method: based on the processing results of each sample task, the sample task processing flow is divided to obtain a third sample task processing flow and a fourth sample task processing flow, wherein the sample task processing result corresponding to the third sample task processing flow is better than the sample task processing result corresponding to the fourth sample task processing flow; the initial state of the sample task is obtained, the initial state of the sample task is input into the initial second strategy model, and the initial second strategy model is trained according to a preset training objective to obtain the target second strategy model, wherein the preset training objective includes increasing the probability that the output result of the initial second strategy model is the third sample task processing flow and decreasing the probability that the output result of the initial second strategy model is the fourth sample task processing flow; Based on the first and second parameter tuning information, the initial first policy model is trained to obtain the target first policy model. Obtain the model parameters of the target first strategy model and send the model parameters to the end-side device.

20. A model training method applied to a text interaction task, comprising: Multiple sample text interaction processes and corresponding sample text interaction results are obtained. The sample text interaction process includes at least one round of sample text interaction. Each round of sample text interaction includes a sample text interaction action and a sample text interaction state. The sample text interaction state includes a first input text input to the initial first strategy model. The sample text interaction action includes a second response text output by the initial first strategy model. The sample text interaction action is determined by the initial first strategy model based on the sample text interaction state. Based on the text interaction results of each sample, the first parameter tuning information corresponding to the text interaction process of each sample is determined, wherein the first parameter tuning information reflects the overall quality of the text interaction process of the sample. Based on the target second strategy model, the second parameter tuning information corresponding to each round of sample text interaction in each sample text interaction processing flow is obtained, wherein the second parameter tuning information reflects the local quality of each round of sample text interaction; Based on the first and second parameter tuning information, train the initial first policy model to obtain the target first policy model; The target second strategy model is obtained by training using the following method: Based on the interaction results of each sample text, the sample text interaction process is divided to obtain the third sample text interaction process and the fourth sample text interaction process. The sample text interaction result corresponding to the third sample text interaction process is better than the sample text interaction result corresponding to the fourth sample text interaction process. Obtain the initial state of the sample text interaction, input the initial state of the sample text interaction into the initial second strategy model, and train the initial second strategy model according to the preset training objective to obtain the target second strategy model. The preset training objective includes increasing the probability that the output result of the initial second strategy model is the third sample text interaction process and decreasing the probability that the output result of the initial second strategy model is the fourth sample text interaction process.

21. A text generation method, comprising: Get the second input text; The second input text is input into the target first strategy model to obtain the second response text corresponding to the second input text, wherein the target first strategy model is trained by the model training method of any one of claims 1 to 20.

22. A text generation method, applied to a cloud-side device, comprising: The second input text sent by the receiving device; The second input text is input into the target first strategy model to obtain the second response text corresponding to the second input text, wherein the target first strategy model is trained by the model training method of any one of claims 1 to 20; The second response text is sent to the end device.

23. An electronic device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 22.

24. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 22.

25. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 22.

26. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 22.

Images