Optimization method and device for improving near-end strategy based on language model, and electronic equipment

Abstract

The invention discloses an optimization method and device for improving a near-end strategy based on a language model and electronic equipment, and the method comprises the steps: determining a plurality of answers corresponding to question data, and determining a first probability value of a single vocabulary in each answer based on the plurality of answers; determining a second probability value of a single vocabulary in each answer based on a reference model in the language model, and determining a KL divergence based on the first probability value and the second probability value; determining an instant reward for each answer based on a reward model in the language model and the target reward signal; and determining a target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence to update a policy model of the language model based on the target loss function. According to the method provided by the invention, the memory occupation problem in the near-end strategy training process can be relieved, and the memory usage is reduced, so that the concurrency is improved, and the answer generation and training process is accelerated.

Description

Technical Field

[0001] This application relates to the technical field of language models, and in particular, to an optimization method for improving proximal policy based on a language model, an electronic device, and an optimization device for improving proximal policy based on a language model. Background Art

[0002] Language models (such as large language models (LLMs, Large Language Model)) are a hot topic in current artificial intelligence technology. The current large language models refer to large models with more than tens of billions or even hundreds of billions of parameters, mainly based on the transformer structure (a deep neural network structure). The pre-training and post-training processes of LLMs are the prerequisites for the deployment and use of large models. Pre-training means training the LLM in a large-scale, diverse, unlabeled text corpus of trillions of tokens (the information of a single word understood by a computer, which can be understood as a standard word or Chinese word). By using the loss function of the language model (cross-entropy loss), the probability of generating a text sequence is optimized and improved. The goal is to learn the grammar and semantic structure of the language. The important feature of this stage is the huge amount of computation, and a training cluster needs to be formed to complete the training process of the LLM and a large amount of data.

[0003] After the pre-training is completed, the LLM already has the basic language understanding ability. Post-training is carried out after the pre-training of the LLM is completed. The post-training stage pays more attention to the performance of the model on specific tasks and the alignment with human preferences. The post-training process is crucial in the development of language models. Although pre-training endows the model with powerful language understanding ability, these abilities are often general and lack fine-tuning for specific tasks and human preferences. Through supervised fine-tuning and reinforcement learning, post-training enables the model to better adapt to the needs of specific tasks and improve its performance in practical applications. Especially in aligning with human preferences and obtaining logical reasoning ability, reinforcement learning plays a key role. Although there are supplements such as simple schemes like DPO (Direct Preference Optimization), reinforcement learning is still indispensable in terms of generalization and universality. Reinforcement learning guides the model to make more human-logical decisions in complex tasks through a reward mechanism, thereby improving the practicality and reliability of the model.

[0004] PPO (Proximal Policy Optimization Algorithms) is the mainstream method used in the post-training reinforcement learning stage of LLM. PPO is an efficient reinforcement learning algorithm for training the policy of an agent. In the LLM model, the role of the PPO algorithm is to continuously optimize the behavior policy of the agent so that it can make optimal decisions when facing different language tasks. The core idea of the PPO algorithm is to introduce a constraint term when optimizing the policy to ensure that the policy update is not too radical, thus ensuring the stability of the training process. Through continuous iteration and optimization, the PPO algorithm can help the agent gradually improve its ability to handle language tasks. The usage process of PPO in LLM fine-tuning is as follows: (1) Sample a question from the question pool and give it to the LLM for generation. The given question and the generated answer represent the environmental state. (2) Based on the question + answer, calculate the value of the current state and give an immediate reward; in addition, it is also necessary to judge the impact of this action on the subsequent policy generation and give a future reward. (3) The generated answer, question, and the obtained reward, value and other data are used as experience data, and gradient descent and other methods are used to update the policy. (4) After the current policy is updated, a new round of experience data generation and policy update are carried out.

[0005] However, the memory occupancy problem in the PPO training process of the above method includes forward inference of two models, forward and backward calculations of two models, and the obtained reward signal is probabilistic, and the rewards given in some scenarios are not accurate enough. Summary of the Invention

[0006] This application aims to solve at least one of the technical problems in the related art to some extent. To this end, the first object of this application is to propose an optimization method for an improved proximal policy based on a language model, determine multiple answers corresponding to the problem data, and determine the first probability value of a single word in each answer based on the multiple answers. Determine the second probability value of a single word in each answer based on the reference model in the language model, and determine the KL divergence based on the first probability value and the second probability value. Determine the immediate reward of each answer based on the reward model and the target reward signal in the language model. Determine the target loss function based on the problem data, answer, first probability value, immediate reward and KL divergence, so as to update the policy model of the language model based on the target loss function. Thus, it is possible to alleviate the memory occupancy problem in the proximal policy training process, reduce memory usage, improve concurrency, and accelerate the generation and training process of answers.

[0007] The second object of this application is to propose an electronic device.

[0008] The third object of this application is to propose an optimization device for an improved proximal policy based on a language model.

[0009] To achieve the above object, an optimization method for improving the proximal policy based on a language model is proposed in the first aspect embodiment of the present application. The method includes: determining a plurality of answers corresponding to the question data, and determining a first probability value of a single word in each answer based on the plurality of answers; determining a second probability value of a single word in each answer based on a reference model in the language model, and determining the KL divergence based on the first probability value and the second probability value; determining an immediate reward for each answer based on a reward model and a target reward signal in the language model; determining a target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function.

[0010] According to the optimization method for improving the proximal policy based on a language model in the embodiment of the present application, a plurality of answers corresponding to the question data are determined, a first probability value of a single word in each answer is determined based on the plurality of answers, a second probability value of a single word in each answer is determined based on a reference model in the language model, the KL divergence is determined based on the first probability value and the second probability value, an immediate reward for each answer is determined based on a reward model and a target reward signal in the language model, and a target loss function is determined based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function. Thus, this method can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, improve concurrency, and accelerate the generation and training process of answers.

[0011] In addition, the optimization method for improving the proximal policy based on a language model according to the above embodiment of the present application may further have the following additional technical features:

[0012] According to an embodiment of the present application, determining the immediate reward based on the reward model and the target reward signal in the language model includes: determining a first reward of a single word in each answer based on the reward model; determining a second reward of each answer based on the target reward signal; determining the immediate reward based on the sum of the product of the first reward and a first preset coefficient and the product of the second reward and a second preset coefficient.

[0013] According to an embodiment of the present application, the target reward signal includes a reward signal in a first domain and a reward signal in a second domain. Among them, the first domain is the mathematics domain, and the second domain is the code domain. Determining the second reward for each answer based on the target reward signal includes: when the question data belongs to the first domain, if the information extracted from the answer includes the correct answer and the calculation process, determining the second reward as the first preset threshold; if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as the second preset threshold; if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, where the first preset threshold is greater than the second preset threshold; when the question data belongs to the second domain, if the code extracted from the answer can be compiled and executed successfully using the local compiler, determining the second reward as the first preset threshold; if the code extracted from the answer fails to compile or execute, determining the second reward as zero.

[0014] According to an embodiment of the present application, determining the target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence includes: determining the target reward based on the sum of the immediate reward and the KL divergence; determining the relative advantage estimate based on the target reward; determining the target loss function based on the relative advantage estimate, the KL divergence, the question data, the answer, and the first probability value.

[0015] According to an embodiment of the present application, determining the relative advantage estimate based on the target reward includes: determining the average value and the standard deviation based on the target rewards of all answers; determining the relative advantage estimate corresponding to the answer based on the ratio of the difference between the target reward of each answer and the average value to the standard deviation.

[0016] According to an embodiment of the present application, the target loss function is determined by the following formula:

[0017]

[0018] where J(θ) is the target loss function, θ is the parameter of the current policy model, θ old represents the parameter of the previous policy model, q is the question data drawn from the question pool, p(Q) is the probability distribution of all possible questions in the question pool, o i is the i-th answer generated using the policy model for the question data q, N is the number of answers generated for each question data q, is the N answers generated using the policy model, π θ (o iP(o|q) is the first probability value of the answer o determined by the current policy model under the question data q i ; P(o|q) is the first probability value of the answer o determined by the previous policy model under the question data q i ; GAE i is the relative advantage estimate of the i-th answer; D KL (π θ ||π ref ) is the KL divergence between the current policy model π θ and the reference model π ref ; ε and a are hyperparameters, and E is the expectation operator, representing the average over all combinations of question data and answers.

[0019] According to an embodiment of the present application, determining the first probability value of a single word in each answer based on the multiple answers includes: determining a vector of each single word in the multiple answers based on the policy model in the language model; determining a probability distribution based on the vector and a preset function, where the preset function is the softmax function; and taking the logarithm of the probability distribution to obtain the first probability value.

[0020] According to an embodiment of the present application, determining the KL divergence based on the first probability value and the second probability value includes: determining the KL divergence based on the difference between the first probability value and the second probability value.

[0021] To achieve the above object, an electronic device proposed in the second aspect embodiment of the present application includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned optimization method for improving the proximal policy based on the language model is implemented.

[0022] According to the electronic device of the embodiment of the present application, by executing the above-mentioned optimization method for improving the proximal policy based on the language model, the memory occupancy problem in the proximal policy training process can be alleviated, the memory usage can be reduced, so as to improve concurrency and accelerate the generation and training process of answers.

[0023] To achieve the above object, an optimization device for improving the proximal policy based on a language model according to a third aspect embodiment of the present application includes: a first determination module, configured to determine multiple answers corresponding to problem data, and determine a first probability value of a single word in each answer based on the multiple answers; a second determination module, configured to determine a second probability value of a single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value; a third determination module, configured to determine an immediate reward for each answer based on a reward model and a target reward signal in the language model; and an update module, configured to determine a target loss function based on the problem data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update a policy model of the language model based on the target loss function.

[0024] For the optimization device for improving the proximal policy based on a language model according to an embodiment of the present application, the first determination module is configured to determine multiple answers corresponding to problem data, and determine a first probability value of a single word in each answer based on the multiple answers, the second determination module is configured to determine a second probability value of a single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value, the third determination module is configured to determine an immediate reward for each answer based on a reward model and a target reward signal in the language model, and the update module is configured to determine a target loss function based on the problem data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update a policy model of the language model based on the target loss function. Thus, the device can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, so as to improve concurrency and accelerate the generation and training process of answers.

[0025] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of an optimization method for improving the proximal policy based on a language model according to an embodiment of the present application;

[0027] Figure 2 It is a flowchart of an optimization method for improving the proximal policy based on a language model according to a specific example of the present application;

[0028] Figure 3 It is a block diagram of an electronic device according to an embodiment of the present application;

[0029] Figure 4 It is a block diagram of an optimization device for improving the proximal policy based on a language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application and should not be construed as a limitation to the present application.

[0031] Currently, a total of 4 models are involved in the post-training implementation process of PPO in related technologies, namely the actor model (policy model), the reference model, the reward model, and the critic model (state value model). The overall implementation process is divided into two major steps. One is the generation of empirical data; the other is the adjustment of model parameters using the empirical data. The specific steps are as follows: (1) Sample a batch of problem data (prompt) from the problem pool and use the actor model to generate an answer (response). (2) Generate empirical data from the prompt + response data: (a) The actor model calculates log probs for the prompt + response (the result of taking the logarithm of the probability of each word or token in each generated answer (response) by the large prediction model); (b) The init model calculates log probs for the prompt + response; (c) The reward model calculates the immediate reward; (d) Use the critic model to calculate the state value Value; (e) Calculate the KL (Kullback-Leibler Divergence) divergence (relative entropy) using the log probs of a and b; (3) Calculate the reward and GAE (Generalized Advantage Estimation), and the reward calculation is the immediate reward + KL divergence; use the newly obtained reward, as well as value and logprobs to calculate GAE; (4) Use the empirical data from steps (2) and (3) to update the actor model (PPO loss function) and update the critic model (value loss function). However, there are problems with memory occupancy during the PPO training process in this way, including two models performing forward inference, two models performing forward and backward calculations, and the reward signal given by the reward model being probabilistic, and the rewards given in some scenarios are not accurate enough. In addition, the generation and training of empirical data are completely asynchronous, and the policy update in PPO cannot be performed until the data generation is completed.

[0032] To this end, the present application proposes an optimization method for improving the proximal policy based on a language model to alleviate the memory occupancy problem during PPO training, reduce device memory usage, and improve the scheme of directly rewarding the current state-action using only a reward model. A verifiable and accurate reward signal is designed, and the process of completely asynchronous generation of experience data and PPO policy update is improved by using a delayed model update scheme to improve device computing efficiency.

[0033] The following describes the optimization method for improving the proximal policy based on a language model, an electronic device, and an optimization device for improving the proximal policy based on a language model proposed in the embodiments of the present application with reference to the accompanying drawings.

[0034] Figure 1 It is a flowchart of an optimization method for improving the proximal policy based on a language model according to an embodiment of the present application.

[0035] As Figure 1 shown, the optimization method for improving the proximal policy based on a language model in the embodiments of the present application may include the following steps:

[0036] S1. Determine multiple answers corresponding to the question data, and determine the first probability value of each single word in each answer based on the multiple answers.

[0037] S2. Determine the second probability value of each single word in each answer based on the reference model in the language model, and determine the KL divergence based on the first probability value and the second probability value.

[0038] S3. Determine the immediate reward of each answer based on the reward model and the target reward signal in the language model.

[0039] S4. Determine the target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, and update the policy model of the language model based on the target loss function.

[0040] Specifically, first, determine multiple answers corresponding to the question data, and determine the first probability value of each single word in each answer based on the multiple answers. For example, the policy model can be used to generate multiple answers (responses) for a given question data (prompt). And for each generated answer, the policy model outputs the predicted probability distribution of each word, which is then converted into a probability value through a preset function, and the logarithm of these probability values is taken to obtain the first probability value of each word. By generating multiple answers, the model can explore different answer possibilities and increase the diversity of answers. The first probability value reflects the model's prediction of each word and is used for subsequent loss function calculation.

[0041] Determine the second probability value of each individual word in each answer according to the reference model in the language model, and determine the KL divergence based on the first probability value and the second probability value. That is, the reference model is a model with the same structure as the policy model but with fixed parameters, used to generate the second probability value of each individual word in each answer. After determining the first probability value and the second probability value, the KL divergence can be determined according to the first probability value and the second probability value to measure the difference between the two probability distributions. That is, the reference model provides a stable comparison benchmark, which helps to evaluate whether the changes in the policy model are developing in the expected direction. The KL divergence is used to control the amplitude of policy updates to prevent drastic changes in the policy during the update process.

[0042] The immediate reward for each answer can be determined according to the reward model and the target reward signal in the language model. That is to say, the reward model can be used to evaluate the quality of each answer and give the immediate reward, and the target reward signal, such as correctness, relevance, etc., can be combined to further adjust the immediate reward. The immediate reward provides direct feedback to the model on the quality of the answer, guiding the model to optimize the generation of the answer, and by combining the target reward signal, the model can better understand and meet the requirements of specific tasks.

[0043] After determining the first probability value, the KL divergence, and the immediate reward, the target loss function can be determined according to the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model according to the target loss function. That is to say, the question data, the answer, the first probability value, the immediate reward, and the KL divergence can be combined to construct the target loss function. For example, design a loss function that not only considers the policy gradient but also the quality of the answer and the stability of the policy. Such as L(θ) = α * CrossEntropyLoss + β * RewardLoss + γ * KLLoss + δ * EntropyLoss, where α, β, γ, δ are the corresponding coefficients, CrossEntropyLoss is the cross-entropy loss based on the first probability value, encouraging the model to generate more accurate answers, RewardLoss is the loss based on the immediate reward, encouraging the model to generate higher-quality answers, KLLoss is the loss based on the KL divergence, restricting the amplitude of policy updates, and EntropyLoss is the entropy loss, encouraging the model to explore more possible answers. Adjust the weight coefficients of each part in the above loss function through methods such as cross-validation or grid search to find the optimal loss function configuration.

[0044] After determining the target loss function, an optimization algorithm (such as gradient descent) can be used to minimize the target loss function to update the parameters of the policy model. That is, the target loss function comprehensively considers the prediction probability of the policy model, the reward feedback, and the policy stability, guiding the model to develop in the direction of improving the answer quality and stability. By minimizing the target loss function, the model can learn to generate answers that better meet the task requirements and human preferences. This method helps to improve the performance of the language model on specific tasks while maintaining the stability and efficiency of the training process.

[0045] Thus, by continuously generating answers, evaluating answer quality, calculating losses, and updating model parameters, the language model can gradually improve its performance on specific tasks.

[0046] According to an embodiment of the present application, determining the immediate reward based on the reward model and the target reward signal in the language model includes: determining the first reward for each single word in each answer based on the reward model; determining the second reward for each answer based on the target reward signal; determining the immediate reward based on the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient. The first preset coefficient and the second preset coefficient can be determined according to the actual situation.

[0047] Specifically, when determining the immediate reward based on the reward model and the target reward signal in the language model, the calculation method of the immediate reward combines the reward model (Reward Model) and the target reward signal (Target Reward Signal), aiming to comprehensively consider the quality of the answers generated by the model at different levels, so as to more accurately guide the training and optimization of the model. The first reward for each single word in each answer can be determined according to the reward model. The role of the reward model is to evaluate each word (or each token) in the generated answer (response) to determine the quality of each word. The specific steps are as follows: Input the question (prompt) and the generated answer (response) into the reward model. The reward model calculates a reward value for each word (token) in the answer, indicating the quality or relevance of the word in the current context. These reward values constitute the first reward (reward1). The reward model outputs a reward vector with the same length as the answer, and each element corresponds to the reward value of a word.

[0048] Determine the second reward for each answer based on the target reward signal, which evaluates the quality of the entire answer rather than word by word. It can be based on the goals of specific tasks. For example, for math problems, the target reward signal can be whether the answer is correct. For code generation tasks, the target reward signal can be whether the code passes the tests. For natural language tasks, the target reward signal can be the fluency, relevance, or safety of the answer. The specific steps are as follows: The question (prompt) and the generated answer (response) can be input into the target reward signal evaluation module. According to predefined rules or models, calculate the reward value of the entire answer, representing the overall quality of the answer. This reward value constitutes the second reward (reward2), and the target reward signal evaluation module outputs a scalar value representing the reward of the entire answer.

[0049] After determining the first reward and the second reward, the immediate reward can be determined based on the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient. That is to say, the immediate reward is the result of comprehensively considering the first reward and the second reward, and preset coefficients are introduced to balance the contributions of the two. The specific steps are as follows: Weighting of the first reward: Multiply the first reward (reward1) by the first preset coefficient α. Since the first reward is a vector, it remains in vector form after weighting. Weighting of the second reward: Multiply the second reward (reward2) by the second preset coefficient β. Since the second reward is a scalar, it remains a scalar after weighting. Calculation of the immediate reward: Add the weighted first reward and the second reward to obtain the immediate reward (reward). That is, it is determined by the formula immediate reward = first reward * first preset coefficient + second reward * second preset coefficient.

[0050] For example, for math problems, the first reward: The reward model calculates the reward value for each vocabulary (such as numbers, operators), evaluating its rationality in mathematical logic. The second reward: The target reward signal evaluates whether the entire answer is correct (e.g., whether the correct answer is given). The immediate reward: Comprehensively consider the reward of each vocabulary and the correctness of the entire answer to guide the model to generate more accurate math answers. For code generation, the first reward: The reward model calculates the reward value for each vocabulary in the code (such as keywords, variable names), evaluating its syntactic correctness. The second reward: The target reward signal evaluates whether the entire code passes the test cases. The immediate reward: Comprehensively consider the reward of each vocabulary and the correctness of the entire code to guide the model to generate more reliable code. For natural language tasks, the first reward: The reward model calculates the reward value for each vocabulary, evaluating its relevance and fluency in the context. The second reward: The target reward signal evaluates the safety, relevance, or user satisfaction of the entire answer. The immediate reward: Comprehensively consider the reward of each vocabulary and the quality of the entire answer to guide the model to generate answers that better meet human preferences.

[0051] Thus, by introducing the target reward signal, the reward signal can be customized according to the requirements of different tasks, enabling the model to better adapt to specific tasks. By combining the word-by-word reward and the overall reward, the quality of the generated answers can be evaluated more comprehensively. Additionally, it can be applied to multiple fields (such as mathematics, code, natural language, etc.), and the definition of the reward signal can be adjusted according to specific tasks. Through the comprehensive reward mechanism, the performance of the language model on specific tasks can be guided more effectively, while avoiding the limitations that may be brought by simply relying on a single reward signal.

[0052] According to an embodiment of the present application, the target reward signal includes a reward signal for the first field and a reward signal for the second field. Among them, the first field is the mathematics field, and the second field is the code field. Determining the second reward for each answer based on the target reward signal includes: when the question data belongs to the first field, if the information extracted from the answer includes the correct answer and the calculation process, determining the second reward as the first preset threshold; if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as the second preset threshold; if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, where the first preset threshold is greater than the second preset threshold; when the question data belongs to the second field, if the code extracted from the answer can be compiled and executed successfully using the local compiler, determining the second reward as the first preset threshold; if the code extracted from the answer fails to compile or execute, determining the second reward as zero.

[0053] Specifically, the target reward signal is designed to determine the second reward for each answer according to the field to which the question data belongs (the first field is the mathematics field, and the second field is the code field), aiming to provide targeted reward signals for the generated answers according to the characteristics and requirements of different fields, thereby more effectively guiding the training and optimization of the model. That is, the purpose of the target reward signal is to comprehensively evaluate the generated answers and determine whether they meet the quality standards of specific fields. Specifically: in the mathematics field, the target reward signal needs to evaluate whether the answer contains the correct answer and the calculation process. In the code field, the target reward signal needs to evaluate whether the code in the answer can be correctly compiled and executed. In this way, the model can learn the ability to generate high-quality answers in different fields.

[0054] Determine the field to which the question data belongs. When the question data belongs to the first field (mathematics field), if the answer contains the correct answer and the calculation process, the second reward is set to the first preset threshold. For example, the first preset threshold is 1. That is to say, in this case, the answer not only gives the correct answer but also provides the complete calculation process, meeting the high-quality standard of math problems. If the answer contains the correct answer but does not contain the calculation process, the second reward is set to the second preset threshold. For example, the second preset threshold is 0.3. That is to say, in this case, although the answer gives the correct answer, it lacks the calculation process, so the second preset threshold is lower than the first preset threshold. If the answer contains neither the correct answer nor the calculation process, the second reward is set to zero. That is to say, in this case, the answer does not meet the basic requirements of math problems, so no reward is given. For example, the first preset threshold is 1.0 and the second preset threshold is 0.5. For a question with the answer 42, for the math problem: If the answer is "The answer is 42, and the calculation process is as follows:...", the second reward is 1.0. If the answer is "The answer is 42", the second reward is 0.5. If the answer is "The answer is 43", the second reward is 0.

[0055] Determine the field to which the question data belongs. When the question data belongs to the second field (code field), if the code in the answer can be compiled using the local compiler and pass the execution, the second reward is set to the first preset threshold. For example, the first preset threshold is 1. That is to say, in this case, the code can not only be correctly compiled but also pass the execution test, meeting the high-quality standard of code generation. If the code in the answer fails to compile or fails to execute, the second reward is set to zero. That is to say, in this case, the code cannot meet the basic requirements of compilability and executability, so no reward is given. For example, the first preset threshold (threshold_1) = 1.0. For the code problem: If the answer is "python\ndef add(a, b):\n return a + b\n" and the code can be compiled and executed successfully, the second reward is 1.0. If the answer is "python\ndef add(a, b):\n return a - b\n" and the code fails to compile or fails to execute, the second reward is 0.

[0056] Thus, through this design, the target reward signal can provide targeted rewards for the generated answers according to the characteristics and requirements of different fields: In the mathematics field, the reward signal not only focuses on the correctness of the answer but also on the completeness of the calculation process. In the code field, the reward signal focuses on the compilability and executability of the code. This domain-specific reward mechanism can more precisely guide the model to generate high-quality answers while avoiding the limitations that a single reward signal may bring.

[0057] According to an embodiment of the present application, determining a target loss function based on problem data, answers, a first probability value, immediate rewards, and KL divergence includes: determining a target reward based on the sum of the immediate rewards and the KL divergence; determining a relative advantage estimate based on the target reward; and determining the target loss function based on the relative advantage estimate, the KL divergence, the problem data, the answers, and the first probability value.

[0058] Specifically, the determination of the target loss function is a key step in optimizing the policy network. This process comprehensively considers immediate rewards, KL divergence, problem data, answers, and the first probability value. When determining the target loss function based on the problem data, answers, first probability value, immediate rewards, and KL divergence, first determine the target reward based on the sum of the immediate rewards and the KL divergence. Here, the KL divergence is used to measure the magnitude of the policy update to ensure that the update is not too aggressive. That is, the reward value is calculated based on the problem data and answers, as well as the reward model and the target reward signal. Calculate the KL divergence between the new policy (updated policy) and the old policy (policy before update), which represents the magnitude of the policy update, i.e., target reward = immediate reward + KL divergence.

[0059] The relative advantage estimate can be determined based on the target reward. The relative advantage estimate is the difference between the target reward and the average reward, and is used to measure the advantage of the current policy relative to the baseline policy. For example, based on the target reward calculated from the sum of the immediate reward and the KL divergence, the average reward is the average reward value calculated from historical data and used as the baseline. The relative advantage estimate can be determined based on the difference between the target reward and the average reward. Finally, the target loss function can be determined based on the relative advantage estimate, the KL divergence, the problem data, the answers, and the first probability value. That is, the target loss function is the ultimate goal of optimizing the policy network, comprehensively considering the relative advantage estimate, the KL divergence, the problem data, the answers, and the first probability value. For example, in the problem data and answers, visualization techniques can be used to explore the impact of different factors (such as relative advantage estimate, KL divergence, first probability value, etc.) on the policy performance. For example, scatter plots, heat maps, etc. can be used to show the relationship between the relative advantage estimate and the KL divergence of different answers, so as to intuitively understand which factors have the greatest impact on the policy performance.

[0060] Thus, through the above steps, the target loss function can be determined for optimizing the policy network. This process comprehensively considers immediate rewards, KL divergence, problem data, answers, and the first probability value, aiming to maximize the performance of the policy network while ensuring the stability of the policy update. This design of the target loss function based on relative advantage estimate and KL divergence can more effectively guide the training of the policy network and avoid the problem of training instability caused by overly aggressive policy updates. In this way, the model can achieve better performance when dealing with language model tasks.

[0061] Further, according to an embodiment of the present application, determining a relative advantage estimate based on the target reward includes: determining an average value and a standard deviation based on the target rewards of all answers; determining the relative advantage estimate corresponding to each answer based on the ratio of the difference between the target reward of each answer and the average value to the standard deviation.

[0062] Specifically, when determining the relative advantage estimate based on the target reward, the average value and the standard deviation can be determined according to the target rewards of all answers. The average value of the target rewards of all answers represents the overall quality level of the answers, and the standard deviation of the target rewards of all answers represents the degree of dispersion of the answer quality. For example, the average value and the standard deviation can be determined by the corresponding calculation formulas for calculating the average value and the standard deviation. After determining the average difference and the standard deviation, the relative advantage estimate corresponding to each answer can be determined based on the ratio of the difference between the target reward of each answer and the average value to the standard deviation. For example, the relative advantage estimate is determined by the formula to determine the relative advantage estimate. Where reward i is the reward of the i-th answer. mean((reward 1 ,reward 2 ...,(reward N )) is the average value of the rewards of all answers, and std((reward 1 ,reward 2 ,...,reward N )) is the standard deviation of the rewards of all answers. That is, the relative advantage estimate is the difference between the target reward of each answer and the average reward, divided by the standard deviation, representing the relative quality of each answer compared to the overall. Through this normalization process, the relative advantage estimate can more fairly compare the quality of different answers, unaffected by the number of answers and the quality distribution. Where N represents N different answers to the same question, that is, an average reward is constructed, and the current reward is subtracted from the average reward, denoted as the advantage of the current action state compared to the baseline. This method saves memory and computational effort compared to previously using the critic model (state value model) to calculate the state value.

[0063] Thus, through the normalization process, the relative advantage estimate can more fairly compare the quality of different answers, and the relative advantage estimate can adapt to different tasks and data distributions because it is based on the relative quality of the answers rather than the absolute quality. In addition, using the average value and the standard deviation for normalization can reduce the influence of extreme values on the relative advantage estimate and improve the stability of training. Thus, it can be used to optimize the policy network (actor model (policy model)) to improve the quality of answers by maximizing the relative advantage estimate. In the PPO algorithm, the relative advantage estimate can be used as the weight of the policy gradient to guide the update of the policy network.

[0064] Furthermore, according to an embodiment of the present application, the target loss function is determined by the following formula:

[0065]

[0066] where J(θ) is the target loss function, θ is the parameter of the current policy model, and θ old represents the parameter of the previous policy model, q is the question data sampled from the question pool, p(Q) is the probability distribution of all possible questions in the question pool, o i is the i-th answer generated using the policy model for the question data q, N is the number of answers generated for each question data q, is the N answers generated using the policy model, and π θ (o i |q) is the first probability value of determining the answer o i by the current policy model under the question data q, is the first probability value of determining the answer o i by the previous policy model under the question data q, GAE i is the relative advantage estimate of the i-th answer, and D KL (π θ ||π ref ) is the KL divergence between the current policy model π θ and the reference model π ref . ε and a are hyperparameters, and E is the expected value operator, indicating averaging over all combinations of question data and answers.

[0067] Specifically, the process of determining the target loss function is the core step in the reinforcement learning algorithm, which guides the optimization direction of the policy model. In the embodiment of the present application, the target loss function can be determined by the above formula (1), that is, by calculating the probability ratio of generating the answer oi by the current policy model π θ and the previous policy model π θold , and applying a clipping operation to limit the policy ratio within the range of [1 - ∈, 1 + ∈] to prevent the policy update from being too aggressive. And calculate the relative advantage estimate GAEi of each answer to measure the advantage of the answer relative to the average level, and calculate the KL divergence between the current policy π θ model and the reference policy reference model π ref to measure the difference in their probability distributions. Finally, the average value can be obtained by averaging over all combinations of question data and answers to get the expected value.

[0068] In the above embodiments, a total of three models, namely the policy model, the reward model, and the reference model, are involved, which reduces the memory usage compared to the previous four models (policy model, reward model, reference model, and state-value model). Thus, by pruning the policy ratio and restricting the KL divergence, the policy update is prevented from being too aggressive, the stability of the training process is improved, and combined with the advantage estimation, the policy update pays more attention to potential answers, improving the learning efficiency. Additionally, by introducing the hyperparameters ∈ and a, the amplitude of the policy update and the weight of the KL divergence can be flexibly adjusted to adapt to different task requirements. The advantage estimation enables the model to explore better answers, and by optimizing the objective loss function, the performance of the policy model in generating answers can be improved, making it more in line with the task objectives and human preferences, achieving stable, efficient, and flexible policy optimization, thereby improving the performance of the language model on specific tasks.

[0069] According to an embodiment of the present application, determining a first probability value of a single word in each answer based on multiple answers includes: determining a vector of each single word in the multiple answers based on a policy model in a language model; determining a probability distribution based on the vector and a preset function, where the preset function is the softmax function; and taking the logarithm of the probability distribution to obtain the first probability value.

[0070] Specifically, when determining the first probability value of a single word in each answer based on multiple answers, the vector of each single word in the multiple answers can be determined according to the policy model in the language model. That is, the policy model (actor model) is a core part of the language model and can generate an answer (response) based on the input question (prompt). During the process of generating an answer, the policy model generates a vector for each word, representing the likelihood of the word in the current context. That is, after inputting the question (prompt) into the policy model, the policy model can generate an answer word by word, generating a vector representation of each word at each step. These vectors are the model's predicted outputs for each word, usually referred to as logits. For example, assume the question is: "What's the weather like today?" The policy model generates the answer word by word: "The weather is very sunny today." When generating each word (such as "today", "weather", "very", "sunny"), the policy model outputs the vector representation (logits) of each word. Then, a probability distribution can be determined based on the vector and a preset function, where the preset function is the softmax function, and the logarithm of the probability distribution is taken to obtain the first probability value.

[0071] Such as in the formula q,a~π θ (.|q), q represents the batch of question data (such as 128), a represents the batch of answers corresponding to the batch of question data (corresponding to 128*N), and π θDenote the policy model with parameter θ, which represents an LLM model in the examples of this application. The process of generating a is obtained by calculating the probability of each individual word in the answer output by the LLM. The probabilities corresponding to each question's answer can be connected, and through the softmax function and vector transformation, it is converted into the first probability value logprobs. For example, applying the Softmax function inputs the vector of each word output by the policy model into the Softmax function to obtain the probability distribution of each word. Suppose the vector output by the policy model is: z = [2.0, 1.0, 0.1], then the output of the Softmax function is: Softmax(z) = [e 2.0 / (e 2.0 +e 1.0 +e 0.1 ), e 1.0 / (e 2.0 +e 1.0 +e 0.1 ) e 0.1 / (e 2.0 +e 1.0 +e 0.1 )]. After calculation, Softmax(z) ≈ [0.659, 0.245, 0.096].

[0072] The log probability value is the logarithmic form of the probability value, which is usually used in the optimization process because it can avoid the problem of numerical underflow and is more stable when calculating gradients. When calculating the log probability value, the probability distribution output by the Softmax function can be taken the logarithm to obtain the first probability value (log probs) of each word. That is, assuming p is the probability distribution output by the Softmax function, the first probability value is: log probs = log(p). For example, if the output of the Softmax function is p = [0.659, 0.245, 0.096], then the first probability value log probs = log([0.659, 0.245, 0.096]) ≈ [-0.414, -1.404, -2.343].

[0073] Thus, based on the policy model in the language model, the first probability value of each individual word in each answer is determined, and these first probability values will be used in subsequent policy optimization processes, such as calculating policy gradients or KL divergences, etc. In this way, the large prediction model can better optimize the quality of the generated answers.

[0074] According to an embodiment of the present application, determining the KL divergence based on the first probability value and the second probability value includes: determining the KL divergence based on the difference between the first probability value and the second probability value.

[0075] Specifically, when determining the KL divergence based on the first probability value and the second probability value, the KL divergence can be determined according to the difference between the first probability value and the second probability value. The KL divergence is used to measure the difference between two probability distributions. Specifically, the first probability value and the second probability value are the output probability distributions of the policy model and the reference model at different time steps or under different policies respectively. By calculating the KL divergence between these two probability distributions, the magnitude of the policy update can be measured, thereby controlling the stability of the policy update. It should be noted that the smaller the value of the KL divergence, the more similar the two distributions are. When the KL divergence is zero, it means the two distributions are exactly the same. In reinforcement learning, the KL divergence is used to control the magnitude of the policy update. For example, it can prevent the policy update from being too large. If the KL divergence is too large, it means the policy update is too radical, which may lead to unstable training. By restricting the value of the KL divergence, the stability of the policy update can be ensured. The learning rate can be dynamically adjusted. In some algorithms, the learning rate can be dynamically adjusted according to the value of the KL divergence to achieve a more stable training process. Thus, by restricting the value of the KL divergence, the policy update can be prevented from being too large, thereby improving the stability of training.

[0076] In addition, in an embodiment of the application, an approach of completely asynchronous generation of empirical data and PPO policy update is adopted, and a delayed model update scheme is used to improve the computing efficiency of the device. That is, in reinforcement learning, using the second-newest data for policy training and update is a method to improve computing efficiency and stability, and this method is called asynchronous update or delayed update. That is, always use the second-newest data for policy training and update. For example, use the current policy (through policy model 1) to generate a batch of data (data 1), which includes questions (prompts), answers (responses), as well as relevant rewards and probability values (log probs), and then use the generated data (data 1) to update the policy to obtain a new policy (policy model 2). Then generate data again, that is, use the new policy (policy 2) to generate another batch of data (data 2), and these data can be used for further policy update. Update the policy with the second-newest data: Even if new data (data 2) has been generated, still use the old data (data 1) to update policy 2 to obtain policy 3. This method can reduce the delay of policy update and improve the continuity of training. And while generating new data (data 2), use the old data (data 1) to update the policy, so that the computing resources for generation and training can be used simultaneously, improving the computing efficiency.

[0077] For example, assume that we are training a dialogue system: interact with users using Policy 1 (the current dialogue policy), generate a batch of dialogue data (Data 1), including users' questions and the system's answers, use Data 1 to evaluate the performance of Policy 1, calculate the reward and probability values, and then update the policy to obtain Policy 2. Use Policy 2 to interact with users again, generate a new batch of dialogue data (Data 2), use Data 1 to update Policy 2 to obtain Policy 3. Even though Data 2 has been generated, still use Data 1 to update Policy 2 to obtain Policy 3. This way, new generated data can be immediately utilized to improve the policy without waiting for the processing of Data 2.

[0078] Thus, a method of updating the latest policy using the second latest data can reduce the waiting time and improve the overall training speed by parallelly using the computing resources for generation and training, enabling the generation data process and the training process to be synchronized in one stage, which greatly improves the device computing usage efficiency. Although using the second latest data may cause a certain degree of obsolescence, this method can reduce the delay of policy updates, enabling the policy to adapt to environmental changes faster. By delaying the update, the frequency of policy updates can be reduced, thereby reducing the fluctuations of the policy during the training process and improving the stability of the policy. Additionally, using the second latest data for update can, to a certain extent, balance the relationship between exploration (generating new data) and exploitation (using old data to update the policy), enabling the policy to both explore new possibilities and utilize existing knowledge. In a dynamically changing environment, using the second latest data for update can help the policy adapt to environmental changes faster and improve the adaptability of the policy.

[0079] Next, in combination with Figure 2 describe the method of this application.

[0080] As a specific example, the optimization method of the improved proximal policy based on the language model of this application may include the following steps:

[0081] S101, determine multiple answers corresponding to the question data, determine the vector of each single word in the multiple answers based on the policy model in the language model, and determine the probability distribution based on the vector and a preset function, where the preset function is the softmax function, and take the logarithm of the probability distribution to obtain the first probability value.

[0082] S102, determine the second probability value of each single word in each answer based on the reference model in the language model.

[0083] S103, determine the KL divergence based on the difference between the first probability value and the second probability value.

[0084] S104. Determine the first reward for each individual word in each answer based on the reward model in the language model, determine the second reward for each answer based on the target reward signal, and determine the immediate reward based on the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient.

[0085] S105. Determine the target reward based on the sum of the immediate reward and the KL divergence.

[0086] S106. Determine the mean and standard deviation based on the target rewards of all answers, and determine the relative advantage estimate corresponding to each answer based on the ratio of the difference between the target reward of each answer and the mean to the standard deviation.

[0087] S107. Determine the target loss function based on the relative advantage estimate, the KL divergence, the question data, the answers, and the first probability value.

[0088] S108. Update the policy model of the language model based on the target loss function.

[0089] In summary, according to the optimization method of the improved proximal policy based on the language model in the embodiments of the present application, multiple answers corresponding to the question data are determined, and the first probability value of each individual word in each answer is determined based on the multiple answers. The second probability value of each individual word in each answer is determined based on the reference model in the language model, and the KL divergence is determined based on the first probability value and the second probability value. The immediate reward of each answer is determined based on the reward model and the target reward signal in the language model. The target loss function is determined based on the question data, the answers, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function. Thus, this method can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, improve concurrency, and accelerate the answer generation and training process.

[0090] Corresponding to the above embodiments, the present application also proposes an electronic device.

[0091] As Figure 3 shown, the electronic device 200 in the embodiments of the present application may include: a memory 210, a processor 220, and a program stored on the memory 210 and executable on the processor 220. When the processor 220 executes the program, it implements the above-mentioned optimization method of the improved proximal policy based on the language model.

[0092] According to the electronic device in the embodiments of the present application, by executing the above-mentioned optimization method of the improved proximal policy based on the language model, the memory occupation problem in the proximal policy training process can be alleviated, memory usage can be reduced, concurrency can be improved, and the answer generation and training process can be accelerated.

[0093] Corresponding to the above embodiments, the present application also proposes an optimization device for improving the proximal policy based on a language model.

[0094] As Figure 4 shown, the optimization device 100 for improving the proximal policy based on a language model according to an embodiment of the present application includes: a first determination module 110, a second determination module 120, a third determination module 130, and an update module 140.

[0095] Among them, the first determination module 110 is used to determine multiple answers corresponding to the question data, and determine the first probability value of a single word in each answer based on the multiple answers. The second determination module 120 is used to determine the second probability value of a single word in each answer based on the reference model in the language model, and determine the KL divergence based on the first probability value and the second probability value. The third determination module 130 is used to determine the immediate reward of each answer based on the reward model and the target reward signal in the language model. The update module 140 is used to determine the target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function.

[0096] According to an embodiment of the present application, determining the immediate reward based on the reward model and the target reward signal in the language model includes: determining the first reward of a single word in each answer based on the reward model; determining the second reward of each answer based on the target reward signal; determining the immediate reward based on the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient.

[0097] According to an embodiment of the present application, the target reward signal includes a reward signal in the first domain and a reward signal in the second domain. Among them, the first domain is the mathematical domain, and the second domain is the code domain. Determining the second reward of each answer based on the target reward signal includes: when the question data belongs to the first domain, if the information extracted from the answer includes the correct answer and the calculation process, determining the second reward as the first preset threshold; if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as the second preset threshold; if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, where the first preset threshold is greater than the second preset threshold; when the question data belongs to the second domain, if the code extracted from the answer can be compiled and executed successfully using the local compiler, determining the second reward as the first preset threshold; if the code extracted from the answer fails to compile or execute, determining the second reward as zero.

[0098] According to an embodiment of the present application, determining a target loss function based on problem data, answers, a first probability value, immediate rewards, and KL divergence includes: determining a target reward based on the sum of the immediate rewards and the KL divergence; determining a relative advantage estimate based on the target reward; determining the target loss function based on the relative advantage estimate, the KL divergence, the problem data, the answers, and the first probability value.

[0099] According to an embodiment of the present application, determining a relative advantage estimate based on the target reward includes: determining an average value and a standard deviation based on the target rewards of all answers; determining the relative advantage estimate corresponding to each answer based on the ratio of the difference between the target reward of each answer and the average value to the standard deviation.

[0100] According to an embodiment of the present application, the target loss function is determined by the following formula:

[0101]

[0102] where J(θ) is the target loss function, θ is the parameter of the current policy model, θ old represents the parameter of the previous policy model, q is the problem data drawn from the question pool, p(Q) is the probability distribution of all possible questions in the question pool, o i is the i-th answer generated using the policy model for the problem data q, N is the number of answers generated for each problem data q, is the N answers generated using the policy model, π θ (o i |q) is the first probability value of determining the answer o i by the current policy model under the problem data q, is the first probability value of determining the answer o i by the previous policy model under the problem data q, GAE i is the relative advantage estimate of the i-th answer, D KL (π θ ||π ref ) is the KL divergence between the current policy model π θ and the reference model π ref , ε and a are hyperparameters, and E is the expected value operator, representing the average over all combinations of problem data and answers.

[0103] According to an embodiment of the present application, determining the first probability value of a single word in each answer based on multiple answers includes: determining a vector of each single word in the multiple answers based on the policy model in the language model; determining a probability distribution based on the vector and a preset function, where the preset function is the softmax function; taking the logarithm of the probability distribution to obtain the first probability value.

[0104] According to an embodiment of the present application, determining the KL divergence based on the first probability value and the second probability value includes: determining the KL divergence based on the difference between the first probability value and the second probability value.

[0105] It should be noted that for the details not disclosed in the optimization device of the improved proximal policy based on the language model in the embodiments of the present application, please refer to the details disclosed in the optimization method of the improved proximal policy based on the language model in the embodiments of the present application, and will not be elaborated here specifically.

[0106] In the optimization device of the improved proximal policy based on the language model according to the embodiment of the present application, the first determination module is used to determine multiple answers corresponding to the question data, and determine the first probability value of a single word in each answer based on the multiple answers. The second determination module is used to determine the second probability value of a single word in each answer based on the reference model in the language model, and determine the KL divergence based on the first probability value and the second probability value. The third determination module is used to determine the immediate reward of each answer based on the reward model and the target reward signal in the language model. The update module is used to determine the target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function. Thus, the device can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, improve concurrency, and accelerate the generation and training process of answers.

[0107] Note that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0108] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0109] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0111] In this application, unless otherwise clearly defined and limited, terms such as "installed", "connected", "joined", "fixed", etc. shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements or the interaction relationship between two elements, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0112] Although the embodiments of this application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting this application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An optimization method for improving proximal strategy based on language model, characterized in that: The method comprises: Determine a plurality of answers corresponding to the question data, and determine a first probability value of a single word in each answer based on the plurality of answers; Determine a second probability value of a single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value; Determine an immediate reward for each answer based on the reward model in the language model and the target reward signal; A target loss function is determined based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model based on the target loss function.

2. The optimization method for improving the proximal strategy based on a language model according to claim 1, characterized in that: The determining of the immediate reward based on the reward model in the language model and the target reward signal comprises: Determining a first reward for a single word in each answer based on the reward model; determining a second reward for each answer based on the target reward signal; The instant reward is determined based on the sum of a product of the first reward and a first preset coefficient and a product of the second reward and a second preset coefficient.

3. The optimization method for improving the proximal strategy based on a language model according to claim 2, characterized in that: The target reward signal includes a reward signal of a first domain and a reward signal of a second domain, wherein the first domain is a mathematics domain and the second domain is a code domain, and determining the second reward for each answer based on the target reward signal includes: In the case where the question data belongs to the first field, if the information extracted from the answer includes the correct answer and the calculation process, the second reward is determined to be the first preset threshold; if the information extracted from the answer includes the correct answer but does not include the calculation process, the second reward is determined to be the second preset threshold; if the information extracted from the answer does not include the correct answer and the calculation process, the second reward is determined to be zero, wherein the first preset threshold is greater than the second preset threshold; In the case that the question data belongs to the second field, if the code extracted from the answer can be compiled and executed using a local compiler, the second reward is determined to be the first preset threshold; if the code extracted from the answer fails to compile or execute, the second reward is determined to be zero.

4. The optimization method for improving the proximal strategy based on the language model according to claim 1, characterized in that: The determining of a target loss function based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence includes: Determine a target reward based on the sum of the immediate reward and the KL divergence; determining a relative advantage estimate based on the target reward; A target loss function is determined based on the relative advantage estimate, the KL divergence, the question data, the answer, and the first probability value.

5. The optimization method for improving the proximal strategy based on the language model according to claim 4 is characterized in that: Determining a relative advantage estimate based on the target reward includes: Determine the mean and standard deviation based on the target reward for all responses; The relative advantage estimate corresponding to each answer is determined based on the ratio of the difference between the target reward of each answer and the mean value to the standard deviation.

6. The optimization method for improving the proximal strategy based on the language model according to claim 4 is characterized in that: The objective loss function is determined by the following formula: Where J(θ) is the target loss function, θ is the parameter of the current strategy model, and θ old represents the last strategy model parameter, q is the problem data extracted from the problem pool, p(Q) is the probability distribution of all possible problems in the problem pool, and o i is the i-th answer generated by the strategy model for question data q, N is the number of answers generated for each question data q, is the N answers generated using the policy model, π θ (o i |q) is the answer o determined by the current strategy model under question data q i The first probability value of Under question data q, the answer o is determined by the last strategy model i The first probability value, GAE i is the relative advantage estimate of the ith answer, D KL (π θ ||π ref ) is the current policy model π θ With the reference model π ref The KL divergence between them, ε and a are hyperparameters, and E is the expected value operator, which means that all question data and answer combinations are averaged.

7. The optimization method for improving the proximal strategy based on the language model according to claim 1, characterized in that: The determining, based on the plurality of answers, a first probability value of a single word in each answer comprises: Determine a vector for each single word in the plurality of answers based on a policy model in the language model; Determining a probability distribution based on the vector and a preset function, wherein the preset function is a softmax function; Taking the logarithm of the probability distribution obtains the first probability value.

8. The optimization method for improving the proximal strategy based on a language model according to claim 1, characterized in that: The determining the KL divergence based on the first probability value and the second probability value includes: The KL divergence is determined based on a difference between the first probability value and the second probability value.

9. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the optimization method for improving the proximal strategy based on a language model according to any one of claims 1 to 8 is implemented.

10. An optimization device for improving proximal strategy based on a language model, characterized in that: The device comprises: A first determination module, configured to determine a plurality of answers corresponding to the question data, and determine a first probability value of a single word in each answer based on the plurality of answers; A second determination module is used to determine a second probability value of a single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value; A third determination module, configured to determine an immediate reward for each answer based on a reward model and a target reward signal in the language model; An updating module is used to determine a target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, so as to update the policy model of the language model based on the target loss function.

Images