Improved proximal policy optimization method and device based on language model, and electronic equipment

Abstract

The application discloses an improved proximal policy optimization method and device based on a language model, and an electronic device. The method comprises the following steps: determining a plurality of answers corresponding to problem data, and determining a first probability value of a single word in each answer based on the plurality of answers; determining a second probability value of the single word in each answer based on a reference model in the language model, and determining a KL divergence based on the first probability value and the second probability value; determining an immediate reward of each answer based on a reward model in the language model and a target reward signal; determining a target loss function based on the problem data, the answer, the first probability value, the immediate reward and the KL divergence, and updating a policy model of the language model based on the target loss function. The method of the application can alleviate the memory occupation problem in the proximal policy training process, reduce the memory usage, improve the concurrency, and speed up the generation and training process of the answer.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of language models, in particular to an improved proximal policy optimization method based on a language model, an electronic device and an improved proximal policy optimization device based on a language model. BACKGROUND

[0002] Language models (such as large language models (LLM)) are a hot topic in today's artificial intelligence technology. Current large language models refer to models with more than 100 billion or even 1,000 billion parameters, mainly transformer structure (a kind of deep neural network structure) large models. The pre-training and post-training process of LLM is the premise of large model deployment and use. Pre-training refers to learning LLM in a large-scale, diversified five-annotated text corpus of tens of billions of tokens (computer understanding of a single word of information, which can be understood as a standard word or Chinese word). By using the loss function (cross-entropy loss) of the language model, the probability of generating a text sequence is optimized and improved, and the goal is to learn the grammar and semantic structure of the language. The important feature of this stage is the huge amount of calculation, which requires the formation of a training cluster to complete the training process of LLM and a large amount of data.

[0003] After pre-training, LLM has basic understanding ability of language, and post-training is carried out after pre-training LLM is completed. The post-training stage pays more attention to the performance of the model on specific tasks and the alignment of human preferences. The post-training process is crucial in the development of language models. Although pre-training gives the model strong language understanding ability, these abilities are often general and lack fine-tuning for specific tasks and human preferences. Through supervised fine-tuning and reinforcement learning, post-training enables the model to better adapt to the needs of specific tasks and improve its performance in practical applications. Reinforcement learning plays a key role in aligning human preferences and acquiring logical reasoning ability. Although there are simple schemes such as DPO (Direct Preference Optimization) to supplement, in terms of generalization and universality, reinforcement learning is still indispensable. Reinforcement learning guides the model to make decisions that are more consistent with human logic in complex tasks through a reward mechanism, thereby improving the practicality and reliability of the model.

[0004] PPO (Proximal Policy Optimization Algorithms) is the mainstream method used in the post-training reinforcement learning stage of LLM. PPO is an efficient reinforcement learning algorithm used to train the policy of an agent. In the LLM model, the role of the PPO algorithm is to continuously optimize the behavior policy of the agent, enabling it to make optimal decisions when facing different language tasks. The core idea of the PPO algorithm is to introduce a restriction term when optimizing the policy to ensure that the update of the policy is not too aggressive, thereby ensuring the stability of the training process. Through continuous iteration and optimization, the PPO algorithm can help the agent gradually improve its ability to handle language tasks. The use process of PPO in LLM fine-tuning is as follows: (1) sample a question from the question pool and give it to LLM for generation, the given question and the generated answer represent the environment state; (2) calculate the value of the current state according to the question + answer, and give an immediate reward; in addition, it is also necessary to judge the impact of this action on subsequent policy generation and give a future return; (3) the generated answer, question, and reward, value, etc. Data as experience data, using gradient descent and other schemes to update the policy; (4) after the current policy is updated, a new round of experience data generation and policy update is performed.

[0005] The above method has the problem of memory occupation in the PPO training process, including two models for forward inference, two models for forward and backward calculation, and the reward signal obtained is a probability type, which is not accurate enough in some scenarios. SUMMARY

[0006] The present application aims to at least partially solve one of the technical problems in the related art. To this end, the first object of the present application is to propose an improved proximal policy optimization method based on a language model, determine a plurality of answers corresponding to problem data, and determine a first probability value of a single word in each answer based on the plurality of answers, determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value, determine an immediate reward for each answer based on a reward model in the language model and a target reward signal, and determine a target loss function based on the problem data, the answer, the first probability value, the immediate reward, and the KL divergence, to update the policy model of the language model based on the target loss function. Thus, the memory occupation problem in the proximal policy training process can be alleviated, the memory usage can be reduced, and the generation and training process of the answer can be accelerated.

[0007] The second object of the present application is to propose an electronic device.

[0008] The third object of the present application is to propose an improved proximal policy optimization device based on a language model.

[0009] To achieve the above object, the first aspect of the present application proposes an optimization method of improved proximal policy based on a language model, which comprises: determining a plurality of answers corresponding to problem data, and determining a first probability value of a single word in each answer based on the plurality of answers; determining a second probability value of the single word in each answer based on a reference model in the language model, and determining a KL divergence based on the first probability value and the second probability value; determining an immediate reward of each answer based on a reward model in the language model and a target reward signal; determining a target loss function based on the problem data, the answer, the first probability value, the immediate reward and the KL divergence, to update a policy model of the language model based on the target loss function.

[0010] According to the optimization method of improved proximal policy based on a language model, a plurality of answers corresponding to problem data are determined, and a first probability value of a single word in each answer is determined based on the plurality of answers. A second probability value of the single word in each answer is determined based on a reference model in the language model, and a KL divergence is determined based on the first probability value and the second probability value. An immediate reward of each answer is determined based on a reward model in the language model and a target reward signal. A target loss function is determined based on the problem data, the answer, the first probability value, the immediate reward and the KL divergence, to update a policy model of the language model based on the target loss function. Thus, the method can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, improve concurrency, and speed up the generation and training process of answers.

[0011] In addition, the optimization method of improved proximal policy based on a language model according to the above-mentioned embodiments of the present application can have the following additional technical features:

[0012] According to one embodiment of the present application, the determination of the immediate reward based on the reward model in the language model and the target reward signal comprises: determining a first reward of a single word in each answer based on the reward model; determining a second reward of each answer based on the target reward signal; and determining the immediate reward based on the sum of the product of the first reward and a first preset coefficient and the product of the second reward and a second preset coefficient.

[0013] According to one embodiment of the present application, the target reward signal includes a reward signal of a first field and a reward signal of a second field, wherein the first field is a mathematical field, and the second field is a code field, and the determining of the second reward of each answer based on the target reward signal includes: in the case that the question data belongs to the first field, if the information extracted from the answer includes a correct answer and a calculation process, determining that the second reward is a first preset threshold, if the information extracted from the answer includes a correct answer but does not include a calculation process, determining that the second reward is a second preset threshold, and if the information extracted from the answer does not include a correct answer and a calculation process, determining that the second reward is zero, wherein the first preset threshold is greater than the second preset threshold; in the case that the question data belongs to the second field, if the code extracted from the answer can be compiled and executed by a local compiler, determining that the second reward is the first preset threshold, and if the code extracted from the answer fails to be compiled or fails to be executed, determining that the second reward is zero.

[0014] According to one embodiment of the present application, the determining of the target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence includes: determining a target reward based on the sum of the immediate reward and the KL divergence; determining a relative advantage estimate based on the target reward; and determining a target loss function based on the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value.

[0015] According to one embodiment of the present application, the determining of the relative advantage estimate based on the target reward includes: determining a mean value and a standard deviation based on the target rewards of all answers; and determining the relative advantage estimate corresponding to each answer based on the ratio of the difference between the target reward of the answer and the mean value to the standard deviation.

[0016] According to one embodiment of the present application, the target loss function is determined by the following formula:

[0017]

[0018] wherein J(θ) is the target loss function, θ is the parameter of the current policy model, θ old represents the policy model parameter of the last time, q is the question data extracted from the question pool, p(Q) is the probability distribution of all possible questions in the question pool, o i is the i-th answer generated by using the policy model for the question data q, N is the number of answers generated for each question data q, is the N answers generated by using the policy model, π θ (o i|q) is a first probability value of the answer o determined by the current policy model under the question data q i , |q) is a first probability value of the answer o determined by the last policy model under the question data q i , i |q) is a relative advantage estimate of the i-th answer, D KL (π θ ||π ref ) is a KL divergence between the current policy model π θ and the reference model π ref , ε, a are hyperparameters, E is an expectation operator, and represents averaging over all question data and answer combinations.

[0019] According to an embodiment of the present application, the method further comprises: determining a vector of each single word in the plurality of answers based on a policy model in the language model; determining a probability distribution based on the vector and a preset function, wherein the preset function is a softmax function; and taking a logarithm of the probability distribution to obtain the first probability value.

[0020] According to an embodiment of the present application, the method further comprises: determining the KL divergence based on a difference between the first probability value and the second probability value.

[0021] To achieve the above object, a second aspect of the present application provides an electronic device, which comprises a memory, a processor, and a program stored in the memory and capable of running on the processor, and the processor executes the program to implement the above-mentioned optimization method for improved on-policy based on a language model.

[0022] According to the electronic device provided by the embodiments of the present application, by executing the above-mentioned optimization method for improved on-policy based on a language model, the memory occupation problem in the on-policy training process can be alleviated, the memory usage can be reduced, the concurrency can be improved, and the generation and training process of the answer can be accelerated.

[0023] To achieve the above object, the third aspect of the present application proposes an improved proximal policy optimization device based on a language model, which comprises: a first determination module configured to determine a plurality of answers corresponding to problem data, and determine a first probability value of a single word in each answer based on the plurality of answers; a second determination module configured to determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value; a third determination module configured to determine an immediate reward of each answer based on a reward model in the language model and a target reward signal; and an update module configured to determine a target loss function based on the problem data, the answer, the first probability value, the immediate reward and the KL divergence, so as to update a policy model of the language model based on the target loss function.

[0024] According to the improved proximal policy optimization device based on a language model, the first determination module is configured to determine a plurality of answers corresponding to problem data, and determine a first probability value of a single word in each answer based on the plurality of answers, the second determination module is configured to determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value, the third determination module is configured to determine an immediate reward of each answer based on a reward model in the language model and a target reward signal, and the update module is configured to determine a target loss function based on the problem data, the answer, the first probability value, the immediate reward and the KL divergence, so as to update a policy model of the language model based on the target loss function. Thus, the device can relieve the memory occupation problem in the proximal policy training process, reduce the memory usage, improve the concurrency, and speed up the generation and training process of the answer.

[0025] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of the improved proximal policy optimization method based on a language model according to an embodiment of the present application;

[0027] Figure 2 A flowchart of the improved proximal policy optimization method based on a language model according to one specific example of the present application;

[0028] Figure 3 A block schematic diagram of an electronic device according to an embodiment of the present application;

[0029] Figure 4 A block schematic diagram of the improved proximal policy optimization device based on a language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described below in detail with reference to examples illustrated in the accompanying drawings, in which like or similar elements or components throughout the drawings are denoted by the same or similar reference numerals, and the embodiments described below are examples intended to explain the present application and should not be understood as limiting the present application.

[0031] The PPO in the related art currently needs a total of 4 models to participate in the LLM post-training implementation, namely, an actor model (a policy model), a reference model, a reward model, and a critic model (a state value model). The overall implementation process is divided into two steps, one of which is experience data generation, and the other of which is adjusting model parameters using experience data. The specific steps are as follows: (1) sampling a batch of question data (prompt) in the question pool, and using the actor model to generate an answer (response); (2) generating experience data from prompt+response data: (a) the actor model calculates log probs (the logarithm of the probability of each word or token in each answer (response) generated by the large prediction model) for prompt+response; (b) the init model calculates log probs for prompt+response; (c) the reward model calculates the immediate reward; (d) the critic model calculates the state value Value; (e) the KL (Kullback-Leibler Divergence) divergence (relative entropy) is calculated using the log probs of a and b; (3) calculating the reward and GAE (Generalized Advantage Estimation, generalized advantage estimation), the reward calculation is the immediate reward+KL divergence; using the newly obtained reward, and value, logprobs to calculate GAE; (4) using the experience data of steps (2) and (3), updating the actor model (PPO loss function), and updating the critic model (value loss function). The memory occupation problem in the PPO training process of this method includes two models for forward reasoning, two models for forward and reverse calculation, and the reward signal given by the reward model is a probability type, which is not accurate enough in some scenarios. In addition, experience data generation and training are completely asynchronous, and policy update in PPO must be completed after data generation.

[0032] To this end, the application proposes an improved proximal policy optimization method based on a language model to alleviate the memory occupation problem in the PPO training process, reduce the device memory usage, and improve the scheme of using only the reward model to instantly reward the current state action, design a verifiable and accurate reward signal, and improve the completely asynchronous process of experience data generation and PPO policy update, use the delayed model update scheme to improve the device computing efficiency.

[0033] The improved proximal policy optimization method based on a language model, the electronic device, and the improved proximal policy optimization device based on a language model proposed by the embodiments of the application are described below with reference to the accompanying drawings.

[0034] Figure 1 A flowchart of the improved proximal policy optimization method based on a language model according to the embodiments of the application is shown.

[0035] As Figure 1 shown, the improved proximal policy optimization method based on a language model according to the embodiments of the application can include the following steps:

[0036] S1, determining a plurality of answers corresponding to the problem data, and determining a first probability value of a single word in each answer based on the plurality of answers.

[0037] S2, determining a second probability value of the single word in each answer based on a reference model in the language model, and determining a KL divergence based on the first probability value and the second probability value.

[0038] S3, determining an instant reward of each answer based on a reward model and a target reward signal in the language model.

[0039] S4, determining a target loss function based on the problem data, the answer, the first probability value, the instant reward, and the KL divergence, to update a policy model of the language model based on the target loss function.

[0040] Specifically, first, a plurality of answers corresponding to the problem data are determined, and a first probability value of a single word in each answer is determined based on the plurality of answers. For example, a plurality of answers (response) for given problem data (prompt) can be generated using a policy model. And for each generated answer, the policy model outputs the predicted probability distribution of each word, then converts it to a probability value through a preset function, and takes the logarithm of these probability values to obtain the first probability value of each word. By generating multiple answers, the model can explore different answer possibilities and increase the diversity of answers. The first probability value reflects the model's prediction of each word, which is used for subsequent loss function calculation.

[0041] The second probability value of each single word in each answer is determined according to a reference model in the language model, and the KL divergence is determined based on the first probability value and the second probability value. That is, the reference model, which is a model with the same structure as the policy model but with fixed parameters, is used to generate the second probability value of each single word in each answer. After determining the first probability value and the second probability value, the KL divergence can be determined according to the first probability value and the second probability value to measure the difference between the two probability distributions. That is, the reference model provides a stable comparison benchmark, which helps to evaluate whether the changes of the policy model are moving in the expected direction, and the KL divergence is used to control the magnitude of policy updates to prevent the policy from changing dramatically during the update process.

[0042] The immediate reward of each answer can be determined according to the reward model in the language model and the target reward signal. That is, the reward model can be used to evaluate the quality of each answer and give an immediate reward, and the target reward signal, such as correctness, relevance, etc., can be combined to further adjust the immediate reward. The immediate reward provides direct feedback to the model about the quality of the answer, guiding the model to optimize the generation of the answer, and by combining the target reward signal, the model can better understand and meet the needs of specific tasks.

[0043] After determining the first probability value, the KL divergence, and the immediate reward, the target loss function can be determined according to the question data, the answer, the first probability value, the immediate reward, and the KL divergence, so as to update the policy model of the language model according to the target loss function. That is, the target loss function can be constructed by combining the question data, the answer, the first probability value, the immediate reward, and the KL divergence. For example, a loss function is designed that not only considers the policy gradient but also considers the quality of the answer and the stability of the policy. For example, L(θ) = α*CrossEntropyLoss + β*RewardLoss + γ*KLLoss + δ*EntropyLoss, where α, β, γ, δ are the corresponding coefficients, CrossEntropyLoss is the cross-entropy loss based on the first probability value, encouraging the model to generate more accurate answers, RewardLoss is the loss based on the immediate reward, encouraging the model to generate higher quality answers, KLLoss is the loss based on the KL divergence, limiting the magnitude of policy updates, and EntropyLoss is the entropy loss, encouraging the model to explore more possible answers. The weight coefficients of each part of the above loss function are adjusted through cross-validation or grid search, etc., to find the optimal loss function configuration.

[0044] After determining the target loss function, an optimization algorithm such as gradient descent can be used to minimize the target loss function to update the parameters of the policy model. That is, the target loss function takes into account the predicted probability of the policy model, reward feedback and policy stability, guiding the model to improve the quality and stability of the answer, and by minimizing the target loss function, the model can learn to generate answers that better meet the task requirements and human preferences. This method helps to improve the performance of the language model on specific tasks while maintaining the stability and efficiency of the training process.

[0045] In this way, by continuously generating answers, evaluating answer quality, calculating loss and updating model parameters, the language model can gradually improve its performance on specific tasks.

[0046] According to an embodiment of the present application, the instant reward is determined based on the reward model in the language model and the target reward signal, including: determining a first reward for each word in the answer based on the reward model; determining a second reward for each answer based on the target reward signal; determining the instant reward based on the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient. Wherein the first preset coefficient and the second preset coefficient can be determined according to actual conditions,

[0047] Specifically, when determining the instant reward based on the reward model in the language model and the target reward signal, the calculation method of the instant reward combines the reward model (Reward Model) and the target reward signal (Target Reward Signal), aiming to comprehensively consider the quality of the generated answer at different levels, so as to more accurately guide the training and optimization of the model. The first reward for each word in the answer can be determined according to the reward model, and the role of the reward model is to evaluate the generated answer (response) word by word (or token by token) to determine the quality of each word. The specific steps are as follows: input the question (prompt) and the generated answer (response) into the reward model, the reward model calculates a reward value for each word (token) in the answer, which represents the quality or relevance of the word in the current context. These reward values constitute the first reward (reward1), and the reward model outputs a reward vector with the same length as the answer, each element corresponding to the reward value of a word.

[0048] The second reward of each answer is determined according to the target reward signal, which evaluates the quality of the entire answer rather than word by word. The target reward signal can be based on the specific task, for example: for mathematical problems, the target reward signal can be whether the answer is correct. For code generation tasks, the target reward signal can be whether the code passes the test case. For natural language tasks, the target reward signal can be the fluency, relevance or safety of the answer. The specific steps are as follows: the prompt and the generated response can be input into the target reward signal evaluation module, and the reward value of the entire answer is calculated according to the pre-defined rules or model, which represents the overall quality of the answer. This reward value constitutes the second reward (reward2), and the target reward signal evaluation module outputs a scalar value representing the reward of the entire answer.

[0049] After determining the first reward and the second reward, the instant reward can be determined according to the sum of the product of the first reward and the first preset coefficient and the product of the second reward and the second preset coefficient. That is, the instant reward is the result of considering the first reward and the second reward comprehensively, and the contribution of the two is balanced by introducing the preset coefficient. The specific steps are as follows: weighting of the first reward: multiplying the first reward (reward1) by the first preset coefficient a. Since the first reward is a vector, the weighted vector is still maintained. Weighting of the second reward: multiplying the second reward (reward2) by the second preset coefficient b. Since the second reward is a scalar, the weighted scalar is still a scalar. Calculation of the instant reward: adding the weighted first reward and the second reward to obtain the instant reward (reward). That is, the instant reward is determined by the formula instant reward = first reward * first preset coefficient + second reward * second preset coefficient.

[0050] For example, for mathematical problems, the first reward: the reward model calculates the reward value of each word (such as numbers, operators) to evaluate its rationality in mathematical logic. The second reward: the target reward signal evaluates whether the entire answer is correct (for example, whether the correct answer is given). The instant reward: comprehensively considers the reward of each word and the correctness of the entire answer, guiding the model to generate more accurate mathematical answers. For code generation, the first reward: the reward model calculates the reward value of each word (such as keywords, variable names) in the code to evaluate its syntax correctness. The second reward: the target reward signal evaluates whether the entire code passes the test case. The instant reward: comprehensively considers the reward of each word and the correctness of the entire code, guiding the model to generate more reliable code. For natural language tasks, the first reward: the reward model calculates the reward value of each word to evaluate its relevance and fluency in the context. The second reward: the target reward signal evaluates the safety, relevance or user satisfaction of the entire answer. The instant reward: comprehensively considers the reward of each word and the quality of the entire answer, guiding the model to generate answers that meet human preferences.

[0051] Thus, by introducing the target reward signal, the reward signal can be customized according to the needs of different tasks, so that the model can better adapt to specific tasks. And combined with word-by-word rewards and overall rewards, the quality of generated answers can be more comprehensively evaluated, in addition, it can be applied to various fields (such as mathematics, code, natural language, etc.), and the definition of the reward signal can be adjusted according to the specific task. Through the comprehensive reward mechanism, the performance of the language model on specific tasks can be more effectively guided, while avoiding the limitations that may be brought by simply relying on a single reward signal

[0052] According to one embodiment of the present application, the target reward signal includes a reward signal of a first field and a reward signal of a second field, wherein the first field is a mathematical field and the second field is a code field. Determining the second reward of each answer based on the target reward signal includes: in the case that the question data belongs to the first field, if the information extracted from the answer includes the correct answer and the calculation process, determining the second reward as a first preset threshold, if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as a second preset threshold, and if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, wherein the first preset threshold is greater than the second preset threshold; in the case that the question data belongs to the second field, if the code extracted from the answer can be compiled and executed by a local compiler, determining the second reward as the first preset threshold, and if the code extracted from the answer fails to compile or execute, determining the second reward as zero.

[0053] Specifically, the target reward signal is designed to determine the second reward of each answer according to the field to which the question data belongs (the first field is the mathematical field and the second field is the code field), aiming to provide targeted reward signals for the generated answers according to the characteristics and needs of different fields, thereby more effectively guiding the training and optimization of the model. That is, the purpose of the target reward signal is to evaluate the generated answer as a whole and determine whether it meets the quality standards of a specific field. Specifically: in the mathematical field, the target reward signal needs to evaluate whether the answer contains the correct answer and the calculation process. In the code field, the target reward signal needs to evaluate whether the code in the answer can be correctly compiled and executed. In this way, the model can learn the ability to generate high-quality answers in different fields.

[0054] If the problem data belongs to the first domain (mathematics domain), if the answer contains the correct answer and the calculation process, the second reward is set to the first preset threshold, for example, the first preset threshold is 1, that is, in this case, the answer not only gives the correct answer, but also provides the complete calculation process, which meets the high-quality standard of mathematical problems. If the answer contains the correct answer but does not contain the calculation process, the second reward is set to the second preset threshold, for example, the second preset threshold is 0.3, that is, in this case, although the answer gives the correct answer, it lacks the calculation process, so the second preset threshold is lower than the first preset threshold. If the answer does not contain the correct answer or the calculation process, the second reward is set to zero, that is, in this case, the answer does not meet the basic requirements of mathematical problems, so no reward is given. For example, the first preset threshold is 1.0, and the second preset threshold is 0.5. The answer to a question is 42. For the mathematical problem: if the answer is "the answer is 42, the calculation process is as follows:...", the second reward is 1.0. If the answer is "the answer is 42", the second reward is 0.5, and if the answer is "the answer is 43", the second reward is 0.

[0055] If the problem data belongs to the second domain (code domain), if the code in the answer can be compiled and passed by execution using the local compiler, the second reward is set to the first preset threshold, for example, the first preset threshold is 1, that is, in this case, the code not only can be compiled correctly, but also can pass the execution test, which meets the high-quality standard of code generation. If the code in the answer fails to compile or execute, the second reward is set to zero, that is, in this case, the code cannot meet the basic requirements of compilability and executability, so no reward is given. For example, the first preset threshold (threshold_1) = 1.0, for the code problem: if the answer is "python\ndef add(a, b):\n return a+b\n", and the code can be compiled and executed, the second reward is 1.0, if the answer is "python\ndef add(a, b):\n return a-b\n", and the code fails to compile or execute, the second reward is 0.

[0056] Therefore, through this design, the target reward signal can provide targeted rewards for the generated answers according to the characteristics and needs of different domains: in the mathematics domain, the reward signal not only focuses on the correctness of the answer, but also focuses on the integrity of the calculation process, in the code domain, the reward signal focuses on the compilability and executability of the code. This domain-specific reward mechanism can more accurately guide the model to generate high-quality answers, while avoiding the limitations that a single reward signal may bring.

[0057] According to one embodiment of the present application, the target loss function is determined based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, comprising: determining a target reward based on the sum of the immediate reward and the KL divergence; determining a relative advantage estimate based on the target reward; determining the target loss function based on the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value.

[0058] Specifically, the determination of the target loss function is a key step to optimize the policy network, and this process considers the immediate reward, the KL divergence, the question data, the answer and the first probability value. When determining the target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, first, the target reward is determined based on the sum of the immediate reward and the KL divergence. The KL divergence is used to measure the magnitude of policy update, to ensure that the update is not too aggressive. That is, the reward value calculated based on the question data and the answer, and the reward model and the target reward signal. The KL divergence between the new policy (updated policy) and the old policy (policy before update) is calculated, which represents the magnitude of policy update, i.e. target reward = immediate reward + KL divergence.

[0059] The relative advantage estimate can be determined according to the target reward. The relative advantage estimate is the difference between the target reward and the average reward, which is used to measure the advantage of the current policy over the baseline policy. For example, the target reward is calculated based on the sum of the immediate reward and the KL divergence, and the average reward is the average reward value calculated based on historical data, which is used as a baseline. The relative advantage estimate can be determined based on the difference between the target reward and the average reward. Finally, the target loss function can be determined based on the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value. That is, the target loss function is the final goal of optimizing the policy network, which considers the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value. For example, in the question data and answer pool, different factors (such as the relative advantage estimate, the KL divergence, the first probability value, etc.) affecting the performance of the policy can be explored through visualization techniques. For example, scatter plots, heat maps, etc. can be used to show the relationship between the relative advantage estimate and the KL divergence of different answers, so as to intuitively understand which factors have the greatest impact on the performance of the policy.

[0060] Therefore, by the above steps, the target loss function can be determined for optimizing the policy network. This process considers the immediate reward, the KL divergence, the question data, the answer and the first probability value, aiming to maximize the performance of the policy network while ensuring the stability of policy update. This target loss function design based on the relative advantage estimate and the KL divergence can more effectively guide the training of the policy network, while avoiding the problem of unstable training caused by too aggressive policy update. In this way, the model can achieve better performance when processing language model tasks.

[0061] Further, according to an embodiment of the present application, determining the relative advantage estimate based on the target reward comprises: determining a mean value and a standard deviation based on the target rewards of all answers; determining the relative advantage estimate corresponding to each answer based on a ratio of a difference between the target reward of the answer and the mean value to the standard deviation.

[0062] Specifically, when determining the relative advantage estimate based on the target reward, a mean value and a standard deviation can be determined based on the target rewards of all answers, the mean value of the target rewards of all answers representing a total quality level of the answers, and the standard deviation of the target rewards of all answers representing a dispersion degree of the answer quality. For example, the mean value and the standard deviation can be determined by corresponding calculation formulas for calculating the mean value and the standard deviation. After determining the mean value and the standard deviation, the relative advantage estimate corresponding to each answer can be determined based on a ratio of a difference between the target reward of the answer and the mean value to the standard deviation. For example, the relative advantage estimate is determined by the formula wherein rewardi is the reward of the ith answer, mean((reward1, reward2,..., rewardN)) is the mean value of the rewards of all answers, and std((reward1, reward2,..., rewardN)) is the standard deviation of the rewards of all answers. i N N That is, the relative advantage estimate is a difference between the target reward of each answer and the mean reward divided by the standard deviation, representing a degree of advantage or disadvantage of each answer relative to the total, and through such standardization processing, the relative advantage estimate can more fairly compare the quality of different answers and is not affected by the number and quality distribution of answers. Wherein N represents N different answers to the same question, that is, an average reward is constructed, and the current reward is subtracted from the average reward, which is recorded as the advantage of the current action state compared to the baseline. This method saves memory and calculation compared to using the critic model (state value model) to calculate the state value.

[0063] Therefore, through standardization processing, the relative advantage estimate can more fairly compare the quality of different answers, and the relative advantage estimate can adapt to different tasks and data distributions because it is based on the relative quality rather than the absolute quality of the answers. In addition, using the mean value and the standard deviation for standardization can reduce the influence of extreme values on the relative advantage estimate and improve the stability of training. Thus, it can be used to optimize the policy network (actor model), and the quality of the answers is improved by maximizing the relative advantage estimate. In the PPO algorithm, the relative advantage estimate can be used as the weight of the policy gradient to guide the update of the policy network.

[0064] Further, according to an embodiment of the present application, the target loss function is determined by the following formula: ​​

[0065]

[0066] wherein J(θ) is a target loss function, θ is a parameter of a current policy model, θ old represents a parameter of a last policy model, q is question data drawn from a question pool, p(Q) is a probability distribution of all possible questions in the question pool, o i is the i-th answer generated by using the policy model for the question data q, N is the number of answers generated for each question data q, is N answers generated by using the policy model, π θ (o i is a first probability value of the answer o i determined by the current policy model for the question data q, is a first probability value of the answer o i determined by the last policy model for the question data q, GAE i is a relative advantage estimation of the i-th answer, D KL (π θ ||π ref is a KL divergence between the current policy model π θ and a reference model π ref , and ε, a are hyperparameters, E is an expectation operator, and represents an average over all question data and answer combinations.

[0067] Specifically, the process of determining the target loss function is a core step in the reinforcement learning algorithm, which guides the optimization direction of the policy model. In the embodiments of the present application, the target loss function can be determined by the above formula (1), that is, by calculating the probability ratio of the answers oi generated by the current policy model π θ and the last policy model π θold , and applying a clipping operation to limit the policy ratio within the range of [1-∈, 1+∈] to prevent the policy update from being too aggressive. And calculate the relative advantage estimation GAEi of each answer, measure the advantage of the answer relative to the average level, and calculate the KL divergence between the current policy π θ model and the reference policy reference model π ref , measure the difference between the probability distributions of the two, and finally average all question data and answer combinations to obtain the expectation value.

[0068] Compared with the previous four models (policy model, reward model, reference model, and state value model), the above embodiments reduce the memory occupation of the three models (policy model, reward model, and reference model) participating in the common strategy model. Thus, by clipping the policy ratio and limiting the KL divergence, the stability of the training process is improved, and the policy update is prevented from being too aggressive. In addition, by introducing the hyperparameters ∈ and a, the amplitude of the policy update and the weight of the KL divergence can be flexibly adjusted to adapt to different task requirements. Advantage estimation can make the model explore better answers, and by optimizing the target loss function, the performance of the policy model in generating answers can be improved, making it more consistent with the task goal and human preference, and achieving stable, efficient and flexible policy optimization, thereby improving the performance of the language model on specific tasks.

[0069] According to one embodiment of the present application, determining a first probability value of a single word in each answer based on multiple answers includes: determining a vector of each single word in the multiple answers based on a policy model in a language model; determining a probability distribution based on the vector and a preset function, wherein the preset function is a softmax function; and taking the logarithm of the probability distribution to obtain the first probability value.

[0070] Specifically, when determining the first probability value of a single word in each answer based on multiple answers, a vector of each single word in the multiple answers can be determined according to a policy model in a language model. The policy model is a core part of the language model and can generate an answer (response) according to an input question (prompt). In the process of generating an answer, the policy model generates a vector for each word, representing the possibility of the word in the current context. That is, after inputting the question (prompt) into the policy model, the policy model can generate an answer word by word, generating a vector representation of each word at each step. These vectors are the model's predicted output for each word, commonly referred to as logits. For example, assuming the question is: "What's the weather like today?" The policy model generates the answer "Today the weather is very sunny." When generating each word (such as "today," "weather," "very," and "sunny"), the policy model outputs a vector representation (logits) of each word. Then, a probability distribution can be determined based on the vector and a preset function, wherein the preset function is a softmax function, and the first probability value is obtained by taking the logarithm of the probability distribution.

[0071] As in the formula q, a ~ π θ (.|q), q represents batch question data (such as 128), a represents batch answers corresponding to the batch question data (corresponding to 128*N), π θdenotes a policy model parameterized by θ, which represents an LLM model in the examples of the present application. The process of generating a is obtained by computing the probability of each individual word in the answer output by the LLM, which can be mapped back to the question-answer pair, and converted to first probability values logprobs by a softmax function and vector. For example, the vector output by the policy model for each word is input into the Soft max function to obtain the probability distribution of each word by applying the Softmax function. Assuming that the vector output by the policy model is: z = [2.0, 1.0, 0.1], then the output of the Softma function is: Softmax(z) = [e 2.0 / (e 2.0 +e 1.0 +e 0.1 ), e 1.0 / (e 2.0 +e 1.0 +e 0.1 )e 0.1 / (e 2.0 +e 1.0 +e 0.1 )] After calculation, Softmax(z) ≈ [0.659, 0.245, 0.096] is obtained.

[0072] The log probability value is the logarithmic form of the probability value, which is usually used in the optimization process because it can avoid the problem of numerical underflow and is more stable when calculating the gradient. When calculating the log probability value, the probability distribution output by the Softmax function can be taken as the logarithm to obtain the first probability value (log probs) of each word, that is, assuming that p is the probability distribution output by the Softmax function, then the first probability value is: logprobs = log(p). For example, the output of the Softmax function is p = [0.659, 0.245, 0.096], and the first probability value logprobs = log([0.659, 0.245, 0.096]) ≈ [-0.414, -1.404, -2.343] is obtained.

[0073] Thus, the first probability value of each word in the answer is determined based on the policy model in the language model, and these first probability values will be used in the subsequent policy optimization process, such as computing the policy gradient or the KL divergence, etc. In this way, the large prediction model can better optimize the quality of the generated answer.

[0074] According to an embodiment of the present application, the KL divergence is determined based on the first probability value and the second probability value, comprising: determining the KL divergence based on the difference between the first probability value and the second probability value.

[0075] Specifically, when determining the KL divergence according to the first probability value and the second probability value, the KL divergence can be determined according to the difference between the first probability value and the second probability value. The KL divergence is used to measure the difference between two probability distributions. Specifically, the first probability value and the second probability value are the output probability distributions of the policy model and the reference model at different time steps or under different policies, respectively. By calculating the KL divergence between the two probability distributions, the magnitude of policy update can be measured, thereby controlling the stability of policy update. It should be noted that the smaller the value of the KL divergence, the more similar the two distributions are. When the KL divergence is zero, it means that the two distributions are exactly the same. In reinforcement learning, the KL divergence is used to control the magnitude of policy update, for example, to prevent policy update from being too large: if the KL divergence is too large, it means that the policy update is too aggressive, which can lead to unstable training. By limiting the value of the KL divergence, the stability of policy update can be ensured. The learning rate can be dynamically adjusted, and in some algorithms, the learning rate can be dynamically adjusted according to the value of the KL divergence to achieve a more stable training process. Therefore, by limiting the value of the KL divergence, policy update can be prevented from being too large, thereby improving the stability of training.

[0076] In addition, in an embodiment of the application, an experience data generation and PPO policy update completely asynchronous mode is adopted, and a delayed model update scheme is used to improve the computing efficiency of the device. That is, in reinforcement learning, using the second-newest data to train and update the policy is a method to improve the computing efficiency and stability. This method is called asynchronous update or delayed update. That is, the training and update of the policy are always performed using the second-newest data, for example, a batch of data (data 1) is generated using the current policy (policy model 1), which includes prompts, responses, and related rewards and probability values (log probs), and then the generated data (data 1) is used to update the policy to obtain a new policy (policy model 2). Then, data is generated again, that is, a batch of data (data 2) is generated using the new policy (policy 2), which can be used for further policy update. The policy is updated using the second-newest data: even if new data (data 2) has been generated, the old data (data 1) is still used to update the policy 2 to obtain the policy 3. This method can reduce the delay of policy update and improve the continuity of training. Moreover, while generating new data (data 2), the policy is updated using old data (data 1), so that the computing resources for generation and training can be used simultaneously, improving the computing efficiency.

[0077] For example, assume we are training a dialogue system: interact with the user using policy 1 (the current dialogue policy), generate a batch of dialogue data (data 1) including the user's questions and the system's answers, evaluate the performance of policy 1 using data 1, calculate the reward and probability value, and then update the policy to get policy 2, use policy 2 to interact with the user again, generate a new batch of dialogue data (data 2), update policy 2 to get policy 3 using data 1, even though data 2 has been generated, still use data 1 to update policy 2 to get policy 3, so that the newly generated data can be used to improve the policy immediately without waiting for the processing of data 2.

[0078] Thus, the method of updating the latest policy using a sub-new data can reduce the waiting time and improve the overall training speed by using the generated and trained computing resources in parallel, so that the generated data process and the training process can be synchronized in one stage, which greatly improves the efficiency of device computing use. Although using sub-new data may cause some outdated, this method can reduce the delay of policy update, so that the policy can adapt to the changes in the environment more quickly, and through delayed update, the frequency of policy update can be reduced, thereby reducing the volatility of the policy in the training process and improving the stability of the policy. In addition, using sub-new data for updating can balance the relationship between exploration (generating new data) and utilization (using old data to update the policy) to some extent, so that the policy can both explore new possibilities and utilize existing knowledge. In a dynamically changing environment, using sub-new data for updating can help the policy adapt to changes in the environment more quickly and improve the adaptability of the policy.

[0079] The method of the present application will be described below in conjunction with Figure 2 .

[0080] As a specific example, the method for optimizing the improved near-end policy based on the language model of the present application can include the following steps:

[0081] S101, determine a plurality of answers corresponding to the question data, determine the vector of each single word in the plurality of answers based on the policy model in the language model, and determine the probability distribution based on the vector and the preset function, wherein the preset function is a softmax function, and the first probability value is obtained by taking the logarithm of the probability distribution.

[0082] S102, determine the second probability value of each single word in each answer based on the reference model in the language model.

[0083] S103, determine the KL divergence based on the difference between the first probability value and the second probability value.

[0084] S104, determining a first reward of each word in each answer based on a reward model in the language model, determining a second reward of each answer based on the target reward signal, and determining an immediate reward based on a sum of a product of the first reward and a first preset coefficient and a product of the second reward and a second preset coefficient.

[0085] S105, determining the target reward based on a sum of the immediate reward and the KL divergence.

[0086] S106, determining a mean value and a standard deviation based on the target rewards of all answers, and determining a relative advantage estimation corresponding to each answer based on a ratio of a difference between the target reward of the answer and the mean value and the standard deviation.

[0087] S107, determining a target loss function based on the relative advantage estimation, the KL divergence, the question data, the answer, and the first probability value.

[0088] S108, updating the policy model of the language model based on the target loss function.

[0089] In summary, according to the optimization method of the improved proximal policy based on the language model, a plurality of answers corresponding to the question data are determined, and a first probability value of each word in each answer is determined based on the plurality of answers. A second probability value of each word in each answer is determined based on a reference model in the language model, and a KL divergence is determined based on the first probability value and the second probability value. An immediate reward of each answer is determined based on a reward model in the language model and a target reward signal. A target loss function is determined based on the question data, the answer, the first probability value, the immediate reward, and the KL divergence. The policy model of the language model is updated based on the target loss function. Thus, the method can alleviate the memory occupation problem in the proximal policy training process, reduce memory usage, improve concurrency, and speed up the generation and training process of the answer.

[0090] Corresponding to the above embodiment, the present application also provides an electronic device.

[0091] As Figure 3 shown, the electronic device 200 of the embodiment of the present application can include a memory 210, a processor 220, and a program stored on the memory 210 and executable on the processor 220. When the processor 220 executes the program, the optimization method of the improved proximal policy based on the language model is implemented.

[0092] According to the electronic device of the embodiment of the present application, by executing the optimization method of the improved proximal policy based on the language model, the memory occupation problem in the proximal policy training process can be alleviated, the memory usage can be reduced, the concurrency can be improved, and the generation and training process of the answer can be accelerated.

[0093] Corresponding to the above-mentioned embodiments, the application further provides an optimization device of an improved proximal policy based on a language model.

[0094] As shown in Figure 4 the optimization device 100 of the improved proximal policy based on the language model according to the embodiments of the application comprises a first determination module 110, a second determination module 120, a third determination module 130 and an updating module 140.

[0095] The first determination module 110 is configured to determine a plurality of answers corresponding to the question data, and determine a first probability value of a single word in each answer based on the plurality of answers. The second determination module 120 is configured to determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value. The third determination module 130 is configured to determine an immediate reward of each answer based on a reward model in the language model and a target reward signal. The updating module 140 is configured to determine a target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, and update a policy model of the language model based on the target loss function.

[0096] According to one embodiment of the application, the immediate reward is determined based on the reward model in the language model and the target reward signal, comprising: determining a first reward of the single word in each answer based on the reward model; determining a second reward of each answer based on the target reward signal; and determining the immediate reward based on a sum of a product of the first reward and a first preset coefficient and a product of the second reward and a second preset coefficient.

[0097] According to one embodiment of the application, the target reward signal comprises a reward signal of a first field and a reward signal of a second field, wherein the first field is a mathematical field and the second field is a code field. The second reward of each answer is determined based on the target reward signal, comprising: in the case that the question data belongs to the first field, if the information extracted from the answer includes a correct answer and a calculation process, determining the second reward as a first preset threshold value, if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as a second preset threshold value, and if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, wherein the first preset threshold value is greater than the second preset threshold value; in the case that the question data belongs to the second field, if the code extracted from the answer can be compiled and executed by using a local compiler, determining the second reward as the first preset threshold value, and if the code extracted from the answer fails to be compiled or fails to be executed, determining the second reward as zero.

[0098] According to one embodiment of the present application, the target loss function is determined based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, comprising: determining a target reward based on a sum of the immediate reward and the KL divergence; determining a relative advantage estimate based on the target reward; determining the target loss function based on the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value.

[0099] According to one embodiment of the present application, the relative advantage estimate is determined based on the target reward, comprising: determining a mean value and a standard deviation based on the target reward of all answers; determining the relative advantage estimate corresponding to each answer based on a ratio of a difference between the target reward of the answer and the mean value to the standard deviation.

[0100] According to one embodiment of the present application, the target loss function is determined by the following formula:

[0101]

[0102] wherein J(θ) is the target loss function, θ is a parameter of the current policy model, θ old represents the policy model parameter of the last time, q is the question data extracted from the question pool, p(Q) is a probability distribution of all possible questions in the question pool, o i is the i-th answer generated by using the policy model for the question data q, N is the number of answers generated for each question data q, is the N answers generated by using the policy model, π θ (o i |q) is the first probability value of the answer o i determined by the current policy model under the question data q, is the first probability value of the answer o i determined by the last policy model under the question data q, GAE i is the relative advantage estimate of the i-th answer, D KL (π θ ||π ref ) is the KL divergence between the current policy model π θ and the reference model π ref , ε, a are hyperparameters, E is an expectation operator, and represents an average over all question data and answer combinations.

[0103] According to one embodiment of the present application, the first probability value of a single word in each answer is determined based on multiple answers, comprising: determining a vector of each single word in the multiple answers based on the policy model in the language model; determining a probability distribution based on the vector and a preset function, wherein the preset function is a softmax function; and taking a logarithm of the probability distribution to obtain the first probability value.

[0104] According to an embodiment of the present application, the KL divergence is determined based on the first probability value and the second probability value, including: determining the KL divergence based on a difference between the first probability value and the second probability value.

[0105] It should be noted that details of the optimization device for improved on-policy strategy based on a language model in the embodiments of the present application are not disclosed, please refer to the details disclosed in the optimization method for improved on-policy strategy based on a language model in the embodiments of the present application, which will not be described here.

[0106] According to the optimization device for improved on-policy strategy based on a language model in the embodiments of the present application, the first determining module is configured to determine a plurality of answers corresponding to the question data, and determine a first probability value of a single word in each answer based on the plurality of answers, the second determining module is configured to determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value, the third determining module is configured to determine an immediate reward of each answer based on a reward model in the language model and a target reward signal, and the updating module is configured to determine a target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, and update a policy model of the language model based on the target loss function. Thus, the device can alleviate the memory occupation problem in the on-policy training process, reduce the memory usage, improve the concurrency, and speed up the generation and training process of the answer.

[0107] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description, either functionally or chronologically, as well as changes being made concerning the order of implementation. The logic and / or steps represented in the flow diagrams and / or described herein can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus) or a propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0108] It is to be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following techniques, which are well known in the art of making integrated circuits, can be used alone or in any combination to implement the application: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0109] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. Descriptive expressions of the above terms in the present specification do not necessarily refer to the same embodiment or example. Also, the specific feature, structure, material or characteristic described can be combined in any appropriate manner in one or more embodiments or examples.

[0110] In addition, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implying a number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0111] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0112] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for improving the optimization of a near-optimal policy based on a language model, characterized in that, The method comprises: determining a plurality of answers corresponding to the question data, and determining a first probability value of a single vocabulary in each answer based on the plurality of answers; determining a second probability value of the single vocabulary in each answer based on a reference model in the language model, and determining a KL divergence based on the first probability value and the second probability value; determining an immediate reward of each answer based on a reward model in the language model and a target reward signal; determining a target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence, to update a policy model of the language model based on the target loss function; The determination of the immediate reward based on the reward model in the language model and the target reward signal comprises: determining a first reward of a single vocabulary in each answer based on the reward model; determining a second reward of each answer based on the target reward signal; determining the immediate reward based on the sum of the product of the first reward and a first preset coefficient and the product of the second reward and a second preset coefficient; The target reward signal comprises a reward signal of a first field and a reward signal of a second field, wherein the first field is a mathematical field, and the second field is a code field, and the determination of the second reward of each answer based on the target reward signal comprises: In the case that the question data belongs to the first field, if the information extracted from the answer includes a correct answer and a calculation process, the second reward is determined as a first preset threshold, if the information extracted from the answer includes a correct answer but does not include a calculation process, the second reward is determined as a second preset threshold, and if the information extracted from the answer does not include a correct answer and a calculation process, the second reward is determined as zero, wherein the first preset threshold is greater than the second preset threshold; In the case that the question data belongs to the second field, if the code extracted from the answer can be compiled and executed by a local compiler, the second reward is determined as the first preset threshold, and if the code extracted from the answer fails to be compiled or executed, the second reward is determined as zero.

2. The method of claim 1, wherein the language model is a probabilistic language model. The determination of the target loss function based on the question data, the answer, the first probability value, the immediate reward and the KL divergence comprises: determining a target reward based on the sum of the immediate reward and the KL divergence; determining a relative advantage estimate based on the target reward; determining a target loss function based on the relative advantage estimate, the KL divergence, the question data, the answer and the first probability value.

3. The method of claim 2, wherein the language model is a probabilistic language model. The determination of the relative advantage estimate based on the target reward comprises: determining a mean value and a standard deviation based on the target rewards of all answers; determining the relative advantage estimate corresponding to each answer based on the ratio of the difference between the target reward of the answer and the mean value to the standard deviation.

4. The method of claim 2, wherein the language model is a probabilistic language model. The target loss function is determined by the following formula: wherein, is the target loss function, is the parameter of the current policy model, denotes the policy model parameter of the last time, is the question data drawn from the question pool, is the probability distribution of all possible questions in the question pool, is the i-th answer generated using the policy model for question data q, and N is the number of answers generated for each question data q, is the N answers generated using the policy model, is the first probability value of determining the answer by the current policy model under question data q, is the first probability value of determining the answer by the last policy model under question data q, is the relative advantage estimate of the i-th answer, is the KL divergence between the current policy model and the reference model , , is a hyperparameter, and E is an expectation operator, indicating averaging over all question data and answer combinations.

5. The method of claim 1, wherein the language model is a probabilistic language model. The determination of the first probability value of a single vocabulary in each answer based on the plurality of answers comprises: determining a vector of each single vocabulary in the plurality of answers based on a policy model in the language model; determine a probability distribution based on the vector and a preset function, wherein the preset function is a softmax function; take a logarithm of the probability distribution to obtain the first probability value.

6. The method of claim 1, wherein, The method further includes: determining the KL divergence based on a difference between the first probability value and the second probability value.

7. An electronic device, comprising: The method further includes: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the optimization method of the improved proximal policy based on the language model according to any one of claims 1-6.

8. An apparatus for optimizing an improved on-policy based on a language model, the apparatus comprising: The apparatus includes: a first determining module configured to determine a plurality of answers corresponding to the question data, and determine a first probability value of a single word in each answer based on the plurality of answers; a second determining module configured to determine a second probability value of the single word in each answer based on a reference model in the language model, and determine a KL divergence based on the first probability value and the second probability value; a third determining module configured to determine an immediate reward of each answer based on a reward model in the language model and a target reward signal; an updating module configured to determine a target loss function based on the question data, the answers, the first probability values, the immediate rewards, and the KL divergences, and update a policy model of the language model based on the target loss function; wherein The third determining module is specifically configured to: determine a first reward of the single word in each answer based on the reward model; determine a second reward of each answer based on the target reward signal, wherein the target reward signal includes a reward signal of a first field and a reward signal of a second field, the first field is a mathematical field, the second field is a code field, and the determination of the second reward of each answer based on the target reward signal includes: in a case where the question data belongs to the first field, if information extracted from the answer includes a correct answer and a calculation process, determining the second reward as a first preset threshold, if the information extracted from the answer includes the correct answer but does not include the calculation process, determining the second reward as a second preset threshold, and if the information extracted from the answer does not include the correct answer and the calculation process, determining the second reward as zero, wherein the first preset threshold is greater than the second preset threshold; in a case where the question data belongs to the second field, if code extracted from the answer can be compiled and executed by a local compiler, determining the second reward as the first preset threshold, and if the code extracted from the answer fails to be compiled or executed, determining the second reward as zero; determine the immediate reward based on a sum of a product of the first reward and a first preset coefficient and a product of the second reward and a second preset coefficient.

Images