Value model enhancement-based model reinforcement learning method, system, equipment and medium

Abstract

The invention discloses a value model enhancement-based model reinforcement learning method, system, device and medium, and the method comprises the steps: generating training trajectory data containing states, actions and rewards through interaction of a current strategy model and an environment, and calculating an action probability ratio of new and old strategies; predicting a state value by using a value model based on the trajectory data, and obtaining a dominant function by combining dominant estimation; constructing a value model multi-objective optimization function containing mean square error loss, relative entropy regular term loss and semantic perception loss, and calculating a near-end strategy optimization objective function according to an action probability ratio and a dominant function; and finally, updating the value model by adopting an exponential moving average algorithm, and optimizing the current strategy model according to a near-end strategy optimization objective function until an iteration condition is met, thereby obtaining a strategy model after reinforcement learning. According to the method, the value model in the large language model is enhanced, so that the learning performance and robustness of the large language model in a complex supervision environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and natural language processing, and particularly relates to a model reinforcement learning method, system, device and medium based on value model enhancement. BACKGROUND

[0002] In recent years, large language models (LLM) have been widely applied to tasks such as dialogue generation, question answering systems, language understanding and mathematical reasoning, and have achieved excellent results on numerous benchmark tests. In order to further make the model behavior more in line with human preferences, researchers generally use reinforcement learning from human feedback (RLHF) to fine-tune and optimize pre-trained models.

[0003] A typical RLHF training process includes generating training trajectories through a policy model, then training a reward model using human-labeled data, and then using the reward model to evaluate the pros and cons of the text output by the policy model, and using the value model to evaluate the value of the training trajectory in the reinforcement learning process to assist in generating the advantage function, and then optimizing the policy model. However, in actual applications, the reward model itself often has problems such as label noise, scoring bias and context misjudgment, which are more serious in automatic generation or large-scale scoring scenarios. In the face of the above problems, existing technologies mainly remedy from the level of external signals, such as using reward voting mechanisms to alleviate reward bias, using KL divergence control terms to prevent policy collapse, or filtering samples with inconsistent scores, etc.

[0004] Therefore, in typical RLHF methods, the value model is usually considered as an intermediate estimator, only used to assist in calculating the advantage function, and its structure is often simplified as a Transformer linear head, without being customized and optimized for the structural characteristics and noise interference of language tasks, ignoring the key influence of the value model itself on the stability of the training process and the generalization ability of the policy. This easily leads to two core problems, one is the lack of robustness, in a noisy reward environment, the value model is easy to fit the wrong label, and cannot effectively filter abnormal supervision, thereby affecting the stability of the policy learning process; the second is that in long text generation tasks, the value model often mistakes structural redundancy or repeated endings as high-value segments, leading to phenomena such as "length hacking" by the model, and the output text is longer but the semantics is degraded. In addition, the value model usually does not have an information compression mechanism, which makes the model vulnerable to irrelevant information interference when facing long texts or high redundancy inputs, and it is difficult to focus on key tokens, thereby producing "false advantage" estimates, which seriously misleads the policy update direction. SUMMARY

[0005] The application aims to provide a model reinforcement learning method, system, device and medium based on value model enhancement, to improve the anti-interference ability to reward noise and the modeling ability to key semantics in large language model reinforcement learning, thereby improving the robustness of the large language model.

[0006] To achieve the above-mentioned purpose, the application provides a model reinforcement learning method based on value model enhancement, comprising: generating training trajectory data containing state, action and reward by interacting with the environment through the current policy model, calculating the action probability ratio of new and old policies based on the trained trajectory data; Based on the training trajectory data, the state value is predicted by the value model; based on the predicted state value, the advantage function and the target return are calculated by using advantage estimation; According to the training trajectory data, the predicted state value and the target return, a multi-objective optimization function of the value model is constructed, and a proximal policy optimization target function is calculated based on the action probability ratio and the advantage function; wherein the multi-objective optimization function includes mean square error loss, relative entropy regularization term loss and semantic perception loss; According to the multi-objective optimization function, the value model is updated by using exponential moving average algorithm; the current policy model is optimized according to the proximal policy optimization target function until the preset iteration condition is met, and the policy model after reinforcement learning is obtained.

[0007] Preferably, based on the training trajectory data, the state value is predicted by the value model, and the advantage function is calculated by using generalized advantage estimation based on the predicted state value, comprising: Based on the training trajectory data, the state is mapped to Gaussian latent variable by the encoder of the value model; Based on the Gaussian latent variable, the state value is predicted by the decoder of the value model; The state is mapped to Gaussian latent variable by the encoder of the value model: Based on the predicted state value, the advantage function and the target return are calculated by using nonlinear transformation of generalized advantage estimation.

[0008] Preferably, the state is mapped to Gaussian latent variable by the encoder of the value model according to the following formula: Wherein, z t is the Gaussian latent variable, which represents the compressed representation of the input x t ; is the mean function defined by the parameters φ of the encoder f φ , the input is x t , and the output is the mean vector of the latent variable probability distribution; The encoder fφ a standard deviation function defined by the parameter φ, inputting x t and outputting a standard deviation vector of the latent variable probability distribution; ∈ is a random number sampled from a standard normal distribution , representing reparameterized sampling noise, I is a d*d identity matrix.

[0009] Preferably, the multi-objective optimization function comprises: wherein, denotes the multi-objective optimization function; denotes the mean square error loss; denotes the relative entropy regularizer loss; β IB is the weight of the relative entropy regularizer loss; denotes the semantic awareness loss; T denotes the trajectory length; denotes the predicted state value at time t; R t denotes the target return at time t; ∈ is a random number sampled from a standard normal distribution , I is a d*d identity matrix; x t denotes the input text state at time t; z t denotes the Gaussian latent variable at time t; p φ (z t |x t ) denotes the Gaussian latent variable posterior distribution of the encoder f φ ; r(z) denotes the prior distribution of the Gaussian latent variable z t ; H RS (t) denotes the entropy loss; T ent denotes the high-entropy token set; λ ent denotes the weight coefficient of the entropy loss; L rNLL (t) denotes the perplexity loss; T ppl denotes the high-perplexity token set; λ ppl denotes the weight coefficient of the perplexity loss.

[0010] Preferably, the semantic awareness loss is achieved by the following steps: obtaining the predicted distribution of the token using a frozen language model head sharing embedding layers with the current policy model; based on the predicted distribution, filtering the high-entropy token set according to a preset entropy threshold; based on the predicted distribution, filtering the high-perplexity token set according to a preset perplexity threshold; weighting the high-entropy token set and the high-perplexity token set to obtain the semantic awareness loss.

[0011] Preferably, the entropy loss is calculated by the following formula: wherein y represents a token candidate value; R V (y|x t ) represents a prediction distribution of a token by a frozen language model head sharing an embedding layer with a current policy model; β H represents an entropy smoothing parameter; The perplexity loss is calculated by the following formula: wherein y t represents a token candidate value at time t; represents a target token given at time t; represents a negative log-likelihood value of the target token, and the higher the value, the greater the prediction difficulty of the value model on the token, i.e., the higher the perplexity: β P represents a perplexity smoothing parameter.

[0012] Preferably, the action probability ratio of the new and old policies is calculated by the following formula: wherein ρ t represents an action probability ratio of the new and old policies at time t; x t represents an input text state at time t; a t represents an action at time t; r t represents a reward at time t; π θ represents a current policy model; represents a policy model of the last iteration, represents a transformed value of ρ t , which is used to avoid the drastic influence of extreme values of ρ t on policy update.

[0013] To achieve the above purposes, the application further provides a model reinforcement learning system based on value model enhancement, comprising: a training trajectory data sampling module for generating training trajectory data containing states, actions and rewards by interacting with an environment through a current policy model, and calculating an action probability ratio of new and old policies based on the training trajectory data; an advantage function calculation module for predicting state values through a value model based on the training trajectory data; and calculating an advantage function and a target return by advantage estimation based on the predicted state values; The target function construction module is configured to construct a multi-objective optimization function of the value model according to the training trajectory data, the predicted state value and the target return, and calculate a proximal policy optimization objective function based on the action probability ratio and the advantage function; wherein the multi-objective optimization function comprises a mean square error loss, a relative entropy regularization term loss and a semantic perception loss; the model optimization module is configured to update the value model by using an exponential moving average algorithm according to the multi-objective optimization function; and the current policy model is optimized according to the proximal policy optimization objective function until a preset iteration condition is met, so as to obtain a policy model after reinforcement learning.

[0014] To achieve the above object, the application further provides a terminal device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the model reinforcement learning method based on value model enhancement according to any one of the above when executing the computer program.

[0015] To achieve the above object, the application further provides a computer readable storage medium comprising a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the model reinforcement learning method based on value model enhancement according to any one of the above when the computer program runs.

[0016] Compared with the prior art, the model reinforcement learning method, system, device and medium based on value model enhancement provided by the application firstly generate training trajectory data containing states, actions and rewards by interacting with an environment through a current policy model and calculate an action probability ratio of new and old policies; then predict a state value based on the trajectory data by using a value model, and obtain an advantage function by combining advantage estimation; subsequently construct a multi-objective optimization function of the value model comprising a mean square error loss, a relative entropy regularization term loss and a semantic perception loss, and calculate a proximal policy optimization objective function according to the action probability ratio and the advantage function; finally update the value model by using an exponential moving average algorithm, and optimize the current policy model according to the proximal policy optimization objective function until a preset iteration condition is met. The application reconstructs and enhances the value model in the large language model by introducing an information bottleneck structure and a semantic perception mechanism, which can improve the anti-interference ability to reward noise and the modeling ability to key semantics in the large language model reinforcement learning, thereby improving the learning performance and robustness of the large language model in a complex supervised environment. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings used in the embodiments will be briefly introduced as follows. Obviously, the drawings described below are only some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings also belong to the protection scope of the present application.

[0018] Figure 1 is a flowchart of a model reinforcement learning method based on value model enhancement provided by an embodiment of the present application; Figure 2 is another flowchart of a model reinforcement learning method based on value model enhancement provided by an embodiment of the present application; Figure 3 is a training performance comparison chart of different reinforcement learning methods in a dialogue task provided by an embodiment of the present application; Figure 4 is a training performance comparison chart of different reinforcement learning methods in different tasks in optimization when tested under a rule reward provided by an embodiment of the present application; Figure 5 is a training performance comparison chart of different reinforcement learning methods in a mathematical task in optimization when trained under a rule reward provided by an embodiment of the present application; Figure 6 is a response length change comparison chart of different reinforcement learning methods in a training process provided by an embodiment of the present application; Figure 7 is a structural block diagram of a model reinforcement learning system based on value model enhancement provided by an embodiment of the present application; Figure 8 is a structural block diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0020] Referring to Figure 1 , Figure 1 is a flowchart of a model reinforcement learning method based on value model enhancement provided by an embodiment of the present application, which comprises steps S1 to S4: S1, generating training trajectory data containing state, action and reward by interacting with an environment through a current policy model, and calculating action probability ratios of new and old policies based on the training trajectory data; Exemplarily, the current policy model π θ interacts with the environment to generate a state x t , an action a t and a reward r t . A set of training trajectories is generated.

[0021] Preferably, the action probability ratio ρ t of the new and old policy is calculated by the following formula: wherein ρ t represents the action probability ratio of the new and old policy at time t, which is used to measure the probability difference of the “new policy” and the “old policy” in selecting the action a t under the state x t at time t; x t represents the input text state at time t; a t represents the action at time t; r t represents the reward at time t; π θ represents the current policy model; represents the policy model of the last iteration, represents the transformed value of ρ t , which is used for policy gradient optimization (such as the “clipped target” design of the Proximal Policy Optimization (PPO) objective function) of reinforcement learning, to avoid the extreme value (such as close to 0 or infinity) of ρ t to have a dramatic impact on policy update.

[0022] It is worth noting that, compared with using only the original ratio ρ t , the embodiment of the present application can suppress the variance of policy update by adding a new mathematical transformation, which can solve the problem of policy collapse in a noisy environment, reduce the policy shock caused by noisy rewards, and thus enhance the stability of model training.

[0023] S2, predicting the state value based on the training trajectory data through a value model; and calculating the advantage function and the target return based on the predicted state value through advantage estimation; In an optional embodiment, the step S2 comprises steps S201 to S203: S201, mapping the state to a Gaussian latent variable through an encoder of the value model based on the training trajectory data; Preferably, the state x t is mapped to a Gaussian latent variable z t through the encoder f φ of the value model according to the following formula: where z t is a Gaussian latent variable representing a compressed representation of the input x t ; is a mean function defined by the parameters φ of the encoder f φ , input is x t , and output is a mean vector of the latent variable probability distribution; is a standard deviation function defined by the parameters φ of the encoder f φ , input is x t , and output is a standard deviation vector of the latent variable probability distribution; ∈ is a random number sampled from a standard normal distribution , representing reparameterization sampling noise, and I is a d*d identity matrix.

[0024] It is worth noting that the embodiments of the present application force the distribution of the latent variable to approach the prior distribution, which essentially compresses the information of the input text, thereby retaining only the core signals related to value prediction and filtering out redundant noise, achieving noise reduction of the information bottleneck.

[0025] S202, based on the Gaussian latent variable, predicting the state value through the decoder q ψ of the value model: where, represents the predicted state value at time t.

[0026] S203, based on the predicted state value , calculating the advantage function and the target return by using the generalized advantage estimation of the nonlinear transformation; Specifically, the advantage function A t and the target return R t at time t are calculated by the following formula: where δ t represents the time difference error at time t, δ t+1 represents the time difference error at time t+1, which is used to measure the difference between the "current state value prediction" and the "immediate reward + discounted next state value", and is the core basis of the advantage function; γ (0<γ<1) is a discount factor, which is used to control the importance of future rewards; s t represents the environment state at time t; represents the state value prediction at time t; represents the state value prediction at time t+1; A tThe generalized advantage estimation (GAE) method is used to compute the advantage function in reinforcement learning. GAE combines the ideas of temporal difference error and multi-step prediction, using an exponentially weighted average to calculate the advantage estimate, thereby improving the stability and efficiency of the policy gradient method; L represents the trajectory cutoff length; λ t+1 This represents the GAE decay coefficient at time t+1; The advantage at time t, By performing a smoothing operation, the dominance function is transformed to accommodate noise, which reduces the interference of extreme dominance values ​​on policy updates and improves training stability. α represents the smoothing parameter, and α>0. This dominance transformation avoids numerical instability in the 0 neighborhood, further enhancing the method's stability. R t Representing the target return, combining the advantage function with the current state value prediction, we obtain the result from s t The initial cumulative discount reward target.

[0027] S3. Construct a multi-objective optimization function for the value model based on the training trajectory data, the predicted state value, and the target reward; calculate the proximal policy optimization objective function based on the action probability ratio and the advantage function; wherein, the multi-objective optimization function includes mean squared error loss, relative entropy regularization term loss, and semantic awareness loss; In one optional embodiment, the multi-objective optimization function is: in, Represents a multi-objective optimization function; The mean squared error loss is represented by T; the trajectory length is represented by T. This represents the relative entropy regularization loss, also known as the KL divergence regularization loss; β IB This represents the IB strength baseline, i.e., the weighting parameter of the Information Bottleneck (IB); p φ (z t |x t ) represents encoder f φ The Gaussian latent variable posterior distribution (parameter φ, representing "input x") t Below, latent variable z t The conditional probability distribution of z); r(z) represents the Gaussian latent variable z. t The prior distribution is preferably the standard Gaussian N(0,I) distribution; H represents semantic perception loss; RS (t) represents the entropy loss; T entrepresents a high-entropy token set; λ ent represents a weight coefficient of entropy loss; L rNLL (t) represents a perplexity loss; T ppl represents a high-perplexity token set; λ ppl represents a weight coefficient of perplexity loss.

[0028] It is worth noting that, compared with the mean square error loss in the prior art, the square error loss form of the embodiment of the present application is approximately quadratic in small residual and grows slowly in large residual, and the calculation is more robust against noise; compared with the relative entropy regularization term loss in the prior art, the introduction of β IB The anti-noise structure can improve the generalization ability of the value model under noise.

[0029] In an optional embodiment, the weight β IB is adjusted by the following formula: Wherein, β0 is an initial coefficient.

[0030] It is worth noting that, by using the ratio of the variance of the residual predicted by the value model to the variance of the true return, the embodiment of the present application dynamically scales The essence is to automatically adapt the constraint strength of KL divergence to the relationship between the prediction ability of the value model and the noise level of the return, and solve the underfitting / overfitting problem caused by fixed weight.

[0031] In an optional embodiment, the semantic perception loss is implemented by the following steps A1 to A4: A1, using a frozen language model (LM) head sharing embedding layers with the current strategy model to obtain a prediction distribution of tokens; A2, based on the prediction distribution, filtering a high-entropy token set according to a preset entropy threshold; Exemplarily, the filtered high-entropy token set is: Wherein, is a dynamically set entropy threshold, and after filtering, 80% of the tokens are activated for training.

[0032] A3, based on the prediction distribution, filtering a high-perplexity token set according to a preset perplexity threshold; Exemplarily, the filtered high-perplexity token set is: Wherein, is a dynamically set entropy threshold, and after filtering, 80% of the tokens are activated for training.

[0033] A4, the semantic-aware loss is calculated by weighting the high-entropy word set and the high-ambiguity word set.

[0034] It is worth noting that, compared to the screening of the original entropy / ambiguity value, the present application can more accurately identify high-uncertainty word units by dynamically adjusting the threshold through square root transformation, thereby improving the identification accuracy of key semantic units and avoiding interference from redundant word units.

[0035] Further, in an optional embodiment, the entropy loss is calculated by the following formula: wherein y represents the word candidate value; P V (y|x t ) represents the prediction distribution of the word by the frozen language model head sharing the embedding layer with the current policy model; β H represents the entropy smoothing parameter; The ambiguity loss is calculated by the following formula: wherein y t represents the word candidate value at time t; represents the target word given by the data set or weak supervision rule at time t; represents the negative log-likelihood value of the target word, the higher the value, the more difficult the prediction of the value model for the word, i.e., the higher the ambiguity: β P represents the ambiguity smoothing parameter.

[0036] It is worth noting that, by using the embedding smoothing coefficient and the square root, the present embodiment can avoid the dominance of noise word units in loss calculation and can strengthen the attention of the value model to key semantics, compared to directly using entropy and ambiguity.

[0037] S4, according to the multi-objective optimization function, the value model is updated by using the exponential moving average algorithm; the current policy model is optimized according to the proximal policy optimization objective function until the preset iteration condition is met, and the policy model after reinforcement learning is obtained.

[0038] In an optional embodiment, the proximal policy optimization objective function is: wherein, represents the proximal policy optimization objective function, E t [·] represents the expectation at time t; ∈ rs represents the clipping coefficient, ∈ rs ∈(0,1); clip(x,a,b) represents the clipping function that truncates x to the interval [a,b], represents the expectation at time t; ∈ The clipping operation limits the probability ratio of the new policy to the old policy in the interval [1-∈ rs ,1+∈ rs ], prevents the policy update from being too large, and avoids training shocks or collapse.

[0039] It should be noted that in the training process, the above steps S1-S4 are iterated in a loop. Initially, the policy model is random, and the evaluation of the value model is also inaccurate. However, as the iteration proceeds, the value model becomes more and more robust through the information bottleneck and semantic perception mechanism, and can better resist the noise in the reward signal, thereby giving a more reliable advantage estimate. The policy model receives more reliable signals, and the update direction is more correct, so that it evolves into a high-performance policy more quickly. The high-performance policy produces higher-quality data, which further helps the value model improve.

[0040] Referring to Figure 2 , Figure 2 is another flowchart of model reinforcement learning based on value model enhancement provided by an embodiment of the present application. As shown in Figure 2 , first, trajectory sampling is performed to obtain trajectory data of the interaction between the policy model and the environment; then, the trajectory data is processed by the variational information bottleneck to extract core information related to value and policy; then, value prediction is performed based on the processed data, and the advantage function is obtained through advantage estimation; thereafter, the value model multi-objective optimization function fused with the mean square error, relative entropy regularization term and semantic perception loss is calculated, and the policy model target optimization function combined with the action probability ratio and the advantage function is calculated; then, the parameters of the large language model containing the value model and the policy model are updated according to these functions; finally, it is judged whether it is the last round of training. If not, the above process is repeated in the trajectory sampling link; if yes, the training is ended, and the reinforcement learning is completed.

[0041] In summary, the value model enhanced model reinforcement learning method provided by the embodiment of the application is a systematic enhancement scheme centered on the value model to solve the unstable training problem of the large language model in the noisy human feedback (RLHF). The core improvements are as follows: first, the value model is enhanced by using the variational autoencoder structure of the “encoder + decoder”, to realize state information compression and denoising and robust value prediction; meanwhile, a multi-objective joint loss function is designed, which includes the mean square error loss, the KL divergence regularization loss with dynamic information bottleneck, and the semantic perception loss, to balance the prediction accuracy, noise robustness and key semantic capture; second, the frozen language model head is innovatively used to dynamically identify high-entropy and high-entropy tokens, and the semantic perception loss is constructed based on the tokens, to force the value model to focus on the key semantic units and effectively suppress the “length hacker”; third, a KL divergence weight adaptive adjustment mechanism is designed, which can dynamically adjust the strength of the information bottleneck according to the training error. The embodiment of the application can significantly improve the learning performance and robustness of the large language model in a complex supervised environment.

[0042] To further verify the effectiveness of the embodiment of the application, a large language model trained based on the embodiment of the application is used for dialogue task training.

[0043] Referring to Figure 3 , Figure 3 is a training performance comparison chart of different reinforcement learning methods provided by the embodiment of the application in a dialogue task. As shown in Figure 3 , the horizontal axis is the step number, representing the number of training iterations or steps of interaction with the environment, and the vertical axis is the performance (%), used to measure the effect of the algorithm in completing the task. In the dialogue task training of the model reward, TCR represents the task completion rate, ACR represents the dialogue completion rate, and GCR represents the target completion rate, as shown in Figure 3 , the reinforcement learning method (OURS) provided by the embodiment of the application performs better than the existing PPO (proximal policy optimization algorithm) and GRPO (group relative policy optimization) in terms of: 1. Higher average completion rate, the average performance of the application reaches 83.82%, far exceeding the performance of PPO 40.13% and the performance of GRPO 36.17%. At the same time, the task completion rate in the training is higher than that of PPO and GRPO.

[0044] 2. Better training stability, stronger noise resistance, and smaller performance collapse, the model accuracy of the application decreases slowly in the training, and the performance of PPO and GRPO algorithms decreases rapidly.

[0045] Referring to Figure 4 , Figure 4 is a training performance comparison chart of different reinforcement learning methods provided by the embodiment of the application in different tasks in the optimization test under the rule reward. As shown in Figure 4As shown in the training-time optimization in the rule reward test, the method of the present application performs better than PPO and GRPO: In different domain tasks such as mathematics Math (MATH500, AIME24, Minerva-Math and AMC23), factuality (SampleQA), science (GPQA), knowledge (HLE) and the like, the method of the embodiment of the present application is better than the GRPO and PPO algorithms, and the average performance reaches 26.03%, and in each domain, it surpasses the GRPO and PPO algorithms, which shows that the method of the embodiment of the present application has a better noise optimization effect in multi-domain tasks.

[0046] Referring to Figure 5 , Figure 5 is a training performance comparison chart of different reinforcement learning methods provided by the embodiment of the present application in the training-time optimization in the mathematical task under the rule reward. As shown in Figure 5 the training-time optimization training in the rule reward, the method of the embodiment of the present application performs better than PPO and GRPO: In the mathematical task, the method of the embodiment of the present application can have better anti-noise ability in the training-time under the noise training in the weaker model (qwen2.5 cold start model), and the GRPO and PPO algorithms are both relatively fast in performance decline after training, and the GRPO algorithm even decreases from the original 37.38% average accuracy to 21.63%, but the method of the embodiment of the present application can be improved to 41.49% after training. But the performance gap is smaller in the stronger model (qwen3 model), and the average accuracy of the GRPO algorithm can reach 59.26% after training, and the method of the embodiment of the present application is still better than the GRPO and PPO algorithms to reach the average accuracy of 61.11%.

[0047] Referring to Figure 6 , Figure 6 is a comparison chart of the response length change of different reinforcement learning methods in the training process provided by the embodiment of the present application. As shown in Figure 6 can be known, the method of the embodiment of the present application has a stable model response length of 94-95 word units on the dialogue task, and compared with the length expansion phenomenon of PPO and GRPO, the length change phenomenon of the method of the embodiment of the present application is extremely small, which shows that the problem of "length hacker" can be effectively inhibited.

[0048] On the basis of the above method item, the embodiment of the system item is correspondingly provided.

[0049] Referring to Figure 7 , Figure 7is a structural block diagram of a model reinforcement learning system based on value model enhancement provided by an embodiment of the present application. The model reinforcement learning system based on value model enhancement comprises: A training trajectory data sampling module 21 is configured to generate training trajectory data containing states, actions and rewards by interacting with an environment through a current policy model, and calculate action probability ratios of new and old policies based on the training trajectory data; An advantage function calculation module 22 is configured to predict state values through a value model based on the training trajectory data, and calculate an advantage function and a target return based on the predicted state values by using advantage estimation; A target function construction module 23 is configured to construct a multi-objective optimization function of the value model according to the training trajectory data, the predicted state values and the target return, and calculate a proximal policy optimization target function based on the action probability ratios and the advantage function; wherein the multi-objective optimization function comprises a mean square error loss, a relative entropy regularization term loss and a semantic perception loss. A model optimization module 24 is configured to update the value model by using an exponential moving average algorithm according to the multi-objective optimization function, and optimize the current policy model according to the proximal policy optimization target function until a preset iteration condition is met, so as to obtain a policy model after reinforcement learning.

[0050] In an optional embodiment, the advantage function calculation module 22 is configured to: map the states into Gaussian latent variables through an encoder of the value model based on the training trajectory data; predict the state values through a decoder of the value model based on the Gaussian latent variables; map the states into Gaussian latent variables through the encoder of the value model according to the following formula: calculate the advantage function and the target return by using generalized advantage estimation of nonlinear transformation based on the predicted state values.

[0051] It should be noted that the model reinforcement learning system based on value model enhancement provided by the embodiment of the present application is used to execute all process steps of the model reinforcement learning method based on value model enhancement of the above-mentioned embodiment, and the working principles and beneficial effects of the two are one-to-one correspondence, thus it is not repeated.

[0052] The embodiment of the present application further provides a terminal device, as shown in Figure 8 The structural block diagram of a terminal device provided by an embodiment of the present application is shown. The terminal device comprises a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31, and the processor 31 implements the model reinforcement learning method based on value model enhancement as described in any of the above embodiments when executing the computer program.

[0053] In addition, the embodiments of the present application further provide a computer readable storage medium, which comprises a stored computer program, wherein the computer readable storage medium controls a device on which the computer readable storage medium is located to execute the model reinforcement learning method based on value model enhancement according to any one of the above embodiments when the computer program is executed.

[0054] The processor 31 implements the steps in the above embodiments of the model reinforcement learning method based on value model enhancement when executing the computer program, for example, all the steps of the model reinforcement learning method based on value model enhancement shown in the above embodiments. Figure 1 The processor 31 implements the steps in the above embodiments of the model reinforcement learning method based on value model enhancement when executing the computer program, for example, all the steps of the model reinforcement learning method based on value model enhancement shown in the above embodiments. Figure 7 The processor 31 implements the steps in the above embodiments of the model reinforcement learning method based on value model enhancement when executing the computer program, for example, all the steps of the model reinforcement learning method based on value model enhancement shown in the above embodiments.

[0055] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the terminal device.

[0056] The processor 31 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 31 can also be any conventional processor. The processor 31 is the control center of the terminal device, which connects various parts of the terminal device through various interfaces and lines.

[0057] The memory 32 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, and the like, and the data storage area can store relevant data and the like. In addition, the memory 32 can be a high-speed random access memory, and can also be a non-volatile memory such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like, or can be another volatile solid-state storage device.

[0058] It should be noted that the terminal device described above can include, but is not limited to, a processor and a memory, and those skilled in the art can understand that Figure 8 The structural block diagram shown is only an example of the structure of the terminal device described above, and does not constitute a limitation on the structure of the terminal device described above. The terminal device described above can include more or fewer components than those shown, or combine certain components, or different components.

[0059] The above describes the preferred embodiments of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered within the scope of protection of the present application.

Claims

1. A model reinforcement learning method based on value model enhancement, characterized in that, include: Training trajectory data containing state, action, and reward is generated by the interaction between the current policy model and the environment. The action probability ratio of the new and old policies is calculated based on the training trajectory data. Based on the training trajectory data, the state value is predicted using a value model; based on the predicted state value, the advantage function and target reward are calculated using advantage estimation. The multi-objective optimization function of the value model is constructed based on the training trajectory data, the predicted state value, and the target reward. The proximal policy optimization objective function is calculated based on the action probability ratio and the advantage function. The multi-objective optimization function includes mean squared error loss, relative entropy regularization term loss, and semantic awareness loss. The value model is updated using the exponential moving average algorithm based on the multi-objective optimization function; the current policy model is optimized according to the near-end policy optimization objective function until the preset iteration conditions are met, thus obtaining the policy model after reinforcement learning.

2. The model reinforcement learning method based on value model enhancement as described in claim 1, characterized in that, The step of predicting state value using a value model based on the training trajectory data, and calculating the advantage function using generalized advantage estimation based on the predicted state value, includes: Based on the training trajectory data, the encoder of the value model maps the state to Gaussian latent variables; Based on the Gaussian latent variables, the state value is predicted through the decoder of the value model; The encoder of the value model maps the state to Gaussian latent variables: Based on the predicted state value, the advantage function and target return are calculated using generalized advantage estimation with nonlinear transformation.

3. The model reinforcement learning method based on value model enhancement as described in claim 2, characterized in that, The state is mapped to Gaussian latent variables using the encoder of the value model according to the following formula: Among them, z t Let x be a Gaussian latent variable, representing the condition of input x. t Compressed representation; Indicates that the encoder f φ The parameter φ defines the mean function, and the input is x. t The output is the mean vector of the probability distribution of the latent variables; encoder f φ The parameter φ defines the standard deviation function, and the input is x. t The output is the standard deviation vector of the latent variable probability distribution; ∈ indicates that it follows a standard normal distribution. The random number sampled in the middle represents the reparameterized sampling noise, and I is a d*d identity matrix.

4. The model reinforcement learning method based on value model enhancement as described in claim 1, characterized in that, The multi-objective optimization function includes: in, Represents a multi-objective optimization function; This represents the mean squared error loss; β represents the loss due to the relative entropy regularization term. IB The weights for the relative entropy regularization loss; The semantic perception loss is represented by T; the trajectory length is represented by T. R represents the predicted state value at time t; t Represents the target reward at time t; ∈ indicates that it follows a standard normal distribution. Random numbers sampled from the middle, where I is a d*d identity matrix; x t Indicates the state of the input text at time t; z t p represents the Gaussian latent variable at time t; φ (z t |x t ) represents encoder f φ The posterior distribution of the Gaussian latent variable; r(z) represents the Gaussian latent variable z. t The prior distribution of H; RS (t) represents the entropy loss; t ent Represents a set of high-entropy lexical units; λ ent The weighting coefficients representing entropy loss; L rNLL (t) represents the perplexity loss; T ppl Represents a set of highly perplexing lexical units; λ ppl The weighting coefficients represent the perplexity loss.

5. The model reinforcement learning method based on value model enhancement as described in claim 4, characterized in that, The semantic perception loss is achieved through the following steps: The predicted distribution of lexical units is obtained using a frozen language model head that shares the embedding layer with the current policy model; Based on the predicted distribution, a set of high-entropy lexical units is selected according to a preset entropy threshold. Based on the predicted distribution, a set of high-perplexity lexical terms is selected according to a preset perplexity threshold. The semantic perception loss is obtained by weighting the high-entropy lexical set and the high-perplexity lexical set.

6. The model reinforcement learning method based on value model enhancement as described in claim 5, characterized in that, The entropy loss is calculated using the following formula: Where y represents the candidate value of the lexical unit; p V (y|x t ) represents the predicted distribution of lexical units in the frozen language model head that shares the embedding layer with the current policy model; β H Represents the entropy smoothing parameter; The perplexity loss is calculated using the following formula: Among them, y t This represents the candidate values ​​of the lexical unit at time t; This represents the target word given at time t; β represents the negative log-likelihood value of the target word. A higher value indicates greater difficulty for the value model to predict that word, i.e., higher perplexity. P This represents the perplexity smoothing parameter.

7. The model reinforcement learning method based on value model enhancement as described in claim 1, characterized in that, The ratio of the action probabilities of the new and old strategies is calculated using the following formula: Where, ρ t x represents the ratio of the probabilities of the new and old strategies at time t; t Represents the state of the input text at time t; a t The action at time t; r t Represents the reward at time t; π θ Indicates the current policy model; This represents the policy model from the previous iteration. Indicates the relationship between ρ t The transformed value is used to avoid ρ t Extreme values ​​have a drastic impact on policy updates.

8. A model reinforcement learning system based on value model enhancement, characterized in that, include: The training trajectory data sampling module is used to generate training trajectory data containing state, action and reward through the interaction between the current policy model and the environment, and to calculate the action probability ratio of the new and old policies based on the trained trajectory data; The advantage function calculation module is used to predict the state value based on the training trajectory data using a value model; and to calculate the advantage function and target reward based on the predicted state value using advantage estimation. The objective function construction module is used to construct a multi-objective optimization function for the value model based on the training trajectory data, the predicted state value, and the target reward, and to calculate the proximal policy optimization objective function based on the action probability ratio and the advantage function; wherein, the multi-objective optimization function includes mean squared error loss, relative entropy regularization term loss, and semantic awareness loss; The model optimization module is used to update the value model using an exponential moving average algorithm based on the multi-objective optimization function; and to optimize the current policy model based on the near-end policy optimization objective function until a preset iteration condition is met, thereby obtaining the policy model after reinforcement learning.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the value model-based reinforcement learning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the model reinforcement learning method based on value model enhancement as described in any one of claims 1 to 7.

Images