A large model-oriented value network state adaptive reinforcement learning training method and system and medium

Abstract

The application relates to a large model-oriented value network state adaptive reinforcement learning training method and system and a medium, the method comprises the following steps: interacting with a training environment through a current actor policy network, collecting training samples, and obtaining training data; meanwhile, analyzing policy changes, task performance and policy diversity during training, and constructing training state description information; a critic state evaluation mechanism is constructed by fusing the policy, the performance and the diversity, the update requirement of the value network in the current training stage is analyzed, and whether to perform critic training is dynamically decided according to the evaluation result; after completing the critic update decision, the actor policy network is optimized and trained by using the value estimation result. The application realizes the effective reduction of critic network update overhead, the improvement of training efficiency and the enhancement of the adaptive ability of the learning training process on the basis of maintaining the reinforcement learning training performance and the training stability.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a method, system, and medium for adaptive reinforcement learning training of value network states for large models. Background Technology

[0002] With the rapid development of large language models and reinforcement learning techniques, Actor-Critic reinforcement learning algorithms, represented by Proximal Policy Optimization (PPO), have been widely applied in areas such as model alignment, inference capability optimization, code generation, and training for complex decision-making tasks. In current mainstream reinforcement learning training frameworks, a dual Actor-Critic network structure is typically employed, where the Actor is responsible for policy generation and the Critic for value evaluation; the two are updated alternately to improve policy performance.

[0003] Existing PPO training processes typically employ a fixed update mechanism, which involves performing the following steps in each training cycle: (1) Rollout sampling stage: Use the current Actor strategy to generate training samples.

[0004] (2) Critic training phase: update the value network based on sampled data.

[0005] (3) Actor training phase: Update the policy network using the advantage function output by the Critic.

[0006] This training paradigm has become the standard implementation method in current reinforcement learning systems. However, in actual training, as the model's capabilities improve and training gradually converges, the value function of the Critic network often enters a relatively stable state. At this point, continuing to update the Critic at a fixed frequency not only incurs additional computational overhead but may also affect the stability of the policy network training.

[0007] Some current studies attempt to alleviate the above problems by reducing the Critic learning rate, reducing the number of Critic training rounds, or adjusting the Actor-Critic update ratio. However, these studies still essentially use preset update rules and lack a mechanism to dynamically determine whether the Critic needs to be updated based on the training status.

[0008] Although existing Actor-Critic reinforcement learning frameworks have been widely adopted, they still have the following shortcomings: (1) Fixed updates to Critic lead to a waste of computing resources. The existing PPO algorithm requires the Critic to be updated synchronously with the Actor in every training cycle. As training progresses into the mid-to-late stages, the magnitude of change in the value function gradually decreases, and a large number of Critic updates do not bring significant benefits, but still consume a lot of GPU computing power, which leads to a decrease in overall training efficiency.

[0009] (2) Lack of ability to perceive the degree of strategic change Existing methods simply perform Critic updates according to a pre-defined training process, failing to detect significant changes in the current policy network. When policy distribution changes are minor, continuing to update the Critic often has limited value; however, when the policy changes drastically, more frequent value corrections may be needed. Fixed update mechanisms cannot adaptively adjust based on the policy's state.

[0010] (3) Unable to monitor the degradation of strategy diversity During reinforcement learning training, policies may gradually shrink to include at least a few high-probability actions, resulting in policy collapse and weakened exploration capabilities. Existing PPO training frameworks primarily focus on reward optimization and lack a real-time monitoring mechanism for the degree of policy diversity degradation, which can easily lead to a decline in the model's generalization ability.

[0011] (4) The training frequencies of the Actor (policy network) and the Critic (value network) are too tightly coupled. Existing methods assume a fixed update relationship between Actors and Critics, meaning that Critic training must be completed before each Actor update. This tight coupling mechanism limits the flexibility of the training process, preventing the system from dynamically adjusting the value network update frequency based on the current training state. Summary of the Invention

[0012] This invention addresses the problems of wasted computational resources and low training efficiency caused by fixed-frequency updates of the Critic network during existing reinforcement learning training processes. It provides a value network state adaptive reinforcement learning training method, system, and medium for large models, aiming to effectively reduce the Critic network update overhead, improve training efficiency, and enhance the adaptive capability of the learning and training process while maintaining reinforcement learning training performance and stability.

[0013] This invention provides an adaptive reinforcement learning training method for value network states in large models, the method comprising the following steps: Step S10: Interact with the training environment through the current Actor policy network to collect training samples and obtain training data, which includes state, action, reward and policy information; at the same time, analyze the policy changes, task performance and policy diversity during the training process to construct training state description information, and provide basic input for subsequent Critic state evaluation. Step S20: Construct a Critic state evaluation mechanism that integrates strategy, performance, and diversity, analyzes the update requirements of the value network in the current training phase, and dynamically decides whether to perform Critic training based on the evaluation results. Critic updates are triggered when the training state changes significantly; the current Critic training is skipped when the training state remains stable. Step S30: After completing the Critic update decision, the Actor policy network is optimized and trained using the value estimation results; when the Critic network has not been updated for several consecutive rounds, a forced update is automatically triggered to ensure the reliability of the value function training process.

[0014] A further technical solution of the present invention is that step S10 includes: Step S101: Based on the current Actor policy network, interact with the training environment and collect training samples through the Rollout process; in each training cycle, the Actor generates actions according to the current environment state and interacts with the environment to obtain corresponding feedback information. The feedback information includes the current state and the next state, the action information includes the currently executed action and the corresponding probability of the action, the reward information includes the immediate reward and the cumulative reward, and the policy information includes the current policy distribution and the historical policy distribution. Based on the feedback information, training trajectory data Trajectory is formed through continuous interaction, sampling is completed, and then the training trajectories are organized into training dataset D. Among them, Trajectory = {(State1, Action1, Reward1), (State2, Action2,Reward2), … , (State t Action t Reward t )} Where t represents the length of the current training trajectory, State represents the current state, Action represents the current action being executed, and Reward represents the immediate reward; D= {State, Action, Reward, Return, Policy} Where State is the current state, Action is the current action being performed, Reward is the immediate reward, Return is the cumulative reward, and Policy is the policy information; Step S102: Construct policy change information based on the current policy distribution and the historical policy distribution to form a policy state description quantity PolicyState. The KL divergence between the current policy and the historical reference policy is used as an indicator of the degree of policy change. The policy change description information is used to reflect whether the current policy has been significantly updated, whether the policy search space has shifted significantly, and whether there is a need for re-estimation of the value function in the current training stage. Step S103: Construct task performance information, perform real-time statistics on training tasks, and form a performance state description quantity PerformanceState to reflect the changes in model capabilities during the current training phase. The task performance information includes: task accuracy, average reward, success rate, and task completion quality indicators. Step S104: Perform statistical analysis on the policy output distribution, construct policy diversity information, and form a diversity state description quantity DiversityState to monitor the possible decline in exploration ability during reinforcement learning training. The policy diversity information includes policy entropy, historical average entropy, entropy change trend, and degree of diversity degradation. The policy entropy is used to measure the dispersion of the policy output distribution. Step S105: Construct a training state description vector based on the policy change information, task performance information, and policy diversity information:

[0015] The training state description vector serves as the input for subsequent Critic state evaluation, used to dynamically determine the update requirements of the value network in the current training phase.

[0016] A further technical solution of the present invention is that step S20 includes: Step S201: Establish a training state model to describe the update requirements of the Critic network during reinforcement learning training; wherein, the current training state is defined as:

[0017] in: Indicates the policy state, used to describe the current changes in the policy space; Indicates performance status, used to describe the model's task completion status during the current training phase; The diverse states are used to describe the current policy exploration capability; the policy state, performance state, and diverse states together constitute the Critic training state space, providing a unified state representation for subsequent value network update decisions. Step S202: After establishing the training state space, establish the policy state, performance state, and diversity state feature models. Step S203: Based on the policy state, performance state, and diversity state feature model, establish a comprehensive Critic state model to comprehensively evaluate the current Critic network training requirements. Step S204: Based on the Critic integrated state, establish a gated update model. Through a preset gated update mechanism, realize the transformation of the Critic network from a fixed update mode to a training state-driven update mode, so as to reduce the training overhead of the value network while ensuring the stability of the value function.

[0018] A further technical solution of the present invention is that step S202 includes: Step S2021, adopt strategy distribution difference As a descriptor of policy state, a policy state feature model is established to describe the degree of change in the current policy network relative to historical policies, including the policy distribution differences. Represented as:

[0019] in, Indicates the current Actor policy. Indicates a reference strategy; Step S2022: Establish a performance state feature model to reflect the performance changes of the current model in the training task, wherein the performance state function... Represented as:

[0020] in Accuracy is a performance evaluation metric for models during the current training phase. Step S2023: Establish a policy diversity feature model to reflect changes in policy exploration ability during reinforcement learning training; this involves first calculating the current policy entropy. And calculate its exponential moving average. Further define the state of strategy diversity :

[0021] When strategy outputs become more concentrated and exploration capabilities decline... Gradually increase.

[0022] A further technical solution of the present invention is that, in step S203, the Critic comprehensive state function is defined as follows:

[0023] in , , These represent the weight coefficients corresponding to the strategy state, performance state, and diversity state, respectively; Critic comprehensive state function It is used to comprehensively reflect the update needs of the Critic network under the current training state.

[0024] A further technical solution of the present invention is that the update rule in step S204 includes the following steps: Step S2041, synthesize the Critic state function. With update threshold Perform a comparison; Step S2042, if the Critic comprehensive state function Greater than the update threshold If the condition is met, then Critic network training will be performed; otherwise, skip this round of Critic training and retain the current value network parameters to continue participating in policy optimization.

[0025] A further technical solution of the present invention is that step S30 includes: Step S301: Establish a Critic network feedback model to estimate the value function corresponding to the current state and provide value feedback information to the Actor network; Step S302: Optimize the strategy based on the feedback results of the Critic network feedback model and establish the Actor strategy optimization model. Step S303: Establish a continuous skip constraint model to avoid the value estimate deviating from the true return due to the Critic network not being updated for a long time.

[0026] A further technical solution of the present invention is that step S301 includes: The Critic value function is defined as follows:

[0027] The system outputs the corresponding state value based on the current training state, which serves as an important basis for policy optimization. When the Critic performs an update, the latest value function is used in policy training. When the Critic skips the update, the value function from the most recent training iteration is used for value estimation to ensure continuous and stable training. Step S302 includes: The current policy is defined as follows:

[0028] Where θ represents the Actor network parameters. Further construct the advantage function:

[0029] Where R represents the current training reward, and V(s) represents the Critic output value; Subsequently, the policy network parameters are updated according to the advantage function, so that the Actor continuously approaches the optimal policy; the entire Actor optimization process is always executed in every round, while the Critic dynamically decides whether to update according to the training state, thereby realizing collaborative training between the Actor and the Critic at different update frequencies. Step S303 includes: Define the number of consecutive skips as SkipCount, when Critic finishes updating:

[0030] When Critic skips the update:

[0031] Further, set the maximum number of consecutive skips (SkipMax). Preferably, set SkipMax to a range of 3 to 5 rounds. At this time, a Critic training is triggered, and the consecutive skip count is reinitialized.

[0032] To achieve the above objectives, the present invention also proposes a value network state adaptive reinforcement learning training system for large models. The system includes a memory, a processor, and a value network state adaptive reinforcement learning training program for large models stored in the memory. The value network state adaptive reinforcement learning training program for large models is executed by the processor to perform the steps of the method described above.

[0033] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a value network state adaptive reinforcement learning training program for a large model, wherein the value network state adaptive reinforcement learning training program for a large model is executed by a processor to perform the steps of the method described above.

[0034] This invention presents a value network state adaptive reinforcement learning training method, system, and medium for large models. By introducing multi-track state modeling and a gated update mechanism, it achieves dynamic adjustment of the Critic training frequency. Furthermore, by constructing a Critic adaptive reinforcement learning method based on multi-track state modeling, dynamic score fusion, multi-constraint gated updates, and asynchronous collaborative training, it transforms the Critic update mechanism from fixed-period updates to adaptive updates. Specifically, this invention achieves the following technical effects: 1. A multi-track state modeling method for the value network training process: This invention proposes a state modeling method for the training process of Critic in reinforcement learning. It abstracts the Critic update requirements into a state space representation, including policy drift states, task performance states, and policy diversity states. Through a multi-track state modeling method, a structured expression of the Critic update requirements is achieved.

[0035] 2. Introduce a value network state fusion scoring mechanism based on time-adaptive weight modulation: This invention proposes a value network update scoring method based on multi-state fusion, which weights and fuses policy drift state, task performance state, and policy diversity state to construct a comprehensive Critic scoring function. Through this mechanism, the Critic update decision-making is transformed from fixed threshold judgment to adaptive scoring with dynamic weight adjustment.

[0036] 3. Value network adaptive update method based on multi-constraint gating mechanism: This invention proposes a value network gating update method that dynamically determines whether to perform Critic parameter updates based on the fusion state score results. Through a multi-constraint gating mechanism, the Critic update process is transformed from a single threshold control to a multi-control mechanism that integrates score-driven and continuous protection approaches, effectively improving training stability.

[0037] 4. Training stability control method based on skip protection mechanism and KL safety constraints: This invention proposes a protection mechanism for the long-term stability of value networks, including a continuous skip protection mechanism and a KL security constraint mechanism. This stability control mechanism ensures low-frequency updates of the Critic function while preventing value function degradation. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating a preferred embodiment of the value network state adaptive reinforcement learning training method for large models according to the present invention. Figure 2 This is a schematic diagram of the overall implementation framework of the reinforcement learning training method based on Critic state adaptation; Figure 3 This is a schematic diagram of the Critic state assessment and adaptive update mechanism; Figure 4 This is a schematic diagram of the hardware architecture of the value network state adaptive reinforcement learning training system for large models according to the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0040] To address the problems of wasted computational resources, lack of training state awareness, and low training efficiency caused by fixed-frequency updates of the Critic network in existing reinforcement learning training methods, this invention proposes a reinforcement learning training method based on Critic state adaptation.

[0041] This invention constructs a Critic state evaluation mechanism that integrates the degree of policy change, task performance changes, and policy diversity status. This mechanism performs real-time analysis of the current training state and dynamically determines whether to perform a Critic update based on the evaluation results. Simultaneously, a continuous skip protection mechanism is introduced to reduce unnecessary Critic training iterations while ensuring the stability of the value function.

[0042] In this invention, the Critic network (i.e., the value network) and the Actor network (i.e., the policy network) constitute a dual-network training architecture.

[0043] Specifically, such as Figure 1 back Figure 2 As shown, a preferred embodiment of the value network state adaptive reinforcement learning training method for large models of the present invention includes the following steps: Step S10, Training Data Acquisition and State Information Construction: Through the interaction between the current Actor policy network and the training environment, training samples are collected to obtain training data, which includes state, action, reward and policy information; at the same time, the policy changes, task performance and policy diversity during the training process are analyzed to construct training state description information, providing basic input for subsequent Critic state evaluation; Step S20, Dynamic update decision based on Critic state evaluation: Construct a Critic state evaluation mechanism that integrates strategy, performance, and diversity; analyze the update requirements of the value network in the current training phase; and dynamically decide whether to perform Critic training based on the evaluation results; trigger Critic update when the training state changes significantly; and skip the current Critic training when the training state remains stable. Step S30, Strategy Optimization and Stability Assurance: After completing the Critic update decision, the Actor policy network is optimized and trained using the value estimation results; at the same time, a continuous skip protection mechanism is introduced, which automatically triggers a forced update when the Critic network has not been updated for several consecutive rounds, so as to ensure the reliability of the value function training process.

[0044] This embodiment achieves dynamic adjustment of the Critic update frequency during reinforcement learning training through the above technical solution. It reduces the computational resource consumption caused by redundant value network training while ensuring training performance, and improves training efficiency while ensuring training performance and stability. It provides a more efficient adaptive optimization scheme for large-scale model reinforcement learning training.

[0045] In this embodiment, in the construction of training data acquisition and state information, a reinforcement learning training state description system is first constructed. Training samples are obtained through the interaction between the policy network and the training environment. The policy change information, task performance information and policy diversity information generated during the training process are analyzed to form a training state description vector for Critic state evaluation.

[0046] Step S10 includes: Step S101, based on the current Actor policy network interacting with the training environment, training samples are collected through the Rollout process; in each training cycle, the Actor generates an action according to the current environment state and interacts with the environment to obtain corresponding feedback information, the feedback information includes the current state and the next state, the action information includes the currently executed action and the corresponding probability of the action, the reward information includes the immediate reward and the cumulative reward, and the policy information includes the current policy distribution and the historical policy distribution; Training trajectory data (Trajectory) is formed through continuous interaction based on the state, action, reward, and policy information. After sampling, the training trajectories are organized into a training dataset D. This dataset provides basic data support for subsequent Critic state evaluation and Actor training.

[0047] Among them, Trajectory = {(State1, Action1, Reward1), (State2, Action2,Reward2), … , (State t Action t Reward t )}; Where t represents the length of the current training trajectory, State represents the current state, Action represents the current action being executed, and Reward represents the immediate reward; D= {State, Action, Reward, Return, Policy}; Where State is the current state, Action is the current action being performed, Reward is the immediate reward, Return is the cumulative reward, and Policy is the policy information.

[0048] Step S102: Construct policy change information based on the current policy distribution and historical policy distribution to form a policy state description quantity PolicyState. The KL divergence between the current policy and the historical reference policy is used as an indicator of the degree of policy change. The policy change description information is used to reflect whether the current policy has been significantly updated, whether the policy search space has shifted significantly, and whether there is a need for re-estimation of the value function in the current training phase.

[0049] To measure the degree of change in the policy network during the current training phase, this embodiment constructs policy change description information based on the current policy distribution and historical policy distribution. Specifically, the KL divergence between the current policy and the historical reference policy is used as an indicator of the degree of policy change. The policy change information mainly reflects: whether the current policy has been significantly updated, whether the policy search space has shifted significantly, and whether there is a need for re-estimation of the value function during the current training phase. By continuously statistically analyzing the policy changes, a policy state description quantity, PolicyState, is formed. The larger the PolicyState value, the more significant the policy change, and the higher the need for refitting of the Critic network.

[0050] Step S103: Construct task performance information, perform real-time statistics on training tasks, and form a performance state description quantity PerformanceState to reflect the changes in model capabilities during the current training phase. The task performance information includes: task accuracy, average reward, success rate, and task completion quality indicators.

[0051] To reflect the changes in model capabilities during the current training phase, this embodiment performs real-time statistics on training task performance. Task performance information may include: task accuracy, average reward, success rate, and task completion quality metrics. These metrics reflect the performance of the current strategy on the target task. Through continuous monitoring of task performance, a performance state descriptor, PerformanceState, is generated.

[0052] Step S104: Perform statistical analysis on the policy output distribution to construct policy diversity information and form a diversity state description quantity, DiversityState, to monitor the possible decline in exploration ability during reinforcement learning training. The policy diversity information includes policy entropy, historical average entropy, entropy change trend, and degree of diversity degradation. The policy entropy is used to measure the dispersion of the policy output distribution.

[0053] To monitor potential decline in exploration ability during reinforcement learning training, this invention performs statistical analysis on the policy output distribution. Policy diversity information mainly includes: policy entropy, historical average entropy (EMAEntropy), entropy change trend, and degree of diversity degradation. Policy entropy measures the dispersion of the policy output distribution; higher entropy indicates stronger policy exploration ability. By continuously statistically analyzing the policy diversity state, a diversity state descriptor, DiversityState, is formed.

[0054] Step S105: Construct a training state description vector based on the policy change information, task performance information, and policy diversity information: ; The training state description vector serves as the input for subsequent Critic state evaluation, used to dynamically determine the update requirements of the value network in the current training phase.

[0055] This embodiment constructs a training state description vector after statistically analyzing policy change information, task performance information, and policy diversity information. This training state vector serves as input to the subsequent Critic state evaluation module, used to dynamically determine the update needs of the value network in the current training phase. Simultaneously, through the above steps, this embodiment completes training sample collection and state information construction, providing fundamental support for subsequent dynamic update decisions by the Critic.

[0056] After constructing the training state information, this embodiment evaluates whether the current Critic network needs updating based on policy changes, task performance changes, and policy diversity. The Critic training frequency is then dynamically adjusted based on the evaluation results. The overall process is as follows: Figure 3 As shown.

[0057] Specifically, step S20, the dynamic update decision based on the value network state assessment, includes the following steps: Step S201, Value Network Training State Modeling: Establish a training state model to describe the update requirements of the Critic network during reinforcement learning training; wherein, the current training state is defined as:

[0058] in: Indicates the policy state, used to describe the current changes in the policy space; Indicates performance status, used to describe the model's task completion status during the current training phase; The diverse states represent the current policy exploration capabilities; the policy states, performance states, and diverse states together constitute the Critic training state space, providing a unified state representation for subsequent value network update decisions.

[0059] Step S202, Value Network State Feature Modeling: After establishing the training state space, establish policy state, performance state, and diversity state feature models.

[0060] Step S203, Value Network Comprehensive State Feature Modeling: Based on the policy state, performance state, and diversity state feature model, establish a Critic comprehensive state model to comprehensively evaluate the current Critic network training requirements.

[0061] Step S204: Based on the Critic integrated state, establish a gated update model. Through a preset gated update mechanism, realize the transformation of the Critic network from a fixed update mode to a training state-driven update mode, so as to reduce the training overhead of the value network while ensuring the stability of the value function.

[0062] Specifically, step S202 includes: Step S021, Policy State Modeling: Using Policy Distribution Differences As a descriptor of policy state, a policy state feature model is established to describe the degree of change in the current policy network relative to historical policies, including the policy distribution differences. Represented as:

[0063] in, Indicates the current Actor policy. This represents the reference policy. The larger the policy state, the more significant the current policy update, and the higher the need for refitting the value function.

[0064] Step S2022, Performance State Modeling: Establish a performance state feature model to reflect the performance changes of the current model in the training task, wherein the performance state function... Represented as:

[0065] in Accuracy represents the model's performance metric during the current training phase. As model performance declines, accuracy decreases. The gradual increase indicates that the current Critic network needs further estimation.

[0066] Step S2023, Diverse State Modeling: Establish a policy diversity feature model to reflect changes in policy exploration ability during reinforcement learning training; firstly, the current policy entropy is calculated. And calculate its exponential moving average. Further define the state of strategy diversity :

[0067] When strategy outputs become more concentrated and exploration capabilities decline... Gradually increase.

[0068] In step S203, the Critic comprehensive state function is defined as follows:

[0069] in , , These represent the weight coefficients corresponding to the strategy state, performance state, and diversity state, respectively. The preferred value range is [0.1, 1.0] and satisfies α + β + γ = 1.0. In one preferred embodiment, α = 0.5, β = 0.3, and γ = 0.2 are set. The preferred range for the corresponding update threshold Threshold is [0.3, 0.7]. In one preferred embodiment, Threshold = 0.5 is set.

[0070] Critic Synthesis State Function It is used to comprehensively reflect the update needs of the Critic network under the current training state. When the policy changes significantly, the model performance deteriorates, or the policy exploration ability weakens, the AdaptiveScore will increase accordingly.

[0071] The update rule in step S204 includes the following steps: Step S2041, synthesize the Critic state function. With update threshold Compare them.

[0072] Step S2042, if the Critic comprehensive state function Greater than the update threshold If the condition is met, then Critic network training will be performed; otherwise, skip this round of Critic training and retain the current value network parameters to continue participating in policy optimization.

[0073] Based on the Critic synthesis state, this embodiment establishes a gated update model. The update flag is set as follows:

[0074] The update rules are as follows:

[0075] but:

[0076] Perform Critic network training, otherwise:

[0077] Skip this round of Critic training and retain the current value network parameters to continue participating in policy optimization. Through the above gating update mechanism, the Critic network is transformed from a fixed update mode to a training state-driven update mode, which can effectively reduce the training overhead of the value network while ensuring the stability of the value function.

[0078] After completing the Critic training state modeling and gating update, this embodiment further establishes a collaborative optimization model between the Critic and the Actor. The current training state is fed back through the value network, guiding the policy network to continuously optimize and forming a complete reinforcement learning training loop.

[0079] Specifically, step S30 includes the following steps: Step S301, Value Network Feedback Modeling: Establish a Critic network feedback model to estimate the value function corresponding to the current state and provide value feedback information to the Actor network.

[0080] Step S302, Actor strategy optimization modeling: Complete strategy optimization based on the feedback results of the Critic network feedback model and establish the Actor strategy optimization model; Step S303, Training stability modeling: Establish a continuous skip constraint model to avoid the value estimation deviating from the true return due to the Critic network not being updated for a long time.

[0081] Step S301 specifically includes the following steps: The Critic value function is defined as follows:

[0082] The system outputs the corresponding state value based on the current training state, which serves as an important basis for policy optimization. When the Critic performs an update, the latest value function is used in policy training. When the Critic skips the update, the value function from the most recent training iteration is used for value estimation to ensure continuous and stable training.

[0083] Step S302 specifically includes the following: The current policy is defined as follows:

[0084] Where θ represents the Actor network parameters. Further construct the advantage function:

[0085] Where R represents the current training reward, and V(s) represents the Critic output value.

[0086] Subsequently, the policy network parameters are updated according to the advantage function, so that the Actor continuously approaches the optimal policy. The entire Actor optimization process is always executed in every round, while the Critic dynamically decides whether to update based on the training state, thereby achieving collaborative training between the Actor and the Critic at different update frequencies.

[0087] Step S303 specifically includes the following steps: Define the number of consecutive skips as SkipCount, when Critic finishes updating:

[0088] When Critic skips the update:

[0089] Further, set the maximum number of consecutive skips (SkipMax). Preferably, set SkipMax to a range of 3 to 5 rounds. At each step, a Critic training session is triggered, and the consecutive skip count is reinitialized. This constraint mechanism allows the Critic network to maintain a low training frequency while avoiding value function shifts caused by prolonged periods without updates, thus improving the stability of the reinforcement learning training process.

[0090] This invention presents a value network state adaptive reinforcement learning training method for large models. By introducing multi-track state modeling and a gating update mechanism, it achieves dynamic adjustment of the Critic training frequency. Furthermore, by constructing a Critic adaptive reinforcement learning method based on multi-track state modeling, dynamic score fusion, multi-constraint gating update, and asynchronous collaborative training, it transforms the Critic update mechanism from fixed-period updates to adaptive updates. Specifically, this invention achieves the following technical effects: 1. A multi-track state modeling method for the value network training process: This invention proposes a state modeling method for the training process of Critic in reinforcement learning. It abstracts the Critic update requirements into a state space representation, including policy drift states, task performance states, and policy diversity states. Through a multi-track state modeling method, a structured expression of the Critic update requirements is achieved.

[0091] 2. Introduce a value network state fusion scoring mechanism based on time-adaptive weight modulation: This invention proposes a value network update scoring method based on multi-state fusion, which weights and fuses policy drift state, task performance state, and policy diversity state to construct a comprehensive Critic scoring function. Through this mechanism, the Critic update decision-making is transformed from fixed threshold judgment to adaptive scoring with dynamic weight adjustment.

[0092] 3. Value network adaptive update method based on multi-constraint gating mechanism: This invention proposes a value network gating update method that dynamically determines whether to perform Critic parameter updates based on the fusion state score results. Through a multi-constraint gating mechanism, the Critic update process is transformed from a single threshold control to a multi-control mechanism that integrates score-driven and continuous protection approaches, effectively improving training stability.

[0093] 4. Training stability control method based on skip protection mechanism and KL safety constraints: This invention proposes a protection mechanism for the long-term stability of value networks, including a continuous skip protection mechanism and a KL security constraint mechanism. This stability control mechanism ensures low-frequency updates of the Critic function while preventing value function degradation.

[0094] To achieve the above objectives, this invention also proposes a value network state adaptive reinforcement learning training system for large models, such as... Figure 4 As shown, the system includes a processor 1001, a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002, and a value network state adaptive reinforcement learning training program for large models stored on the processor. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0095] Those skilled in the art will understand that Figure 4 The system structure shown does not constitute a limitation on the system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] like Figure 4As shown, the memory 1005, which serves as a computer storage medium, may include an operating device, a network communication module, a user interface module, and a value network state adaptive reinforcement learning training program for large models.

[0097] exist Figure 4 In the system shown, the network interface 1004 is mainly used to connect to the network server and communicate with the network server; the user interface 1003 is mainly used to interact with the user terminal and receive user input instructions; and the processor 1001 can be used to call the value network state adaptive reinforcement learning training program for large models stored in the memory 1005.

[0098] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a value network state adaptive reinforcement learning training program for large models. When the value network state adaptive reinforcement learning training program for large models is run by a processor, the steps of the method described above are executed, which will not be repeated here.

[0099] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A value network state adaptive reinforcement learning training method for large models, characterized in that, The method includes the following steps: Step S10: Interact with the training environment through the current Actor policy network to collect training samples and obtain training data, which includes state, action, reward and policy information; at the same time, analyze the policy changes, task performance and policy diversity during the training process to construct training state description information, and provide basic input for subsequent Critic state evaluation. Step S20: Construct a Critic state evaluation mechanism that integrates strategy, performance, and diversity, analyzes the update requirements of the value network in the current training phase, and dynamically decides whether to perform Critic training based on the evaluation results. Critic updates are triggered when the training state changes significantly; the current Critic training is skipped when the training state remains stable. Step S30: After completing the Critic update decision, the Actor policy network is optimized and trained using the value estimation results; when the Critic network has not been updated for several consecutive rounds, a forced update is automatically triggered to ensure the reliability of the value function training process.

2. The adaptive reinforcement learning training method for value network states in large models according to claim 1, characterized in that, Step S10 includes: Step S101: Based on the current Actor policy network, interact with the training environment and collect training samples through the Rollout process; in each training cycle, the Actor generates actions according to the current environment state and interacts with the environment to obtain corresponding feedback information. The feedback information includes the current state and the next state, the action information includes the currently executed action and the corresponding probability of the action, the reward information includes the immediate reward and the cumulative reward, and the policy information includes the current policy distribution and the historical policy distribution. Based on the feedback information, training trajectory data Trajectory is formed through continuous interaction, sampling is completed, and then the training trajectories are organized into training dataset D. wherein Trajectory = {(State1, Action1, Reward1), (State2, Action2, Reward2),… , (State t , Action t , Reward t )} Where t represents the length of the current training trajectory, State represents the current state, Action represents the current action being executed, and Reward represents the immediate reward; D= {State, Action, Reward, Return, Policy} Where State is the current state, Action is the current action being performed, Reward is the immediate reward, Return is the cumulative reward, and Policy is the policy information; Step S102: Construct policy change information based on the current policy distribution and the historical policy distribution to form a policy state description quantity PolicyState. The KL divergence between the current policy and the historical reference policy is used as an indicator of the degree of policy change. The policy change information is used to reflect whether the current policy has been significantly updated, whether the policy search space has shifted significantly, and whether there is a need for re-estimation of the value function in the current training stage. Step S103: Construct task performance information, perform real-time statistics on training tasks, and form a performance state description quantity PerformanceState to reflect the changes in model capabilities during the current training phase. The task performance information includes: task accuracy, average reward, success rate, and task completion quality indicators. Step S104: Perform statistical analysis on the policy output distribution, construct policy diversity information, and form a diversity state description quantity DiversityState to monitor the possible decline in exploration ability during reinforcement learning training. The policy diversity information includes policy entropy, historical average entropy, entropy change trend, and degree of diversity degradation. The policy entropy is used to measure the dispersion of the policy output distribution. Step S105: Construct a training state description vector based on the policy change information, task performance information, and policy diversity information: The training state description vector serves as the input for subsequent Critic state evaluation, used to dynamically determine the update requirements of the value network in the current training phase.

3. The adaptive reinforcement learning training method for value network states in large models according to claim 2, characterized in that, Step S20 includes: Step S201: Establish a training state model to describe the update requirements of the Critic network during reinforcement learning training; wherein, the current training state is defined as: in: Indicates the policy state, used to describe the current changes in the policy space; Indicates performance status, used to describe the model's task completion status during the current training phase; The diverse states are used to describe the current policy exploration capability; the policy state, performance state, and diverse states together constitute the Critic training state space, providing a unified state representation for subsequent value network update decisions. Step S202: After establishing the training state space, establish the policy state, performance state, and diversity state feature models. Step S203: Based on the policy state, performance state, and diversity state feature model, establish a comprehensive Critic state model to comprehensively evaluate the current Critic network training requirements. Step S204: Based on the Critic integrated state, establish a gated update model. Through a preset gated update mechanism, realize the transformation of the Critic network from a fixed update mode to a training state-driven update mode, so as to reduce the training overhead of the value network while ensuring the stability of the value function.

4. The adaptive reinforcement learning training method for value network states in large models according to claim 3, characterized in that, Step S202 includes: Step S2021, adopt strategy distribution difference As a descriptor of policy state, a policy state feature model is established to describe the degree of change in the current policy network relative to historical policies, including the policy distribution differences. Represented as: in, Indicates the current Actor policy. Indicates a reference strategy; Step S2022: Establish a performance state feature model to reflect the performance changes of the current model in the training task, wherein the performance state function... Represented as: in Accuracy is a performance evaluation metric for models during the current training phase. Step S2023: Establish a policy diversity feature model to reflect changes in policy exploration ability during reinforcement learning training; this involves first calculating the current policy entropy. And calculate its exponential moving average. Further define the state of strategy diversity : When strategy outputs become more concentrated and exploration capabilities decline... Gradually increase.

5. The adaptive reinforcement learning training method for value network states in large models according to claim 4, characterized in that, In step S203, the Critic comprehensive state function is defined as follows: in , , These represent the weight coefficients corresponding to the strategy state, performance state, and diversity state, respectively; Critic comprehensive state function It is used to comprehensively reflect the update needs of the Critic network under the current training state.

6. The adaptive reinforcement learning training method for value network states in large models according to claim 5, characterized in that, The update rule in step S204 includes the following steps: Step S2041, synthesize the Critic state function. With update threshold Perform a comparison; Step S2042, if the Critic comprehensive state function Greater than the update threshold If the condition is met, then Critic network training will be performed; otherwise, skip this round of Critic training and retain the current value network parameters to continue participating in policy optimization.

7. The adaptive reinforcement learning training method for value network states in large models according to claim 6, characterized in that, Step S30 includes: Step S301: Establish a Critic network feedback model to estimate the value function corresponding to the current state and provide value feedback information to the Actor network; Step S302: Optimize the strategy based on the feedback results of the Critic network feedback model and establish the Actor strategy optimization model. Step S303: Establish a continuous skip constraint model to avoid the value estimate deviating from the true return due to the Critic network not being updated for a long time.

8. The adaptive reinforcement learning training method for value network states in large models according to claim 7, characterized in that, Step S301 includes: The Critic value function is defined as follows: The system outputs the corresponding state value based on the current training state, which serves as an important basis for policy optimization. When the Critic performs an update, the latest value function is used in policy training. When the Critic skips the update, the value function from the most recent training iteration is used for value estimation to ensure the continuous and stable training process. Step S302 includes: The current policy is defined as follows: Where θ represents the Actor network parameters; further construct the advantage function: Where R represents the current training reward, and V(s) represents the Critic output value; Subsequently, the policy network parameters are updated according to the advantage function, so that the Actor continuously approaches the optimal policy; the entire Actor optimization process is always executed in every round, while the Critic dynamically decides whether to update according to the training state, thereby realizing collaborative training between the Actor and the Critic at different update frequencies. Step S303 includes: Define the number of consecutive skips as SkipCount, when Critic finishes updating: When Critic skips the update: Further set the maximum number of consecutive skips (SkipMax) when the condition is met. At this time, a Critic training is triggered, and the consecutive skip count is reinitialized.

9. A value network state adaptive reinforcement learning training system for large models, characterized in that, The system includes a memory, a processor, and a large-model-oriented value network state adaptive reinforcement learning training program stored in the memory, wherein the large-model-oriented value network state adaptive reinforcement learning training program is executed by the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a value network state adaptive reinforcement learning training program for large models, which, when run by a processor, performs the steps of the method as described in any one of claims 1 to 8.

Images