A reinforcement learning method adapted to suppress performance fluctuations

Abstract

The application provides a reinforcement learning method suitable for inhibiting performance fluctuation, a reinforcement learning intelligent agent containing an online policy network and a target policy network queue is constructed, a main experience pool and an evaluation experience pool are combined, and stable evolution and efficient convergence of a policy are realized. In an environment interaction process, the online policy network and the optimal policy in the target policy network queue are alternately sampled and obtain environment interaction information, and the environment interaction information is stored in the main experience pool. In a preset evaluation period, a parallel multi-environment rolling evaluation mechanism is used to obtain policy evaluation rewards, and the target policy network queue is dynamically updated or a policy rollback is performed based on the evaluation results. A double evaluation network is introduced, and an adversarial evaluation training mechanism based on double Q mean value and small double Q random sampling is used to inhibit value function overestimation and policy over-conservatism. Interaction data generated in the evaluation stage is stored in the evaluation experience pool, and the interaction data is used in combination with main experience pool data in the training process, so that the stability, robustness and convergence performance of the reinforcement learning training are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent control, and particularly to an improved reinforcement learning method for suppressing and optimizing evaluation performance fluctuations. This invention is applicable to and can be extended to mainstream reinforcement learning algorithms such as DDPG, TD3, and SAC, especially the hard update method for the target policy, which is applicable to all reinforcement learning methods. Background Technology

[0002] Reinforcement learning is a machine learning method that involves an agent continuously interacting with its environment and iteratively optimizing its policy based on feedback signals. It has been widely applied in fields such as continuous control, robot control, and complex simulation systems. In continuous action space tasks, deep reinforcement learning algorithms based on policy gradients have attracted significant attention from researchers due to their ability to directly handle high-dimensional continuous actions and thus possess high sample utilization efficiency. Representative algorithms include the Deep Deterministic Policy Gradient (DDPG) algorithm, the Double Delay Deep Deterministic Policy Gradient (TD3) algorithm, and the Soft Actor-Critic (SAC) algorithm.

[0003] The aforementioned algorithms typically employ a training framework combining an online policy network and a target network. By delaying or soft-updating the target network parameters, they mitigate the accumulation of function approximation errors and improve the stability of the training process. However, in actual training and performance evaluation, policy performance is often evaluated directly based on the online network or the soft-updated target network. Since the parameters of such networks continuously change during training, the evaluation process is susceptible to factors such as stochastic exploration policies, gradient estimation noise, and environmental randomness. This leads to significant fluctuations in the reward evaluation results during the training phase, thereby reducing the stability of the algorithm's convergence process and the reliability of performance evaluation.

[0004] Especially in the early stages of training or before the policy converges, drastic fluctuations in evaluation rewards can easily lead to misjudgments of policy performance, thereby affecting model saving, parameter updates, and decision control during training, reducing training stability and convergence reliability. Therefore, how to reduce the fluctuations in evaluation rewards during reinforcement learning training and improve the stability and reliability of evaluation results without increasing computational overhead remains a pressing problem to be solved in existing reinforcement learning algorithms such as DDPG, TD3, and SAC. Summary of the Invention

[0005] To address the issues of large fluctuations in evaluation rewards and insufficient training stability caused by the use of online policy networks or soft-updated target networks for evaluation during training in existing reinforcement learning algorithms such as DDPG, TD3, and SAC, this invention proposes a reinforcement learning method suitable for suppressing performance fluctuations. This method achieves smooth and stable evaluation rewards and reliable improvement in policy performance during training by introducing target policy network queue management and rollback and switching mechanisms.

[0006] See Figure 1 The reinforcement learning agent's environment interaction diagram shows that the agent consists of an online policy network, a target policy network queue, and an evaluation module. The policy model is responsible for predicting a set of actions based on the current state. The evaluation module assesses the future value of actions in the corresponding state, and temporal difference loss drives the parameter updates of the evaluation network. The policy and evaluation modules obtain information such as state and reward through interaction with the training environment. The experience pool stores interaction samples (state s, action a, reward r, new state s') for subsequent batch training, where (s, a, r, s') represents a single experience, [(s, a, r, s')] represents a batch of experiences, v represents the future value under the corresponding state and action, and a' is the new action predicted by the policy network.

[0007] The target policy network queue is initially empty, with each node... Record strategy network parameters Total cumulative return With the number of assessments The best one is placed at the front of the queue. The sorting method is determined based on the comprehensive score of the policy network, and the formula is: , ; Based on the comprehensive score, policy nodes are sorted and a target policy network queue is dynamically maintained. The best node is placed at the front of the queue. When a new node is added, the worst policy node is deleted according to the capacity K, and the node with the best comprehensive score is selected from the queue as the target policy network for experience collection. Adding a new node requires meeting certain conditions. R onlin > S max ,in R onlin For the latest evaluation of returns on online strategy networks, S max The optimal score in the target policy network queue.

[0008] Figure 1The online policy network (APN) generates actions based on the current state information, interacts with the training environment, and obtains state transitions and rewards. The target policy undergoes hard updates using parameters from the APN to output relatively stable action decisions, thereby reducing the uncertainty caused by frequent parameter changes during policy evaluation. After receiving the actions output by the APN, the training environment generates a corresponding experience (s, a, r, s') and stores it in the main experience pool. Simultaneously, the optimal target policy network also participates in action sampling, alternating with the APN through time-division multiplexing. Furthermore, the target policy periodically performs performance evaluation, and the resulting batch interaction experiences [(s, a, r, s')] are stored in the evaluation experience pool. By storing training and evaluation experiences separately, the implementation logic is simplified, while sample utilization is improved without increasing additional counting resources. Simultaneously, samples from the evaluation experience pool are merged with samples from the main experience pool and participate in the iterative training process of the policy network and evaluation network. This method suppresses fluctuations in environmental rewards, improving the stability and reliability of the reinforcement learning training process.

[0009] The above time-division multiplexing method alternates the action sampling method as follows: (1) Set the conditions for entering the evaluation cycle. P as follows: ,in T eval The hyperparameter represents the evaluation period. R train The total reward at the end of the current training cycle. R tar The latest evaluation return for the target network; (2) The formula for determining whether to use an online policy network or a target policy network for sampling actions is as follows: , t Number of training iterations t 0 is the starting step number, where β ∈[0,1], R max To maximize the total reward when the training environment cycle is completed, R tar The latest evaluation return for the target network is determined by deterministic sampling. R tar > R max This is a common occurrence; R onlin To evaluate the returns of online strategy networks, S max The optimal score in the target policy network queue; P 1 is the first condition for judgment.

[0010] (3) The above-mentioned online policy network and target policy network alternately perform action sampling using time-division multiplexing. This can be further enhanced by adding a time-division multiplexing method for alternating action sampling in sliding window mode, as shown in the following formula: P 2 is the second condition. T eval > L win > L tar , S win ~ Uniform (0, L win - L tar The formula expresses that during the training phase, when the conditions are met, each... L win The next iteration S win Execution begins at [location] L tar Sub-target network action sampling.

[0011] Specifically, the architecture diagram of this reinforcement learning is as follows: Figure 2 As shown in the figure, this diagram illustrates the overall process of a deep reinforcement learning algorithm based on the Actor-Critic architecture. It integrates a stochastic policy, a double-Q network, a target network, and an experience replay mechanism, exhibiting a complete and clearly hierarchical structure. The system mainly consists of a training environment, a policy module, an evaluation module, and an experience pool module. The double-Q network is the preferred method, outputting the corresponding future value for the same state and action pair. During the training of the policy and evaluation networks, an alternating evaluation mechanism based on the mean mode and the stochastic mode of double-Q is introduced. The double-Q minimum mode is a special case of the stochastic mode, allowing the double-Q network to exert adversarial constraints on the value function under different generation modes. This suppresses the systematic overestimation of the value function and the excessive conservatism during policy updates, thereby improving the stability and convergence performance of reinforcement learning training.

[0012] (1) First, regarding the training environment and policy module (Actor): The environment transitions to a new state based on the current state s and generates a reward r, forming an interaction sample (s, a, r, s'). The policy module includes an online policy network and a target policy network queue. The policy network outputs the parameters (μ, σ) of the action distribution, where σ can be a hyperparameter. Then, actions a are obtained by sampling actions through a normal distribution. The online policy network is responsible for interacting with the environment and updating gradients, while the target policy network remains relatively stable through a hard update mechanism and is used to generate the next action a'. In addition, the policy network also introduces a maximum value loss to constrain the policy update direction.

[0013] (2) Secondly, regarding the evaluation module (Critic), the preferred method is a dual-Q network structure. The online dual-Q network outputs Q1 and Q2 respectively, and the gradient is updated by calculating the mean squared error loss of the target evaluation network. The target dual-Q network adopts a soft update method, which is smoothly updated from the online network. In the value calculation process, the system uses the target network to evaluate the next state, and calculates the total future value V by using the smaller value or the mean. The dual-Q adversarial suppression of bias effectively alleviates the overestimation problem in traditional Q learning, thereby improving the stability and convergence performance of training.

[0014] The specific method for evaluating the adversarial training mode of the dual Q-network mentioned above is as follows: For the evaluation functions Q1(s,a,θ1) and Q2(s,a,θ2) of the two independent Q-networks, various Q-value fusion forms are defined as follows:

[0015] in δ ∈[0,1], can be fixed as a hyperparameter or a random floating-point number; during training, the training loss of the Q-network is used. L Q The value adaptively selects the valuation mode: Where ε is a hyperparameter, Q rand Minimum for approximating diversity Q value, Q critic This indicates the Q-value evaluation mode used when training the Q-network. Q acotr The Q-value evaluation mode used when training the policy network is represented. The above Q-value evaluates the training logic of the network and policy network in adversarial training between the mean mode and the random mode, which to a certain extent suppresses the problems of overestimation of the value function and over-conservatism of the policy, and improves the stability and convergence performance of reinforcement learning training.

[0016] Optionally, further... Q critic and Q acotr The formula for periodic switching is as follows:

[0017] in t This refers to the number of iterations during the training process. T F For example, a flip cycle of 1000. C F Number of flips C F < T F For example, 100.

[0018] In a continuous environment, the objective functionJ Q (θ) is:

[0019] in , Optionally, α' can sample multiple samples and then calculate the mean to accelerate convergence. θ The target evaluation network adopts a soft update approach:

[0020] Objective function in discrete environment J Q (θ) is: ,in

[0021] The policy network is updated in the following way: Objective function in continuous environment J π (φ) is: , in Optionally, during α-sampling, multiple samples can be sampled and then the average can be calculated to accelerate convergence.

[0022] Objective function in discrete environment J π (φ) is:

[0023] (3) Regarding the experience replay and sampling mechanism: The main experience pool and the evaluation experience pool are used to store the interaction data of the training and evaluation phases, respectively. When sampling, the main experience pool combines the "high reward priority" and "recent priority" strategies to make key experiences easier to select, thereby improving the efficiency of sample utilization. The sampled experiences are batched and merged before being fed into the policy network and the evaluation network for joint training. This offline experience replay mechanism not only increases the diversity of samples but also significantly improves the sample utilization rate, which is beneficial to the stability and convergence speed of training. Overall, this architecture integrates a variety of advanced reinforcement learning ideas such as stochastic policy modeling, double Q adversarial bias suppression, hard update stable training of the target network, and double experience pool replay. It is suitable for complex control and decision problems in discrete and continuous action spaces and has strong engineering practicality and theoretical integrity. The above evaluation experience pool experience collection method is as follows: under the condition of N evaluation environments running in parallel, the state S obtained by synchronously generating actions and executing interactions on each evaluation environment using the policy to be evaluated. t Action A t Reward R t New state S t+1 Termination state D t :

[0024] in d s For the state dimension, d α For the action dimension. When any evaluation environment terminates prematurely, retaining the last interaction information of that environment allows for the unified construction of a multi-environment evaluation experience structure, defining an environment survival indicator vector: For environments that have already been terminated The interaction data is corrected according to the following rules:

[0025] Where t * Indicates environment The last evaluation step that did not terminate; evaluation experience pool D eval The insertion operation is defined as follows: When all evaluation environments terminate, the average return of the parallel evaluation environments is output as the strategy evaluation result, thereby achieving a strategy evaluation that balances evaluation stability and evaluation experience integrity.

[0026] Batch experience is combined proportionally with master experience and evaluation experience. The merging method is as follows:

[0027] (a) Proportional consolidation: , where λ∈[0,1] is a fixed hyperparameter, or it can be a random floating-point number;

[0028] (b) Direct merger: .

[0029] (4) Figure 3 The policy network training flowchart illustrates the complete closed-loop training and evaluation process for the online policy network and target network hard updates. It covers key stages such as initialization, interactive sampling, policy updates, periodic evaluation, and target network management, reflecting a design philosophy that prioritizes both training stability and optimal performance. The overall process unfolds logically according to the sequence of "initialization—iterative training—evaluation and selection—network update—termination decision." The process begins with the initialization phase. After system startup, the training environment and main experience pool are created sequentially to support subsequent state transitions and data storage. Then, the online policy network is constructed as the core model that directly interacts with the environment and continuously learns. Next, the target network queue is initialized to store historical or high-performing policy networks. Finally, the environment is initially explored using a random policy, and a certain number of random experiences are sampled to fill the experience pool, alleviating the problems of sample scarcity and distribution bias in the early stages of training and providing basic data support for subsequent learning.

[0030] Once in the training phase, the process revolves around an iterative training loop. In each iteration, the system first determines whether to continue training; if the termination condition is met, the process ends and the final result is output. If training continues, the current policy is further evaluated. When the evaluation reward is better than the training reward, it indicates that the target network has stronger generalization ability at the current stage, and the system will use the target network for action sampling; otherwise, the online policy network is used for action sampling. This adaptive selection mechanism achieves a better balance between exploration and exploitation.

[0031] After action sampling is completed, the system interacts with the environment to generate new experience data and stores it in the main experience pool. Subsequently, batch sampling is performed from the main experience pool and the evaluation experience pool and merged into training batches for gradient updates to the online policy network. This process ensures the diversity and independence of training samples, thereby improving the stability and convergence efficiency of learning. Simultaneously, the experience data is also written to the evaluation experience pool, providing data support for subsequent policy evaluation.

[0032] When the training process reaches the preset evaluation period, the system triggers the policy evaluation mechanism. At this time, the online policy network is used for rollback evaluation, and the evaluation results are compared with the evaluation results of the target network. If the online network performs better than the target network, a hard update operation is performed on the target network, copying the current parameters of the online network into the target network, and recording the online network in the target network list as a potential optimal policy candidate.

[0033] After the target network is updated, the system also introduces a performance rollback protection mechanism. By determining whether the current reward is lower than a preset percentage of the historical best reward, indicating a significant performance degradation, the system immediately rolls back to the historical best target network to prevent performance collapse due to policy instability or gradient oscillations during training. The policy network rollback mechanism is implemented as follows: If the overall score S of the current target policy network c The best overall score S among the other networks in the queue is lower than that of the others. max When the preset ratio threshold is reached, a rollback operation is performed on the current target policy network, that is, the optimal target policy network is adopted as the current target policy network. The judgment formula is as follows: , where γ is a hyperparameter, γ∈[0,1], and P3 is the third judgment condition.

[0034] (5) Pseudocode of core algorithm logic: Algorithm 1 is the pseudocode of the basic method, and Algorithm 2 is the pseudocode of the entire process:

[0035] As shown in the pseudocode of Algorithm 2, the policy network and the evaluation network adopt a better delayed update method, and the number of delayed update steps is set. L d , the evaluation network is trained L d For d rounds of iteration, the corresponding batch of experiences B[j] is recorded, where 0 ≤ j < d, and then it is the turn of the policy network to be trained, that is, the policy network is trained after a delay L d steps; symmetrically, based on the recorded batch of experiences B[j], the policy network is also trained L d After d rounds of iteration, the evaluation network starts to be trained again, and so on; obviously, the evaluation network and the policy network satisfy the delayed update conditions for each other. Therefore, the order of the policy network and the evaluation network in terms of training can be interchanged logically.

[0036] Implementing the solution of the present invention, by introducing the target policy network queue management and the fallback and switching mechanism, the smooth stability of the evaluation reward and the reliable improvement of the policy performance are achieved during the training process; thus overcoming the problems in the prior art that reinforcement learning algorithms such as DDPG, TD3, and SAC use an online policy network or a soft-updated target network for evaluation during the training process, which easily leads to large fluctuations in the evaluation reward and insufficient training stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings here are incorporated into the specification and form a part of this specification, showing the embodiments in line with this application, and are used together with the specification to explain the principles of this application. In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other similar implementation methods can also be obtained based on the principles of these drawings.

[0038] Figure 1 is the intelligent agent environment interaction diagram of the present invention; Figure 2 is the algorithm architecture diagram of the present invention; Figure 3 is the flow chart of the policy network training of the present invention; Figure 4 is the structure diagram of the evaluation network in an embodiment of the present invention; Figure 5 is the structure diagram of the policy network in an embodiment of the present invention; Figure 6a is the performance comparison diagram of each algorithm in the HalfCheetah environment; Figure 6b is the performance comparison diagram of each algorithm in the Hopper environment; Figure 6c A performance comparison chart of each algorithm in the Walker2d environment; Figure 6d A performance comparison chart of each algorithm on the Humanoid environment; Figure 6e A performance comparison chart of each algorithm on the Ant environment; Detailed Implementation

[0039] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some preferred embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Example

[0042] This embodiment has broad applicability to machine learning algorithms, especially demonstrating significant advantages in deep reinforcement learning scenarios. This embodiment uses a classic continuous control task as an example to illustrate the optimal application process of this invention in agent policy optimization. Experimental environment: Hardware consists of an Intel Xeon Platinum 8558P CPU and an NVIDIA L40 GPU; the software platform is the PyTorch reinforcement learning framework and the OpenAI Gym simulation platform; the simulation environment is the "Hopper-v5", "HalfCheetah-v5", "Walker2d-v5", "Humanoid-v5", and "Ant-v5" environments within OpenAI Gym, specifically including the following: In this embodiment, firstly according to Figure 1 Reinforcement learning agent environment interaction graph and Figure 2The algorithm architecture diagram constructs a reinforcement learning algorithm architecture based on the Actor-Critic method. Then, according to... Figure 4 Evaluation of network structure diagram and Figure 5 Create a network structure for the policy network structure diagram.

[0043] Preferred evaluation network structure, such as Figure 4 As shown, the evaluation network inputs are state information s with dimensions (B, Ds) and action information a with dimensions (B, Da), where B is the batch size, Ds is the state dimension, and Da is the action dimension. Initially, the state and action information are concatenated and cated to obtain x(B, Ds + Da) as input. This is then split into two paths to form a dual-Q network structure. The input dimension of the first-hand (FC) network is Ds + Da, and the output dimension is 256. Both paths pass through their respective FCs, where each FC has an input and output dimension of 256. After ReLU activation, they pass through the next FC layer, where the input dimension is 256 and the output dimension is 1. The final output is the two values ​​Q1 and Q2, both with dimensions (B, 1). Finally, the objective function in the continuous environment is... J Q (θ) The loss is calculated by updating the online evaluation network through gradient, and the target evaluation network adopts a soft update method.

[0044] Preferred strategy network structure such as Figure 5 As shown, The input is state information s with dimensions (B, Ds). The initial feature extraction consists of two fully connected (FC) layers and a ReLU activation layer. The first FC layer has an input dimension of Ds and an output dimension of 256, while the second FC layer has both an input and output dimension of 256. After feature extraction, the output FC layer extracts multiple probability distribution mean features. This embodiment... The output is the mean parameter dimension (B, Da), where Da is the action dimension. Finally, the objective function in a continuous environment is used... J π (φ) The loss is calculated by updating the online policy network through gradient updates; the target policy network queue is completed through hard updates.

[0045] In this embodiment, a master experience pool and an evaluation experience pool are created. The pseudocode implementation of the entire algorithm is shown in Algorithm 1. During training, the optimal model can be obtained by saving the policy network parameters at the peak based on the total reward of the interaction with the environment. The specific content of the algorithm is shown in the table below:

[0046] The experimental comparison results of this embodiment are as follows: Figures 6a-6eThe diagram shows a comparison of the training performance of different reinforcement learning algorithms. The horizontal axis represents the number of training steps, the vertical axis represents the cumulative environmental reward, the curve represents the average performance of 8 different random seed experiments, and the shaded area reflects the fluctuation range of the results (such as standard deviation or confidence interval).

[0047] Overall, all algorithms showed rapid increases in rewards during the initial training phase, indicating they could quickly learn basic motion strategies. The growth rate then gradually slowed, entering a stable improvement phase. Among them, simtd3e, which incorporates twice the evaluation experience in its training batches, demonstrated the best convergence speed and final performance, significantly outperforming other methods for most training phases and maintaining continuous improvement in the later stages, indicating its advantages in sample efficiency and policy optimization stability. simtd3 followed, outperforming the traditional TD3 and SAC, demonstrating the superior performance of the improved TD3 series methods in this continuous control task. In contrast, SAC and TD3 showed lower final rewards and relatively larger fluctuation ranges, reflecting slightly inferior convergence quality and stability in this environment.

[0048] Overall, simtd3 shows significant advantages in all environments except Ant, particularly in Hopper and HalfCheetah. simtd3e exhibits significantly faster early convergence speeds in all environments, outperforming simtd3 in all except the Humanoid environment. These results demonstrate that the simtd3 series methods with improved strategies achieve higher and more stable cumulative rewards in HalfCheetah-v5, validating their effectiveness in complex continuous control tasks.

[0049] The table below shows the average drawdown of the performance data fluctuations of each algorithm. The data comes from the performance fluctuations of each of the eight different random seeds trained one million times. The smaller the data, the smaller the fluctuation. According to the data, the average drawdown of this algorithm in all environments is significantly smaller than that of the SAC and TD3 algorithms, which proves the effectiveness of this algorithm in suppressing performance fluctuations.

[0050]

[0051] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning method suitable for suppressing performance fluctuations, characterized in that, Includes the following steps: S1. Construct a reinforcement learning agent that interacts with the environment and an experience pool; the reinforcement learning agent includes a policy module and at least one evaluation module; the policy module includes an online policy network and a target policy network queue; the policy module is used to output the mean of the action distribution, and the standard deviation of the action distribution is given by a pre-set hyperparameter or output by the policy module; the target policy network queue is used to record a finite number of policy network parameters and their evaluation performance data; the evaluation module is used to output the future value of the state; the experience pool includes a main experience pool. S2. Execute training. During training, in the process of interacting with the environment, the action is sampled through the strategy module according to the current environment state, and the action is executed to obtain the environmental feedback and its interaction information. The environmental feedback and corresponding interaction information are stored in the main experience pool. The online policy network and the target policy network alternately perform action sampling in a time-division multiplexing manner; S3. When the training reaches a preset evaluation period, the online policy network performs rolling evaluation in parallel multi-environment to obtain the latest evaluation reward. When the latest evaluation reward is greater than the optimal score in the target policy network queue, the parameters of the online policy network are copied as a whole and inserted into the target policy network queue, and the corresponding evaluation reward value is recorded. S4. Based on the evaluation reward, the following steps are performed: when the latest evaluation reward is less than or equal to the optimal score in the current target policy network queue, the optimal target policy network is used to perform rolling evaluation in the environment, and the evaluation reward is used as the evaluation result of the current evaluation period. At the same time, the cumulative reward value and evaluation count of the corresponding policy network in the target policy network queue are updated. When the latest evaluation reward is less than the preset proportion threshold of the optimal score, a policy rollback operation is performed.

2. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that, The performance evaluation of the target policy network queue includes: S5. Set the target policy network queue capacity to K, initially empty; each node... Used to record policy network parameters Total cumulative return With evaluation count The formula for defining the comprehensive scoring of the policy network is as follows: ; Based on the comprehensive score, the policy nodes are sorted and the target policy network queue is dynamically maintained, with the optimal node at the front of the queue. Add new nodes; while adding new nodes, remove the worst policy node according to capacity K, and select the node with the best comprehensive score from the queue as the target policy network to participate in the collection of experience. in For the latest evaluation of returns on online strategy networks, When the target policy is the best score in the network queue.

3. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that, The online policy network and the target policy network alternately perform action sampling in a time-division multiplexing manner, including: S6. Determine whether to enter the evaluation cycle. The conditions P for entering the evaluation cycle are as follows: ,in The hyperparameter represents the evaluation period. The total reward at the end of the current training cycle. The latest evaluation return for the target network; S7. Determine whether the sampling action is performed using an online policy network or a target policy network. The determination formula is as follows: , Let be the starting step number, where β∈[0,1]. The maximum total reward recorded when the training environment cycle is completed. The latest evaluation return for the target network; To evaluate the returns of online strategy networks, The optimal score in the target policy network queue; S8. The online policy network and the target policy network are time-division multiplexed to perform action sampling alternately.

4. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 3, characterized in that, The online policy network and the target policy network alternately perform action sampling in a time-division multiplexing manner, and also include: In sliding window mode, action sampling is performed alternately using time-division multiplexing, as shown in the following formula: The formula expresses that during the training phase, when the conditions are met, each The next iteration Execution begins at [location] Sub-target network action sampling.

5. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that, The policy network rollback operation includes: S9. If the overall score of the current target policy network is... The best overall score among the other networks in the queue was lower. When the preset ratio threshold is reached, a rollback operation is performed on the current target policy network, and the optimal target policy network is adopted as the current target policy network. The judgment formula is as follows: , in For superparameters, ∈[0,1].

6. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that, Also includes: S10. An evaluation module is constructed using two independent networks, namely a dual-Q structure, which outputs the corresponding future value for the same state and action pair. During the training of the policy module and the evaluation module, the mean mode and the random mode based on the double Q are evaluated alternately. The double Q minimum mode is a special case of the random mode, so that the double Q network applies adversarial constraints to the value function under different generation modes. S11. Construct an evaluation function containing two independent Q-networks. And define several Q-value fusion forms as follows: , , , Where s represents the state and a represents the action. The evaluation functions are respectively parameter; ∈[0,1], representing hyperparameters or random floating-point numbers; during training, the training loss is calculated based on the Q-network training loss. The value adaptively selects the valuation mode: ,in For superparameters, This indicates the Q-value evaluation mode used when training the Q-network. This indicates the Q-value evaluation mode used when training the policy network.

7. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 6, characterized in that, Also includes: S12, regarding the above and stated The formula for periodic adjustments is as follows: in This refers to the number of iterations during the training process. For the flip cycle, This represents the number of flips.

8. A reinforcement learning method suitable for suppressing performance fluctuations according to claim 6, characterized in that, It also includes: S13, updating the evaluation module. The evaluation module is updated in the following way: In a continuous environment, the objective function for: ,in , The target evaluation module adopts a soft update approach: The During sampling, one or more samples are sampled. When using multiple samples, the mean is calculated to accelerate convergence. If the objective function in a discrete environment is ,in .

9. A reinforcement learning method suitable for suppressing performance fluctuations according to claim 6, characterized in that: It also includes: S14, the update strategy module, which is updated in the following way: In a continuous environment, the objective function for: ,in During α-sampling, one or more samples are sampled, and when multiple samples are used, the average is calculated to accelerate convergence; In a discrete environment, the objective function is... for: .

10. A reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that: The experience pool also includes an evaluation experience pool; the steps also include: S15. Under the condition of multiple evaluation environments running in parallel, actions are synchronously generated and interactions are executed in each evaluation environment using the strategy to be evaluated, and the rewards of each environment are gradually accumulated to complete the evaluation; the reward information and corresponding interaction data generated by the rolling test of each environment during the evaluation period are stored in the evaluation experience pool; during the training phase, the training batch experience comes from... D main Main experience pool and evaluation experience pool D eval And supports any of the following methods: (1) merge online experience batches from the main experience pool and evaluation experience batches from the evaluation experience pool according to a preset ratio; (2) directly connect and merge online experience batches from the main experience pool and evaluation experience batches from the evaluation experience pool. The method for collecting evaluation experience is as follows: S16. Under the condition that N evaluation environments are running in parallel, use the strategy to be evaluated to synchronously generate actions on each evaluation environment and execute the interaction to obtain the state S. t Action A t Reward R t New state S t+1 Termination state D t : , Where d s For the state dimension, d α The action dimension is N, where N is a natural number. When any assessment environment terminates prematurely, the last interaction information of that environment is retained, and a unified multi-environment assessment experience is constructed, defining an environment survival indicator vector: For environments that have already been terminated The interaction data is corrected according to the following rules: Where t * Indicates environment The last evaluation step that did not terminate; evaluation experience pool D eval The insertion operation is defined as follows: Once all evaluation environments have terminated, the average return of the parallel evaluation environments is output as the strategy evaluation result, thereby achieving a strategy evaluation that balances evaluation stability and the integrity of evaluation experience.

11. A reinforcement learning method suitable for suppressing performance fluctuations according to claim 10, characterized in that, Also includes: S17. Training batch experience is proportionally merged with master experience and evaluation experience, and the merging method is as follows: (1) By merger ratio: , where λ∈[0,1] is a fixed hyperparameter or a random floating-point number; ; (2) Direct merger: .

12. The reinforcement learning method suitable for suppressing performance fluctuations according to claim 1, characterized in that: Step S2 further includes: employing a better delayed update method for the strategy module and the evaluation module. S18. Set the number of steps for delayed update L d , and train the evaluation module L d Record the corresponding batch of experiences B[j] in each round of iteration, where 0 ≤ j < d; then it's the turn of the policy network to be trained, that is, after a delay L d steps, train the policy module; the policy module is also trained based on the recorded batch of experiences B[j] L d After each round of iteration, start training the evaluation module again; iterate in this way; the evaluation module and the policy module mutually satisfy the conditions for delayed update; the order of the policy network and the evaluation module in terms of training logic can be interchanged.

13. A reinforcement learning method suitable for suppressing performance fluctuations according to any one of claims 1 to 12, characterized in that: The method is applicable to reinforcement learning tasks in continuous or discrete action spaces and policy training in simulation environments.

Images