An intelligent guidance method based on proximal strategy optimization

Abstract

This invention discloses an intelligent guidance method based on proximal strategy optimization, comprising: constructing a guidance optimization objective function; wherein the optimization objectives include minimizing miss distance, energy consumption during guidance, and time; constructing an intelligent guidance network model based on proximal strategy optimization; and training the intelligent guidance network based on proximal strategy optimization. Compared with traditional proportional guidance methods, this method allows for adaptive selection of guidance rate and exhibits stronger game performance in terms of time and hit performance.

Description

Technical Field

[0001] The present invention relates to the field of intelligent guidance technology, and in particular to an intelligent guidance method based on proximal strategy optimization. Background Art

[0002] The core of a guidance system lies in the guidance law. The choice of guidance method directly impacts the missile's accuracy and destructive power on its target. Guidance method design requires comprehensive consideration of multiple factors, including missile performance, operational environment, technical proficiency, and tactical application. During missile flight, the guidance law calculates the appropriate control variables based on target position and situation information acquired through radar or remote communications, combined with the missile's altitude, speed, and air resistance.

[0003] Classical guidance laws offer the advantage of reliability, but they are overly simple, lack intelligence, and often have fixed coefficients, limiting performance. Compared to traditional guidance laws, intelligent guidance laws offer greater efficiency, flexibility, and precision, enabling more complex decision support.

[0004] Therefore, technical personnel in this field are committed to developing an intelligent guidance method based on proximal strategy optimization, combining the proximal strategy optimization method with the traditional guidance method proportional guidance method, optimizing the weapon guidance method, and providing theoretical and technical support for improving the accuracy and intelligence level of the weapon guidance system. Summary of the Invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the traditional guidance method based on proportional guidance has the problems of weak flexibility in the selection of guidance control law and poor game confrontation ability.

[0006] To achieve the above object, the present invention provides an intelligent guidance method based on proximal strategy optimization, comprising the following steps:

[0007] Step 1: constructing a guidance optimization objective function; wherein the optimization objectives include miss distance, energy consumption during guidance, and time minimization;

[0008] Step 2: Construct an intelligent guidance network model based on proximal strategy optimization;

[0009] Step 3: Intelligent guidance network training based on proximal strategy optimization.

[0010] Furthermore, the step 1 specifically includes the following steps:

[0011] Step 1.1. Set a function to determine the end of pursuit: set a maximum miss distance. When the distance between the projectile and the target is less than the maximum miss distance, it is judged that the mission is successful and the pursuit ends. When the distance is greater than the maximum miss distance or the time step is greater than the maximum time, the mission fails and the pursuit is also considered to have ended.

[0012] Step 1.2: Design the reward for minimizing the amount of misses;

[0013] Step 1.3: Design the reward for minimizing the energy consumption of the guidance process;

[0014] Step 1.4: Minimize time reward design.

[0015] Furthermore, the miss amount reward in step b 1.2 is defined as: whenever the pursuit ends:

[0016] r_dist=-ln(r 2 +ε1),

[0017] Among them, ε1 is a very small positive number, and r is the distance between the projectile and the target.

[0018] Furthermore, the step 1.2 also includes setting a function for dynamically reducing the maximum miss distance.

[0019] Furthermore, the energy consumption reward of the guidance process in step 1.3 is defined as:

[0020]

[0021] where t f Indicates the current total time, a is the acceleration of the projectile, V is the velocity of the projectile, is the projectile lead angle.

[0022] Furthermore, the time reward in step 1.4 is defined as:

[0023]

[0024] Among them, α is a positive adjustment coefficient.

[0025] Furthermore, the step 2 specifically includes the following steps:

[0026] Step 2.1: Modeling the relationship between proportional guidance action and environment:

[0027] The guidance method optimization process is considered as a Markov decision process. The missile is taken as the intelligent agent, the target's maneuver trajectory is set in advance as part of the environment, and the state, action, and reward parameters are set for the intelligent agent, which ultimately form the environment.

[0028] Step 2.2, action and state space design:

[0029] The range of the amplification coefficient is used as the action boundary value in the continuous action space, and the policy network of the continuous action space is adopted;

[0030] The design of the state space reflects the state of the intelligent agent. It can reflect the air combat situation information during the guidance method optimization process, and thus evaluate the pros and cons of the situation of the missile and target in real time. It is also used to provide quantitative evaluation and reward feedback in the reward function design.

[0031] Furthermore, the step 2.1 specifically includes the steps of:

[0032] Step 2.1.1. Set up the agent class for the initialization and behavior of the projectile and target: set the initial position and velocity properties of the projectile and target; also set the motion function, selected from the range scale factor;

[0033] Step 2.1.2: Set up the environment class to create an environment that facilitates the movement and interaction of the agent:

[0034] Initialize various parameters as well as action space and state space, and determine the time step;

[0035] Define the reset function required for each round of loop to initialize the missile and target, state, and time step;

[0036] Setting a reward function and other functions, wherein the other functions include determining whether a hit is made and setting a random number seed;

[0037] Update the target and projectile parameters based on the passed action, and return the new status, reward, completion status, and other fields.

[0038] Furthermore, the design of the state space in step 2.2 includes the following state variables:

[0039] The distance between the projectile and the target, the rate of change of distance, the projectile lead angle, the projectile entry angle, the difference between the target direction angle and the projectile direction angle, the relative height between the target and the projectile, the relative speed between the target and the projectile, the height and speed of the projectile.

[0040] Furthermore, the step 3 specifically includes the following steps:

[0041] Step 3.1. Design a network and loss function based on proximal strategy optimization:

[0042] Based on the proximal policy optimization algorithm, training is performed on the continuous action space. The proximal policy optimization network includes the Actor network and the Critic network.

[0043] The Actor network includes an input layer, a hidden layer, and an output layer. The Actor network is responsible for action selection. The output is obtained by adding the mean of the predicted action to the noise multiplied by the variance, and the actual action is obtained by clipping according to the action range.

[0044] The Actor network objective function is as follows:

[0045]

[0046] In the previous part, r t (θ) is the strategy ratio, is an estimate of the advantage function, used to evaluate the advantage of taking a specific action; in the latter part, ε is the clipping rate, used to calculate the loss ratio after clipping; the loss functions take their minimum values, then take the average to calculate the average loss, and finally backpropagate to calculate the gradient;

[0047] The critic network includes an input layer, a hidden layer, and an output layer. The critic network is responsible for evaluating the value of the state-action pair. The critic network loss function is the mean square error between the state-action pair value and the state-action value in the experience area. The gradient is also calculated by backpropagation.

[0048] The proximal policy optimization network uses the gradient descent method in its training function to update the error functions of the Actor network and the Critic network respectively;

[0049] Step 3.2: Complete the guidance network training based on the reinforcement learning architecture:

[0050] After setting the modules to be imported, build the main function training framework; before importing the main function, set up the Agent and Env class modules, the proximal strategy optimization algorithm setting module, and the buffer class setting module;

[0051] Initialize the Agent and Env instances, initialize the GPU device, various parameters, initialization strategy and experience replay pool; query whether the guidance network has past weights saved, if so, import and train them, otherwise initialize the model network;

[0052] Enter the loop, for each round, reset the environment to get the state, initialize the time step, reward, completion, and storage array;

[0053] Enter the sub-loop:

[0054] Call the action selection function to obtain discrete action probabilities, continuous parameters, etc. to obtain actions;

[0055] Call the function to get the value of the state-action pair, and get the next state and reward through the environment's step function; add the state and reward to the buffer;

[0056] Update time steps, states, etc., and store trajectories;

[0057] If the task is completed or the update conditions are met: end and call the buffer, and update the visualization results;

[0058] If the update conditions are met: the loss value of the training buffer is calculated empirically and the buffer is reset;

[0059] Check the sub-loop condition: if the task is not completed or the time is not up, return to the sub-loop; otherwise, store the reward at this time;

[0060] Check loop condition: whether all rounds are trained, if yes, draw reward, otherwise continue training.

[0061] The beneficial effects of the present invention are that, compared with the traditional proportional guidance method, the guidance rate can be adaptively selected, and the game performance reflected in time and hit performance is stronger.

[0062] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of the method steps of a preferred embodiment of the present invention;

[0064] Figure 2 is a flow chart of the intelligent agent and environment settings of a preferred embodiment of the present invention;

[0065] Figure 3 It is a geometric relationship diagram of the tracking process of a preferred embodiment of the present invention;

[0066] Figure 4 1 is a PPO network function structure diagram of a preferred embodiment of the present invention;

[0067] Figure 5 This is a flow chart of the training main loop function of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following describes preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0069] In response to the problems of limited flexibility and poor game confrontation ability in the selection of guidance control laws in traditional guidance methods based on proportional guidance, this application uses the Proximal Policy Optimization (PPO) algorithm to learn and optimize the proportional coefficient; at the same time, it optimizes multiple objectives combining time performance, miss distance and energy consumption during the missile shooting process; optimizes the fixed coefficient and variable coefficient separately, and gives a relatively better variable coefficient distribution.

[0070] This application uses a proximal policy optimization algorithm to learn and optimize the proportional coefficient. Within the reinforcement learning training framework, given a value function, an intelligent guidance network is derived. Compared to traditional proportional guidance methods, the guidance rate can be adaptively selected, and the game performance in terms of time and hit performance is stronger.

[0071] The significance of this application lies in the research on airborne weapon guidance based on deep reinforcement learning, the simulation and optimization of the aircraft guidance method during airborne weapon combat, and the optimization of the traditional proportional guidance law based on several indicators such as miss distance, overload energy consumption, and hit time. In view of the continuous action space of the guidance coefficient in the proportional guidance law, based on the problems of weak flexibility and poor game confrontation ability of the guidance method corresponding to the traditional guidance control law, the dynamic model of the airborne weapon is constructed to improve the optimization dimensions such as miss distance, time, maneuverability, energy consumption, and the reward sparsity problem of the current airborne weapon guidance method. On this basis, the guidance coefficient is represented as a continuous choice. Further, under the PPO algorithm of the actor-critic architecture, a method for the continuous action space is made, thereby optimizing the guidance performance of the traditional proportional guidance.

[0072] like Figure 1 As shown, the steps of the intelligent guidance method based on proximal strategy optimization of this application are as follows:

[0073] Step 1: Construct guidance optimization objective function;

[0074] Step 2: Construct an intelligent guidance network model based on proximal strategy optimization;

[0075] Step 3: Intelligent guidance network training based on proximal strategy optimization.

[0076] The specific steps are as follows:

[0077] Step 1: Constructing the guided optimization objective function

[0078] Step (1) Reward function design

[0079] The optimization goal usually determines the design of the reward function in reinforcement learning. Considering the need to intercept the target as quickly as possible while ensuring the miss rate, it is necessary to combine multiple rewards. At the same time, in the specific implementation, additional parameter adjustments are required to prioritize the important goals and thus achieve a balance between achieving all goals simultaneously.

[0080] To achieve the optimization goals of minimizing miss distance, guidance process energy consumption, and time, different reward functions are set here. First, a function is needed to determine the end of the pursuit process: a maximum miss distance is set. When the distance between the missile and the target is less than the maximum miss distance, the mission is considered successful and the pursuit ends. When the distance is greater than a certain maximum value or the time step is greater than the maximum time, the mission fails and the pursuit is considered over.

[0081] Step (2) Design of reward for minimizing miss amount

[0082] The miss bonus is defined as:

[0083] r_dist=-ln(r 2 +ε1)

[0084] Here, ε1 is a very small positive number, r_dist is the miss distance advantage reward, and r is the distance between the projectile and the target. To increase the reward faster as the miss distance decreases, a negative logarithmic function and the square of the distance are used. This defaults to 0 when the pursuit process is not yet complete.

[0085] At the same time, in order to minimize the miss amount, a function that dynamically reduces the maximum miss amount is set in order to minimize the maximum miss amount: when the training result reaches a certain accuracy, the miss amount will automatically decrease.

[0086] Since the miss-distance-related reward is a result-oriented reward and is only fed back at the end of the task, the agent will not receive the due reward for the quality of a certain action during the task. The sparse reward problem describes this situation.

[0087] Step (3) Design of reward to minimize energy consumption during guidance process

[0088] In order to prevent the problem of sparse rewards, other rewards need to be set to provide real-time feedback during the task.

[0089] Since the choice of the proportional coefficient also affects the energy, and the energy consumption is related to the process, the energy consumption reward function is set accordingly.

[0090]

[0091] where t fIndicates the current total time, a is the acceleration of the projectile, V is the velocity of the projectile, is the projectile lead angle. To represent the energy consumption, the square of the projectile acceleration is integrated with respect to time and divided by the velocity, total time, and projectile lead angle to eliminate the influence of these parameters.

[0092] By combining the miss distance reward and the energy consumption reward and normalizing them using the mean and variance obtained through training, the total reward function is made smooth and continuous. The reward function is designed to minimize both the miss distance and the total energy consumption.

[0093] Step (4) Minimize time reward design

[0094] For time, a fixed miss distance and energy consumption can be used, and a separate optimization method can be adopted to intercept the target as quickly as possible while ensuring the miss distance. Therefore, time rewards and penalties can be set. Considering the necessity of early pursuit time, nonlinear time rewards and penalties can be adopted. The longer the time, the greater the penalty. For example,

[0095]

[0096] Here α is a positive, small adjustment coefficient, t f For the current total time, use an exponential function to make the penalty greater the longer the time.

[0097] Step 2: Build an intelligent guidance network model based on proximal strategy optimization

[0098] Step (1) Modeling the relationship between proportional guidance action and environment

[0099] Because reinforcement learning models Markov processes, the guidance process is considered a Markov process. The missile is considered the agent. For simplicity, the target's maneuver trajectory can be pre-determined and thus considered part of the environment. Parameters such as state, action, and reward are then set for the agent, ultimately forming the environment.

[0100] like Figure 3 As shown, according to the Markov process, the relevant content is modeled as follows:

[0101] First, set up the Agent class to initialize and behave the projectile and target: set their initial properties such as position and velocity; also set the "movement" function, selecting from a range of proportional coefficients.

[0102] Set up the Environment (Env) class to create an environment that facilitates the agent's movement and interaction. First, initialize various parameters, the action space, and the state space, and determine the time step. A too large time step will increase the chance of missing the target, while a too small time step will increase the difficulty of training. Next, define the "reset" function required for each episode: this is used to initialize the missile and target, the state, the time step, and so on.

[0103] Set up the reward function and other functions, such as determining whether to hit or not and setting the random number seed.

[0104] Update the target and projectile parameters based on the passed action, and return the new status, reward, completion status, and other fields.

[0105] Step (2) Action and state space design

[0106] The maneuvering process of a missile is typically composed of rudder control commands. However, this application aims to train the guidance coefficient. Therefore, a proportional coefficient selected within a specific range is used as the action, and a policy network with a continuous action space is adopted. Considering that variable-coefficient proportional guidance has more maneuvering space, the range of the proportional coefficient can be appropriately expanded to serve as the action boundary value in the continuous action space.

[0107] The design of the state space needs to reflect the state of the intelligent agent. During the guidance process, it is best to reflect the air combat situation information, thereby evaluating the pros and cons of the current situation of the missile and target pursuit problem in real time, and at the same time providing quantitative evaluation and reward feedback in the reward function design.

[0108] Since this information is relevant to the tracking process, e.g. Figure 4 As shown, according to the tracking process, the state variables are: the distance r between the projectile M and the target T, the rate of change of the distance q is the line of sight (LOS) angle, the projectile lead angle (the difference between the projectile direction angle σ and the line of sight angle q) Projectile entry angle (target direction angle σ T and the difference between the sight angle q) Target direction angle σ T The difference between the target and the projectile direction angle σ, the relative height Δh between the target and the projectile, the relative speed (target speed V T and the difference between the projectile velocity V), as well as the height and speed of the projectile.

[0109] Step 3: Intelligent Guidance Network Training Based on Proximal Strategy Optimization

[0110] Step (1) Design a PPO-based network and loss function

[0111] like Figure 5 As shown in the figure, the training is performed based on the PPO algorithm for continuous action space, including an actor network and a critic network.

[0112] The Actor Network consists of three layers: an input layer, a hidden layer, and an output layer. The network input is a 9-dimensional state vector, the hidden layer has a dimension of 256, and the output is a one-dimensional continuous action. This network is responsible for action selection. It outputs the predicted action by adding the mean, noise, and variance to the predicted action. The actual action is then clipped based on the action range.

[0113] Its objective function is as follows:

[0114]

[0115] In the previous part, r t (θ) is the strategy ratio, is an estimate of the advantage function, used to assess the advantage of taking a particular action. In the latter part, ε is the clipping rate, which is used to calculate the loss ratio after clipping. The loss functions are minimized and averaged to calculate the average loss, and the gradient is then backpropagated.

[0116] The Critic network also has three layers: input, hidden, and output. Its input is the state vector plus the action, a total of 10 dimensions. Its output is a one-dimensional value function, responsible for evaluating the value of the state-action pair. Its loss function is the mean squared error between the state-action value and the state-action value in the experience region. Backpropagation is also used to calculate the gradient.

[0117] For the PPO total network, in its training function, the gradient descent method is used to update the error functions of the two networks separately.

[0118] Step (2) Complete the guidance network training based on the reinforcement learning architecture

[0119] like Figure 5 As shown in the figure, after setting the modules to be imported, the main function training framework can be built. The main training function needs to import the previously set Agent and Env class modules, PPO algorithm setting module, and buffer class setting module.

[0120] First, you need to initialize the Agent and Env instances, initialize the Graphics Processing Unit (GPU), various parameters, initialize the strategy, and the experience replay pool. You also need to check whether the model's network has saved past weights. If so, import and train them; otherwise, initialize the model network. Next, enter a loop. For each episode, reset the environment to obtain the state, initialize the time step, reward, completion status, storage array, etc.

[0121] Enter the sub-loop:

[0122] Call the action selection function to obtain discrete action probabilities, continuous parameters, and other parameters to obtain the action. Call the function to obtain the value of the state-action pair, and use the environment's step function to obtain the next state and reward. Add the state, reward, and other parameters to the buffer. Update the time step, state, and other parameters, and store the trajectory.

[0123] If the task is completed or the update conditions are met: end and call the buffer, and update the visualization results.

[0124] If the update conditions are met: calculate the loss value from the training buffer experience and reset the buffer.

[0125] Check the sub-loop condition: If the task is not completed or the time is not up, return to the sub-loop; otherwise, store the reward at this time. Check the loop condition: whether the episode is fully trained, if so, draw the reward, otherwise continue training.

[0126] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An intelligent guidance method based on proximal strategy optimization, characterized in that: The following steps are involved: Step 1: Constructing a guidance optimization objective function; wherein the optimization objectives include miss distance, energy consumption during guidance, and time minimization; Step 1 specifically includes the following steps: Step 1.

1. Set a function to determine the end of pursuit: Set a maximum miss distance. When the distance between the projectile and the target is less than the maximum miss distance, it is judged as a success and the pursuit ends. When the distance is greater than the maximum miss distance or the time step is greater than the maximum time, the mission fails and the pursuit ends. Step 1.2: Design a reward for minimizing miss distance. To minimize miss distance, set a function that dynamically reduces the maximum miss distance. Specifically, when the training results reach a certain accuracy, the miss distance is automatically reduced. The miss distance reward is defined as follows: each time the pursuit ends: ， in, is a very small positive number, r is the distance between the projectile and the target; Step 1.3: Design a reward to minimize the energy consumption of the guidance process. Combine the miss distance reward and the energy consumption reward, and normalize them using the mean and variance obtained through training to make the total reward function smooth and continuous. The energy consumption reward of the guidance process is defined as: ， in, t f Indicates the current total time, a is the projectile acceleration, V is the projectile velocity, is the projectile lead angle; Step 1.4: Minimize the time reward design. The time reward is defined as: ， in, is a positive adjustment coefficient, t f is the current total time; Step 2: Build an intelligent guidance network model based on proximal strategy optimization; Step 3: Intelligent guidance network training based on proximal strategy optimization.

2. The intelligent guidance method based on proximal strategy optimization according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Modeling the relationship between proportional guidance action and environment: The guidance method optimization process is considered as a Markov decision process. The missile is taken as the intelligent agent, the target's maneuver trajectory is set in advance as part of the environment, and the state, action, and reward parameters are set for the intelligent agent, which ultimately form the environment. Step 2.2, Action and State Space Design: The range of the amplification coefficient is used as the action boundary value in the continuous action space, and the policy network of the continuous action space is adopted; The design of the state space reflects the state of the intelligent agent. It can reflect the air combat situation information during the guidance method optimization process, and thus evaluate the pros and cons of the situation of the missile and target in real time. It is also used to provide quantitative evaluation and reward feedback in the reward function design.

3. The intelligent guidance method based on proximal strategy optimization according to claim 2, characterized in that: The step 2.1 specifically includes the following steps: Step 2.1.

1. Set up the agent class for the initialization and behavior of the projectile and target: set the initial position and velocity properties of the projectile and target; also set the motion function, selecting from the range scale factor; Step 2.1.2: Set up the environment class to create an environment that facilitates the movement and interaction of the agent: Initialize various parameters as well as action space and state space, and determine the time step; Define the reset function required for each round of loop to initialize the missile and target, state, and time step; Setting a reward function and other functions, wherein the other functions include determining whether a hit is made and setting a random number seed; Update the target and projectile parameters based on the passed action, and return the new status, reward, completion status, and other fields.

4. The intelligent guidance method based on proximal strategy optimization according to claim 2, characterized in that: The design of the state space in step 2.2 includes the following state variables: The distance between the projectile and the target, the rate of change of distance, the projectile lead angle, the projectile entry angle, the difference between the target direction angle and the projectile direction angle, the relative height between the target and the projectile, the relative speed between the target and the projectile, the height and speed of the projectile.

5. The intelligent guidance method based on proximal strategy optimization according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3.

1. Design a network and loss function based on proximal policy optimization: Based on the proximal policy optimization algorithm, training is performed on the continuous action space. The proximal policy optimization network includes the Actor network and the Critic network. The Actor network includes an input layer, a hidden layer, and an output layer. The Actor network is responsible for action selection. The output is obtained by adding the mean of the predicted action to the noise multiplied by the variance, and the actual action is obtained by clipping according to the action range. The Actor network objective function is as follows: In the previous part, is the strategy ratio, is an estimate of the advantage function, which is used to evaluate the advantage of taking a specific action; in the latter part, is the shear rate, which is used to calculate the loss ratio after shearing; the loss function takes their minimum value, then takes the average to calculate the average loss, and finally backpropagates to calculate the gradient; The critic network includes an input layer, a hidden layer, and an output layer. The critic network is responsible for evaluating the value of the state-action pair. The critic network loss function is the mean square error between the state-action pair value and the state-action value in the experience area. The gradient is also calculated by backpropagation. The proximal policy optimization network uses the gradient descent method in its training function to update the error functions of the Actor network and the Critic network respectively; Step 3.2: Complete the guidance network training based on the reinforcement learning architecture: After setting the modules to be imported, build the main function training framework; before importing the main function, set up the Agent and Env class modules, the proximal strategy optimization algorithm setting module, and the buffer class setting module; Initialize the Agent and Env instances, initialize the GPU, various parameters, initialization strategy and experience replay pool; query whether the guidance network has past weights saved, if so, import and train them, otherwise initialize the model network; Enter the loop, for each round, reset the environment to get the state, initialize the time step, reward, completion, and storage array; Enter the sub-loop: Call the action selection function to obtain discrete action probability and continuous parameters to obtain the action; Call the function to get the value of the state-action pair, and get the next state and reward through the environment's step function; add the state and reward to the buffer; Update time step, state, and store trajectory; If the task is completed or the update conditions are met: end and call the buffer, and update the visualization results; If the update conditions are met: the loss value of the training buffer is calculated empirically and the buffer is reset; Check the sub-loop condition: if the task is not completed or the time is not up, return to the sub-loop; otherwise, store the reward at this time; Check loop condition: whether all rounds are trained, if yes, draw reward, otherwise continue training.

Images