Triple optimization SAC reinforcement learning method for robot continuous control

Abstract

The invention relates to the technical field of robot control, and discloses a robot continuous control-oriented triple optimization SAC reinforcement learning method, which comprises the following steps that a robot executes an initial strategy in a real environment to collect empirical data; initializing a strategy network, a value network and a target network, and performing experience playback and training updating on the strategy network, the value network and the target network through a triple optimization SAC reinforcement learning method; and continuously circulating the network updating process to obtain an optimal control sequence of task execution of the robot. The problem of underestimation is relieved by improving a strategy updating mechanism and stable value function learning, the problem that the robot cannot execute the optimal strategy is solved, underestimation deviation in the strategy is effectively relieved, the accuracy of robot action selection is improved, and more accurate guidance is provided for the robot to adopt correct actions.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a triple optimization SAC reinforcement learning method for continuous control of a robot. Background Art

[0002] In traditional reinforcement learning, the robot's position coordinates, speed, joint angle and other sensor perception information are defined as states, and the actuator control information such as motor drive and joint torque are defined as actions. Since their corresponding values ​​need to be precisely defined and discretized, as the dimensions of the state and action space increase, the computational complexity rises rapidly, which easily leads to the "dimensionality explosion" problem. Deep reinforcement learning allows robots to make decisions in high-dimensional or infinite action spaces by using neural networks to approximate and predict. For example, the "A Quadruped Robot Motion Control Method Based on Deep Reinforcement Learning" with application number 202410449230.X randomly selects the interaction data between the quadruped robot and the environment and inputs it into the target policy network for calculation, while adding action noise vectors to obtain candidate policy data, update the target policy network parameters based on the preset strategy, input the candidate policy data into the neural system for iteration, obtain the policy parameter set, and send environmental interaction commands to the quadruped robot based on the policy parameter set to control the movement of the quadruped robot.

[0003] However, in the process of using deep reinforcement learning methods to control the robot's motion, the estimation bias of value is a key problem that deep reinforcement learning cannot avoid. This will directly cause the robot to choose actions with incorrectly estimated values, and even cause the robot's mission to fail. The estimation bias of value comes from the noisy prediction value output by the neural network. After the greedy algorithm and temporal difference learning, the positive error is further amplified. This estimation bias will cause the robot to tend to choose actions with overestimated values, thus deviating from the optimal solution.

[0004] In the discrete control domain, Double DQN decouples the maximization operator of the greedy algorithm into action selection and action evaluation by introducing the target Q network, effectively alleviating the overestimation phenomenon. In the continuous control domain, the TD3 algorithm (Twin Delayed Deep Deterministic Policy Gradient) and the SAC algorithm (Soft Actor-Critic) use the clipped double Q learning technology to limit the upper limit of the dual value evaluator, but this method's excessive compression of the estimated lower limit can easily lead to underestimation, which can also lead to divergence or suboptimal problems.

[0005] The impact of underestimation bias on robot control performance has been widely concerned, but a comprehensive and mature solution has not yet been developed. To this end, the present invention proposes a triple optimization SAC reinforcement learning method for continuous robot control. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a triple optimization SAC reinforcement learning method for continuous control of robots, which alleviates the underestimation problem by improving the strategy update mechanism and stabilizing the value function learning, so as to solve the problem that the robot cannot execute the optimal strategy.

[0007] In order to solve the above technical problems, the present invention provides a triple optimization SAC reinforcement learning method for continuous control of robots, comprising:

[0008] Step S1: The robot executes the initial strategy in the real environment to collect experience data;

[0009] Step S2: Initialize the network structure, including the strategy network, value network and target network;

[0010] Step S3: Experience replay and training update of the policy network, value network and target network through triple optimization SAC reinforcement learning method;

[0011] Step S4: Periodically iterate and continuously execute the strategy network update, value network update, and target network update processes of step S3 to obtain the optimal control sequence for the robot to perform the task. The loop will be stopped only when the number of iterations reaches the preset maximum number of training times.

[0012] As an improvement of the triple optimization SAC reinforcement learning method for continuous control of robots in the present invention:

[0013] Initialize the policy network as follows: use a random seed to initialize the weights and biases of the policy network. The input of the policy network is the current state s of the robot, and the output is a probability distribution r of all possible actions that the robot may take. p (s,a);

[0014] Initialize the value network as follows: use a random seed to initialize the weights and biases of the value network. The input of the value network is the state information sampled by the robot in the experience replay pool and the actual action information of the robot at the current moment. The output is the expected cumulative reward obtained by the robot after taking a specific action:

[0015]

[0016] Among them, α is the temperature parameter. After the robot performs action a in the current state s, it transfers to the next state s￠ at the next time t￠. The environmental feedback reward obtained during the state transfer process is r(s,a). s￠ is input into the strategy network to obtain the robot's predicted action a￠ in state s￠; Q(s,a) represents the expected reward that can be obtained by following the strategy π starting from executing action a in state s.

[0017] Q π (s￠,a￠) represents the expected return after executing the action a' predicted by the policy network in the next state s'; policy entropy It is defined as the entropy of choosing all possible actions according to the policy π in a given state s:

[0018]

[0019] The target network is initialized as follows: the initial parameters of the target network directly copy all the parameters of the value network.

[0020] As a further improvement of the triple optimization SAC reinforcement learning method for continuous control of robots in the present invention:

[0021] The triple optimization SAC reinforcement learning method specifically includes:

[0022] (1) Hierarchical strategy update based on simulated annealing;

[0023] (2) Shear-adaptive moving exponential averaging;

[0024] (3) Delayed strategy update.

[0025] As a further improvement of the triple optimization SAC reinforcement learning method for continuous control of robots in the present invention:

[0026] The simulated annealing-based stratification strategy is updated as follows:

[0027] (1) Determine the maximum entropy strategy based on the energy model:

[0028] The optimization goal of the maximum entropy strategy based on the energy model is:

[0029]

[0030] π * represents the optimal strategy, defined as an optimal Boltzmann distribution p * (x), entropy H(p) is obtained by formula (2), γ is the discount factor, r t￠ It is represented as the feedback reward that the robot gets after performing a specific action at each future time t￠;

[0031] Defining the negative value Q(s,a) as the scoring function, the strategy energy function E(s,a) can be derived as:

[0032] E(s,a)=-Q(s,a) / a (5)

[0033] Next, the maximum entropy strategy π(s,a) and the optimal Boltzmann distribution based on the energy model are derived The connection between:

[0034]

[0035] Among them, the normalization constant Z(s) is defined as:

[0036]

[0037] D KL Weighs the current strategy π(s,a) against the optimal Boltzmann distribution The differences:

[0038]

[0039] The policy update process can be described as minimizing the current policy π(a|s) and the optimal Boltzmann The KL divergence between them makes the new strategy p￠(a|s) close to the optimal strategy π * (a|s):

[0040]

[0041] (2) Hierarchical strategy update mechanism based on simulated annealing

[0042] The estimated value of the first value network Considered as the current solution, the value network The energy function is expressed as The estimated value of another value network Seen as a new interpretation, value network The energy function is expressed as The probability that the new solution is selected is for:

[0043]

[0044] Wherein, T is the initial temperature of the simulated annealing algorithm, and the cooling function k is defined in the present invention as:

[0045]

[0046] Where c∈(0,1) is the cooling proportional constant, t and Represent the current time step and the total time step respectively;

[0047] Combining equation (1) and equation (10), the loss function of the hierarchical strategy update based on simulated annealing is expressed as:

[0048]

[0049] Where t￠ represents the expected number of training steps for switching simulated annealing, and p q represents the policy network.

[0050] As a further improvement of the triple optimization SAC reinforcement learning method for continuous control of robots in the present invention:

[0051] The shear-adaptive moving exponential average is:

[0052] (1) Determine the relationship between soft update and moving exponential average:

[0053] SAC uses a soft update method to replicate the value network parameters through a fixed smoothing factor b To update the target network parameters

[0054]

[0055] Assume there are n data: The sliding average at time t It is expressed as:

[0056]

[0057] in, When the sliding window is fixed to n, the moving exponential average is equivalent to soft update;

[0058] (2) Design adaptive learning weights:

[0059] The priority p corresponding to the time difference residual of priority experience i i for:

[0060]

[0061] Wherein, m is the priority adjustment index;

[0062]

[0063] in, Represents the value network, represents the target network;

[0064] The adaptive learning weight w is:

[0065]

[0066] Among them, n is the total number of historical time series difference residuals, e is a very small number, and λ is the shear ratio parameter;

[0067] The shear-adaptive moving exponential average can be expressed as:

[0068]

[0069] As a further improvement of the triple optimization SAC reinforcement learning method for continuous control of robots in the present invention:

[0070] The delay strategy is updated as follows:

[0071] The update frequency of the policy network should be lower than that of the value network. By updating the value network more frequently, the residual error can be effectively minimized before the policy is updated, providing clearer value guidance for the policy update.

[0072] The beneficial effects of the present invention are mainly reflected in:

[0073] 1. The present invention designs a hierarchical strategy update mechanism based on the metal simulated annealing algorithm, selects value guidance based on the metropolitan probability criterion, effectively reduces the underestimation bias in the strategy, and improves the accuracy of the robot's action selection;

[0074] 2. The clipping adaptive moving exponential average method designed by the present invention takes into account the fluctuation of the historical time series difference residual, can dynamically adjust the update amplitude of the target network, effectively alleviate the underestimation phenomenon in the value, and enhance the accuracy of judging the output action of the robot;

[0075] 3. The present invention introduces a delayed strategy update method, which reduces the cumulative error in training through high-frequency value updates, can effectively reduce the variance fluctuation of training, stabilize the learning of the value function, and provide more precise guidance for the robot to take correct actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0077] Figure 1 A schematic diagram of a triple optimization SAC reinforcement learning method for continuous control of a robot according to the present invention;

[0078] Figure 2 Ant-v4 continuous control task for the MuJoCo benchmark;

[0079] Figure 3 HalfCheetah-v4 continuous control task for the MuJoCo benchmark;

[0080] Figure 4 This is a performance comparison chart of the ablation experiment;

[0081] Figure 5 It is a comparison chart of standard deviation and mean;

[0082] Figure 6 This is the performance graph of each algorithm in the Ant-v4 continuous control task;

[0083] Figure 7 The performance diagram of each algorithm in the HalCheetah-v4 continuous control task. DETAILED DESCRIPTION

[0084] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto:

[0085] Example 1: A triple optimization SAC reinforcement learning method for continuous control of robots, such as Figure 1 As shown, step S1: environmental interaction and experience collection

[0086] In this step, the robot executes the initial strategy in the real environment and collects enough empirical data to optimize the strategy. The continuous state space is defined as Contains sensor perception information such as robot posture and speed. The continuous action space is defined as Contains actuator control information such as motor drive and joint torque.

[0087] For a fixed sampling time t, there is a state transition probability p(s t+1 |s t ,a t ): It indicates the current state of the robot. According to the random strategy π(a t |s t ): Take Action Then, transfer to the next state During the transfer process, the robot will also receive a bounded reward r∈[r min ,r max ] to measure the quality of the action chosen by the robot in this state. Select a suitable data container to establish an experience replay pool The experience data of the robot's interaction with the environment is represented as a tuple The data are stored in the form of for training the triple optimized SAC (TESAC) algorithm of the present invention.

[0088] Step S2: Initialize the network structure

[0089] The triple optimized SAC (TESAC) of the present invention is based on the maximum entropy learning framework and the actor-critic method, including a policy network pq , Value Network and With the target network and Among them, q, as well as are the parameters of the strategy network, value network, and target network respectively.

[0090] Step S2.1: Initialize the policy network

[0091] Use random seeds to initialize the weights and biases of the policy network. The policy network p q The input is the current state s of the robot, including the robot's position, speed and other state information, which comes from the real-time perception of the sensor. The policy network is responsible for guiding the robot to select action a and outputting a probability distribution r of all possible actions that the robot may take. p (s,a), which serves as the control input for actuators such as drive motors.

[0092] Step S2.2: Initialize the value network

[0093] Use random seeds to initialize the weights and biases of the value network. The input of the value network is the experience replay pool. The state information sampled by the robot and the actual action information of the robot at the current moment are responsible for evaluating the value of the action selected by the robot at the current moment t. The output is the expected cumulative reward obtained by the robot after taking a specific action:

[0094]

[0095] Among them, α is the temperature parameter. After the robot performs action a in the current state s, it transfers to the next state s￠ at the next time t￠. The environmental feedback reward obtained during the state transfer process is r(s,a). s￠ is input into the strategy network to obtain the robot's predicted action a￠ in state s￠; Q(s,a) represents the expected reward and entropy of the strategy that can be obtained by following strategy π starting from executing action a in state s, that is, the expected return, Q π (s￠,a￠) represents the expected reward after executing the action a' predicted by the policy network in the next state s'.

[0096] Policy Entropy It is defined as the entropy of all possible actions selected according to the strategy π in a given state s, which is used to measure the randomness of the strategy:

[0097]

[0098] Maximizing entropy increases the diversity of actions chosen by the robot, making the continuous control of the robot smoother.

[0099] Step S2.3: Initialize the target network

[0100] The target network is a copy of the value network, and its initial parameters are obtained by directly copying all the parameters of the value network. It aims to provide a stable learning target for the temporal difference learning of the value network, thereby reducing the risk of the robot damaging itself or the environment due to erroneous actions.

[0101] Step S3: Experience replay and training update

[0102] From the Experience Replay Pool A batch of experience samples are randomly selected from the learning. The goal of traditional reinforcement learning is usually to maximize the maximum long-term reward that the robot can obtain when performing tasks. The expected sum of cumulative rewards is defined as:

[0103]

[0104] Among them, r t￠ It is represented as the feedback reward that the robot receives after performing a specific action at each future time t￠. The discount factor γ determines the immediate reward r of these future rewards relative to the current time t t The importance of.

[0105] The goal of Triple Optimization SAC (TESAC) is to find the optimal control sequence for the robot to perform a specific task under the Markov framework, which is also the optimal strategy π * , defined as an optimal Boltzmann distribution p * (x). This strategy simultaneously pursues the expectation of cumulative rewards and the maximization of strategy entropy:

[0106]

[0107] Step S3.1: Hierarchical strategy update based on simulated annealing

[0108] Triple Optimization SAC (TESAC) converts the optimal strategy π * Defined as the optimal Boltzmann distribution p * (x), where entropy Encourage the strategy distribution to be flat and eventually converge. The optimal strategy π * Allowing robots to take more diverse and valuable actions when performing tasks.

[0109] Defining the negative value Q(s,a) as the scoring function, the strategy energy function E(s,a) can be derived as:

[0110] E(s,a)=-Q(s,a) / a (5)

[0111] Next, the connection between the maximum entropy strategy based on the energy model and the optimal Boltzmann distribution is derived:

[0112]

[0113] Among them, the normalization constant Z(s) is defined as:

[0114]

[0115] KL divergence (Kullback-Leibler divergence)D KL Weighs the current strategy π(s,a) against the optimal Boltzmann distribution The differences:

[0116]

[0117] The policy update process can be described as minimizing the current policy π(a|s) and the optimal Boltzmann The KL divergence between them makes the new strategy p￠(a|s) close to the optimal strategy π * (a|s):

[0118]

[0119] The present invention proposes a hierarchical strategy update mechanism based on simulated annealing. In the early stage of training, the agent does not fully understand the environment and the output noise is large, so the shear double Q learning is still used to suppress the overestimation problem. In the later stage of training, the network noise continues to decrease, and the value of low energy E(s,a) is selected through the metropolitan probability criterion to alleviate the underestimation problem caused by shear double Q learning, thereby improving the accuracy of robot decision making.

[0120] The estimated value of the first value network Considered as the current solution, the value network The energy function is expressed as The estimated value of another value network Seen as a new interpretation, value network The energy function is expressed as The probability that the new solution is selected is for:

[0121]

[0122] Where T is the initial temperature of the simulated annealing algorithm, and the cooling function k is:

[0123]

[0124] Where c∈(0,1) is the cooling proportional constant, t and Represent the current time step and the total time step respectively;

[0125] Combining equation (1) and equation (10), the loss function of the hierarchical strategy update based on simulated annealing is expressed as:

[0126]

[0127] Where t￠ represents the expected number of training steps for switching simulated annealing.

[0128] Step S3.2: Value Network Update

[0129] The value network helps the robot evaluate the long-term value of each possible action, thereby selecting the action that will bring the greatest cumulative reward. The learning goal of the value network is to minimize and The output of the target network and The error between them is minimized by gradient descent, and the loss function is expressed as:

[0130]

[0131] Step S3.3: Target network update based on clipped adaptive moving exponential averaging

[0132] The target network reduces fluctuations in the learning process by providing a relatively fixed target value, helping the robot learn a more stable control strategy. Triple Optimized SAC (TESAC) uses a soft update method to replicate the value network parameters through a fixed smoothing factor b. To update the target network parameters

[0133]

[0134] In order to further understand the mechanism of soft update, we need to first understand the moving exponential average. Assume there are n data: The sliding average at time t It is expressed as:

[0135]

[0136] in, When the sliding window is fixed to n, the moving exponential average is equivalent to the soft update. Both use the idea of ​​exponential weighted average to calculate the local average by giving a higher weight to recent data. In order to improve the anti-noise ability of soft update, the present invention combines the idea of ​​preferential experience playback and calculates the local average value according to the residual of the current time series difference δ and the mean of the residual of the historical time series difference The shear adaptive soft update method is designed to account for the fluctuation of the network. The time difference residual δ(s,a) measures the error between the predicted value of the value network and the target value of the target network:

[0137]

[0138] in, Represents the value network, represents the target network;

[0139] Prioritized experience playback aims to assign different priorities to each experience, so as to prioritize the training of areas with lower network parameter accuracy. The larger the temporal difference residual of experience i, the higher the corresponding priority p. i Higher:

[0140]

[0141] In value iteration, the learning goal of the target network is to continuously approach the value network parameters. A higher temporal difference residual indicates that the accuracy of the value network output is low. At this time, the target network should reduce the learning ratio of the value network. On the contrary, when the temporal difference residual is small, the target network should appropriately increase the learning ratio of the value network. After clarifying the learning rules of the target network, combined with the ideas of moving exponential average and priority experience replay, the present invention proposes an adaptive learning weight w:

[0142]

[0143] Where n is the total number of historical time series difference residuals, which is equivalent to the size of the sliding window. e is a very small number, and λ is the shear ratio parameter. In order to avoid training oscillation caused by excessive changes in w, the output of w is sheared to maintain it in a reasonable range. Combining equations (14) and (18), the sheared adaptive moving exponential average can be expressed as:

[0144]

[0145] Step S3.4: Delay strategy update

[0146] Inaccurate value estimation can easily lead to residual error accumulation during training, which in turn causes fluctuations in the value function. To solve this problem, a delayed policy update method is introduced. The policy network is updated less frequently than the value network. By updating the value network more frequently, the residual error is effectively minimized before the policy is updated, providing the robot with clearer value guidance when choosing actions.

[0147] Step S4: Periodic iteration

[0148] This step continuously loops through the policy network update, value network update, and target network update processes of step S3, aiming to continuously learn from the sampled samples to improve the strategy, so as to find the optimal control sequence for the robot to perform the task. In actual tasks, the optimal strategy is converted into specific control signals, which control the robot's actuators (such as motors, servos, hydraulic or pneumatic devices). To prevent excessive computing resources and time consumption, the maximum number of training iterations can be pre-set. Before this condition is met, iterative learning will continue, constantly learning from experience samples to optimize the strategy until its behavior pattern can cope with the complex challenges in the environment.

[0149] experiment:

[0150] The benchmark environment for this experiment is MuJoCo's Ant-v4 and HalfCheetah-v4 continuous control tasks, which are shown in Figure 2 and Figure 3 Among them, the Ant-v4 action space includes 7 dimensions such as the left foreleg joint torque and the right foreleg joint torque, the state space includes 27 dimensions such as the three-dimensional coordinates of the trunk and the angle between the right front links, and the reward function includes three parts such as duration, movement reward, and external contact pressure. The HalfCheetah-v4 action space includes 6 dimensions such as the torque of the posterior thigh trochanter and the torque of the tibial trochanter, the state space includes 17 dimensions such as the front angle and the second trochanter angle, and the reward function consists of two parts: movement reward and operation reward.

[0151] This experiment uses mainstream model-free deep reinforcement learning algorithms as the baseline. These algorithms have been widely verified in theoretical research or practical applications, including SAC (Soft Actor-Critic), DDPG (Deep Deterministic Policy Gradient), TD3 (Twin Delayed Deep Deterministic Policy Gradient), TRPO (Trust Region Policy Optimization) and PPO (Proximal Policy Optimization) algorithms. In order to reduce the contingency of the experiment, each algorithm is trained five times independently in each control task, and a random seed is initialized for each training. The maximum time step of a single training is 1000000, and an evaluation is performed every 2000 training steps. The average return of 10 episodes is calculated for each evaluation.

[0152] 1. Underestimation bias experiment

[0153] Both overestimation and underestimation biases will lead to incorrect strategy selection actions, which will directly lead to a decrease in algorithm performance. In order to verify the effectiveness of the hierarchical policy update (HPU) and the clipping adaptive moving exponential average (CAE) method proposed in this paper, we selected SAC as the algorithm baseline. Through ablation experiments, we obtained the performance differences between SAC (SAC+HPU) combined with only the hierarchical policy update mechanism, SAC (SAC+CAE) combined with only the clipping adaptive moving exponential average, and the triple optimized SAC (TESAC) proposed in this paper. The ablation experiment results are shown in Figure 4 In the experiment, under the same number of training steps, both modifications effectively improved the performance of SAC, and the triple optimized SAC (TESAC) combining the two achieved the best results. This shows that the proposed improved methods of hierarchical policy update (HPU) and clipping adaptive moving exponential average (CAE) can effectively alleviate the underestimation problem in SAC and improve the policy performance, greatly improving the decision-making stability and accuracy of the robot during autonomous tasks.

[0154] 2. Training variance

[0155] A stable learning goal can reduce the variance of training and help obtain more consistent and reliable results in multiple robot tasks. This experiment uses SAC as a comparison baseline, and conducts experimental comparisons on three modifications: SAC combined with delayed policy update (DPU) (SAC+DPU), SAC combined with clipping adaptive moving exponential averaging (CAE) (SAC+CAE), and the triple optimized SAC (TESAC) of the present invention. The standard deviation mean of five independent experiments is used to describe the variance trend. The experimental results are shown in Figure 2. Figure 5 As shown. Compared with the baseline SAC, the SAC method using only delayed policy update (SAC+DPU) and the SAC method using only clipping adaptive moving exponential averaging (SAC+CAE) performed comparably, reducing the training standard deviation by 32.1% and 31.9%, respectively. The triple optimized SAC (TESAC) algorithm of the present invention, which combines the advantages of DPU and CAE, performed the best, reducing the training standard deviation by 48.1%. This shows that the delayed policy update and clipping adaptive moving exponential averaging methods proposed in the present invention can effectively stabilize the algorithm training process, thereby providing a more robust strategy for the robot's autonomous tasks.

[0156] 3. Performance

[0157] The reward mean curve is already in Figure 6 and Figure 7 exhibit, Figure 6 This is the performance diagram of each algorithm in the Ant-v4 continuous control task. Figure 7The performance graph of each algorithm in the HalCheetah-v4 continuous control task. The curves in the figure correspond to the mean rewards of five independent experiments to measure the performance of the strategy, and the shaded part corresponds to the standard error of the rewards of five independent experiments. In the Ant-v4 and HalfCheetah-v4 tasks, the performance of the triple optimized SAC (TESAC) is better than the mainstream baseline algorithm. The triple optimized SAC proposed in the present invention shows optimistic rising potential in Ant-v4 and HalfCheetah-v4, and there is still room for improvement. Compared with the original SAC algorithm, the triple optimized SAC achieved 42.2% and 11.1% performance improvements in Ant-v4, HalfCheetah-v4 and other tasks, respectively. These results show that the three improvement measures proposed in the present invention effectively enhance the performance of SAC, enable the robot to respond to environmental changes faster, and improve the efficiency of task execution.

[0158] Finally, it should be noted that the above examples are only some specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many variations. All variations that can be directly derived or associated with the content disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A triple optimization SAC reinforcement learning method for continuous robot control, characterized by: Step S1: The robot executes the initial strategy in the real environment to collect experience data; Step S2: Initialize the network structure, including the strategy network, value network and target network; Step S3: Experience replay and training update of the policy network, value network and target network through triple optimization SAC reinforcement learning method; Step S4: Periodically iterate and continuously execute the strategy network update, value network update, and target network update processes of step S3 to obtain the optimal control sequence for the robot to perform the task. The loop will be stopped only when the number of iterations reaches the preset maximum number of training times.

2. The triple optimization SAC reinforcement learning method for continuous control of robots according to claim 1, characterized in that: Initialize the policy network as follows: use a random seed to initialize the weights and biases of the policy network. The policy network p q The input is the current state s of the robot, and the output is the probability distribution r of all possible actions that the robot can take. p (s,a); Initialize the value network as follows: use a random seed to initialize the weights and biases of the value network. The input of the value network is the state information sampled by the robot in the experience replay pool and the actual action information of the robot at the current moment. The output is the expected cumulative reward obtained by the robot after taking a specific action: Among them, α is the temperature parameter. After the robot performs action a in the current state s, it transfers to the next state s￠ at the next time t￠. The environmental feedback reward obtained during the state transfer process is r(s,a). s￠ is input into the strategy network to obtain the robot's predicted action a￠ in state s￠; Q(s,a) represents the expected reward that can be obtained by following the strategy π starting from executing action a in state s. Q π (s￠,a￠) represents the expected return after executing the action a' predicted by the policy network in the next state s'; policy entropy It is defined as the entropy of choosing all possible actions according to the policy π in a given state s: The target network is initialized as follows: the initial parameters of the target network directly copy all the parameters of the value network.

3. The triple optimization SAC reinforcement learning method for continuous control of robots according to claim 2 is characterized in that: The triple optimization SAC reinforcement learning method specifically includes: (1) Hierarchical strategy update based on simulated annealing; (2) Shear-adaptive moving exponential averaging; (3) Delayed strategy update.

4. The triple optimized SAC reinforcement learning method for continuous control of robots according to claim 3 is characterized by: The simulated annealing-based stratification strategy is updated as follows: (1) Determine the maximum entropy strategy based on the energy model: The optimization goal of the maximum entropy strategy based on the energy model is: π * represents the optimal strategy, defined as an optimal Boltzmann distribution p * (x), entropy It is obtained by formula (2), γ is the discount factor, r t￠ It is represented as the feedback reward that the robot gets after performing a specific action at each future time t￠; Defining the negative value Q(s,a) as the scoring function, the strategy energy function E(s,a) can be derived as: E(s,a)=-Q(s,a) / a (5) Next, the maximum entropy strategy π(s,a) and the optimal Boltzmann distribution based on the energy model are derived The connection between: Among them, the normalization constant Z(s) is defined as: D KL Weighs the current strategy π(s,a) against the optimal Boltzmann distribution The differences: The policy update process can be described as minimizing the current policy π(a|s) and the optimal Boltzmann The KL divergence between them makes the new strategy p￠(a|s) close to the optimal strategy π * (a|s): (2) Hierarchical strategy update mechanism based on simulated annealing The estimated value of the first value network Considered as the current solution, the value network The energy function is expressed as The estimated value of another value network Seen as a new interpretation, value network The energy function is expressed as The probability that the new solution is selected is for: Wherein, T is the initial temperature of the simulated annealing algorithm, and the cooling function k is defined in the present invention as: Where c∈(0,1) is the cooling proportional constant, t and Represent the current time step and the total time step respectively; Combining equation (1) and equation (10), the loss function of the hierarchical strategy update based on simulated annealing is expressed as: Where t￠ represents the expected number of training steps for switching simulated annealing, and p q represents the policy network.

5. The triple optimized SAC reinforcement learning method for continuous control of robots according to claim 4, characterized in that: The shear-adaptive moving exponential average is: (1) Determine the relationship between soft update and moving exponential average: SAC uses a soft update method to replicate the value network parameters through a fixed smoothing factor b To update the target network parameters Assume there are n data: The sliding average at time t It is expressed as: in, When the sliding window is fixed to n, the moving exponential average is equivalent to soft update; (2) Design adaptive learning weights: The priority p corresponding to the time difference residual of priority experience i i for: Where m is the priority adjustment index; in, Represents the value network, represents the target network; The adaptive learning weight w is: Among them, n is the total number of historical time series difference residuals, e is a very small number, and λ is the shear ratio parameter; The shear-adaptive moving exponential average can be expressed as:

6. The triple optimized SAC reinforcement learning method for continuous control of robots according to claim 5, characterized in that: The delay strategy is updated as follows: The update frequency of the policy network should be lower than that of the value network. By updating the value network more frequently, the residual error can be effectively minimized before the policy is updated, providing clearer value guidance for the policy update.

Images