Robot control optimization method based on deep reinforcement learning

Abstract

The invention discloses a robot control optimization method based on deep reinforcement learning, and the method comprises the steps: firstly initializing a control system, constructing an experience playback buffer pool, and setting an Actor network strategy function, a Critic network weight, a hyper-parameter, and a target network parameter; then, the robot generates an action according to the Actor network, collects environment feedback after executing the action, and stores the state transition tuple into a buffer pool; next, a small batch of data is sampled, a time difference error is calculated, and a network parameter is updated accordingly to minimize a value estimation deviation, synchronously optimize the network parameter, maximize an expected value of a state-action value function, and improve control performance. According to the method, a multi-thread architecture and a deep reinforcement learning technology are fused, the real-time performance, stability and generalization ability of robot control are improved, the problem of algorithm convergence in a non-stationary environment is effectively solved, the cost of adjustment and calculation is reduced, and the robustness of long-term stable operation and deployment feasibility in a complex scene are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology, and in particular relates to a robot control optimization method based on deep reinforcement learning. Background Technology

[0002] Current research on robot control algorithms is shifting from traditional methods to intelligent approaches. While traditional control algorithms, such as PID control, exhibit good stability and ease of implementation in simple, linear systems, their limitations are becoming increasingly apparent when faced with complex, dynamic, and nonlinear modern robotic tasks. Furthermore, traditional methods lack flexibility in multi-task learning and dynamic environments, making it difficult to meet the diverse needs of robots in complex scenarios.

[0003] In contrast, reinforcement learning algorithms exhibit significant advantages. Through the interaction between the robot and its environment, reinforcement learning learns the optimal strategy using a trial-and-error mechanism, without requiring a precise system model, and can effectively address complex, dynamic, and nonlinear control problems. Furthermore, reinforcement learning can handle multi-objective optimization problems, guiding the robot to simultaneously optimize multiple performance metrics, such as speed, energy consumption, and accuracy, through reward mechanisms. Its data-driven learning approach also supports the robot's continuous learning and adaptation in constantly changing environments, further improving control performance. Therefore, reinforcement learning algorithms are gradually becoming an important development direction in the field of robot control, providing new ideas and methods for efficiently solving complex tasks.

[0004] The AC (Actor-Critic) algorithm is an important framework in reinforcement learning, combining the advantages of policy gradient and value function estimation. It consists of two parts: the Actor selects actions based on the policy, while the Critic evaluates the policy and provides feedback. Through the Critic's value function estimation, the Actor can update the policy more efficiently, reducing variance and improving learning efficiency. Furthermore, the AC algorithm is suitable for continuous action spaces and can handle complex tasks such as robot control and game AI. Its variants, such as A2C and A3C, further improve performance; A3C accelerates the training process through asynchronous multi-threaded learning. Despite its significant theoretical advantages, the AC algorithm still faces some challenges in practical deployment. Because real-world environments are often non-stationary, state distributions and reward functions may dynamically change over time, leading to systematic biases in the Critic network's estimation of the value function. This can interfere with the Actor network's policy update process and compromise the algorithm's overall stability. In addition, the AC algorithm is highly sensitive to hyperparameter settings. Key parameters such as the learning rate, discount factor, and the estimation method of the advantage function need to be finely adjusted according to the specific task characteristics (e.g., state space dimensionality, action space complexity, and sparsity of the reward signal). These parameters often have complex coupling relationships, further increasing the difficulty of parameter tuning and experimental costs. At the same time, with the increase in environmental complexity, the high dimensionality of the state and action spaces significantly exacerbates the computational burden of the algorithm. The AC framework needs to rely on a large amount of environmental interaction data to achieve effective approximation of the value function and policy function, resulting in a significant increase in computational resource consumption and time overhead during the training process, which limits its widespread application in resource-constrained scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a robot control optimization method based on deep reinforcement learning, which solves the problems of insufficient stability, adaptability and computational efficiency of traditional methods in complex environments, and achieves more accurate and efficient robot control.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by this invention is a robot control optimization method based on deep reinforcement learning, comprising the following steps:

[0007] S1. Initialize the robot control system parameters, including Actor network parameters, Critic network weights, temperature network parameters, hyperparameters, and target network parameters; clear the experience replay buffer.

[0008] S2. The robot control system generates an action according to the Actor network policy function. After executing the action, it collects environmental feedback information and confirms the experience replay sampling weight. It integrates the environmental feedback information and the experience replay sampling weight into a state transition tuple as an experience sample. The experience sample is written into the experience replay buffer pool.

[0009] S3. Extract samples from the experience replay buffer pool, determine the time difference error based on the samples, update the network parameters based on the error value to minimize the value estimation bias, and simultaneously update the Actor network parameters and the Critic network weights to maximize the expected value of the state-action value function.

[0010] S4. Repeat S2 to S3 to optimize robot control.

[0011] Furthermore, the robot control system parameters in S1 include Actor network parameters, Critic network weights, temperature network parameters, hyperparameters, and target network parameters: among which, the Actor network parameters include policy network parameters ψ, which are used to generate the policy function for actions;

[0012] Critic network weights, including N Q-factors function Network parameter φ i , i = 1, ..., N; φ i The expected cumulative reward for the robot to take motion action a in joint state s;

[0013] The temperature network parameter α is used to adjust the diversity of exploration behaviors;

[0014] The target network parameters include N Q parameters. function Target network parameters φ i,target The Q function Target network parameters φ i,target Used to provide a stable estimate of the target value.

[0015] Furthermore, in S1, the hyperparameters include:

[0016] The target network soft update coefficient ρ is used to control the update speed of the target network parameters;

[0017] Mini-batch size (batch) is used to set the number of samples used in each training session.

[0018] The update rate UTD is used to determine the number of times network parameters are updated during a single training session.

[0019] Priority experience replay weight p t This is used to adjust the sampling probability of samples in the experience replay buffer;

[0020] The weight threshold β of the neural network parameters is used to control the conditions or range for updating the network parameters.

[0021] Temperature network learning rate λ α , used to determine the update magnitude of the temperature network parameter α;

[0022] Policy network learning rate λ ψ , used to control the update magnitude of the policy network parameter ψ;

[0023] Target network learning rate λ φ Used to control the target network parameter Q function Target network parameters φ i,target New range;

[0024] Target entropy H T Initialize target entropy H T = -dimA, where dimA is the action space dimension and the target entropy H is... T Used to guide the direction of robot control optimization;

[0025] Q function Number of networks M, in updating Q function The network is used to calculate the target value.

[0026] Furthermore, the specific process of S2 includes:

[0027] S201, at each time step t, the policy network at the current joint s t Output distribution π(·|s) under the state t Action a is obtained by sampling from ) t And perform the action to push the robot into a new joint state. t+1 ;

[0028] S202, In the context of action a t After responding, the robot collects environmental feedback information, including new joint states. t+1 Reward value r t and the robot's current joint state s t and action a t Simultaneously, based on the time difference error and the reward value r t Determine the empirical playback sampling weight p t And store;

[0029] S203, The collected environmental feedback information and experience playback sampling weight p t Integrate into a state transition tuple (s) t ,a t ,r t ,s t+1 ,p t Then it is written to the experience replay buffer pool.

[0030] Furthermore, the empirical replay sampling weight p t The method for determining it is as follows:

[0031]

[0032] Where TD-error is the time difference error, and y is the target value of the time difference. Is the i-th Q function Network for joint states s t and action a t The estimated value, ε, is a non-zero positive number used to provide a non-zero lower bound for the TD-error of all samples, ensuring that each experience has a non-zero probability of being selected in the sampling, r t It represents the reward value obtained, and softmax() is the normalization function.

[0033] Furthermore, the specific process of S3 includes:

[0034] S301. From the experience playback buffer D, according to the sampling weight p t Samples related to robot motion control are selected to form a sample set B, with a sample size of batch.

[0035] S302. Determine the forward propagation and parameter weights of the neural network; perform forward propagation of the neural network on the sample set B obtained by sampling, and determine whether the parameter weights exceed the preset threshold β during the forward propagation process; if they exceed the threshold β, perform neuron initialization operation.

[0036] S303. Determine the time difference target value y:

[0037]

[0038] Where, r t γ is the reward value, and γ is the discount factor. Represents the i-th Q function Target network, s t+1 a t+1 These are the new joint states and actions, α is a temperature network parameter used to adjust the diversity of exploration behaviors, and π ψ (∣s t+1 ) indicates that in state s t+1 Output distribution of the lower policy network;

[0039] S304. Update the Critic network weights;

[0040] S305, Update target network parameters;

[0041] S306. Update the empirical replay sampling weights based on the updated target network parameters;

[0042] S307, Update temperature network parameters;

[0043] S308, Update policy network parameters;

[0044] S309. The parameters of the Actor network, the weights of the Critic network, the parameters of the temperature network, the hyperparameters, and the parameters of the target network are updated cyclically, with an update rate UTD.

[0045] Furthermore, S304 updating the Critic network weights specifically involves updating the Q value in the Critic network weights. function Network parameter φ i :

[0046]

[0047] in, It is Q function The learning rate of the network The loss function J(φ) represents i ) for φ i gradient, Is the i-th Q function Network on state s t and action a t The estimated value is y, which is the time difference target value, B is the sample set obtained by sampling, and s is a sample in the sample set B.

[0048] Furthermore, S305 updating the target network parameters specifically involves updating Q using a soft update strategy. function Target network parameters φ target,i :

[0049] φ target,i ←ρφ target,i +(1-ρ)φ i

[0050] Where ρ is the target network soft update coefficient, φ i Q is the weight in the Critic network. function Network parameters.

[0051] Furthermore, the method for updating the temperature network parameter α in S307 is as follows:

[0052]

[0053] Where, λ α It is the temperature network learning rate. H represents the gradient of the loss function J(α) with respect to α. t It is the target entropy, π ψ (a t |s t ) indicates that in state s t Choose action a t The probability is given by s, where s is a sample in the sample set B, and B is the sample set obtained by sampling.

[0054] Furthermore, the method for updating the policy network parameter ψ in S308 is as follows:

[0055]

[0056] Where, λ ψ It is the learning rate of the policy network. Let s represent the gradient of the loss function J(ψ) with respect to ψ, s be a sample in the mini-batch sample set B, B be the sample set obtained from sampling, α be the temperature network parameters, and N be the Q value. function Number of target networks Q represents function Network, π ψ (a t |s t ) is in state s t Choose action a t The probability, where i is the index symbol.

[0057] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention proposes a robot control optimization method based on deep reinforcement learning, employing a multi-threaded architecture to achieve high-frequency interaction between the robot and the environment and real-time policy updates, significantly improving the accuracy and real-time response capability of the control system. By integrating multi-Q networks and target network mechanisms, this invention effectively suppresses overestimation bias in Q-value estimation, reduces the variance of the value function, and thus enhances the stability of the learning process. Furthermore, it introduces a priority experience replay technique to optimize data utilization efficiency, and combines a continuous backpropagation method to address sparse reward environments, improving the algorithm's adaptability and generalization performance in complex scenarios. The integrated temperature network mechanism dynamically adjusts the exploration-utilization balance, avoiding the policy from getting trapped in local optima and strengthening global optimization capabilities. The computational architecture based on the producer-consumer model enables parallel processing of data acquisition and network updates, significantly accelerating training convergence speed and ensuring smooth and stable motion control characteristics in dynamic environments. This invention effectively solves the convergence problem of Actor-Critic algorithms in non-stationary environments, reduces the sensitivity to hyperparameter configuration, reduces experimental parameter tuning costs, alleviates the computational burden of high-dimensional state and action spaces, significantly enhances the deployment feasibility of the algorithm on resource-constrained platforms, and provides technical support for the long-term robust operation of robots in dynamic and complex scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is the overall flowchart of the algorithm of this invention;

[0060] Figure 2 This is a diagram of the Actor-Critic network architecture in this invention;

[0061] Figure 3 This is a schematic diagram of the experience playback buffer and priority sampling mechanism of the present invention;

[0062] Figure 4 This is a schematic diagram illustrating the combination of the multi-Q network and the target network of the present invention;

[0063] Figure 5 This is a flowchart of the temperature network update mechanism of the present invention;

[0064] Figure 6 This is a schematic diagram of the producer-consumer parallel architecture of the present invention, illustrating the division of responsibilities and data interaction methods between the producer thread and the consumer thread.

[0065] Figure 7 This is a comparison chart of the rewards at different time steps for six algorithms—TD3, SAC, REDQ, TQC, TQC20, and CAC—used to control a Humanoid robot in the MuJoCo simulation environment.

[0066] Figure 8 This is a comparison chart of the average rewards for six algorithms at different time steps. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] like Figure 1 This embodiment provides a robot control optimization method based on deep reinforcement learning. It utilizes a multi-threaded architecture to achieve real-time interaction and policy updates between the robot and its environment, thereby improving the accuracy and real-time performance of robot motion control. The method mainly includes the following three processes:

[0069] S1. When the control policy training process starts, the robot control system is first initialized, including the weight parameters of the Actor and Critic networks, to give the policy model preliminary action generation capabilities. Simultaneously, key hyperparameters required for algorithm operation are set, including the learning rate, discount factor, and entropy coefficient, to control the policy update step size, long-term reward discounting degree, and exploratory behavior intensity. Furthermore, an experience replay buffer is initialized to store state transition tuples generated during robot interaction with the simulated or real environment. These state transition tuples contain a transition sequence consisting of a state, action, immediate reward, and the next state. At the same time, the target network parameters are initialized. This target network has the same structure as the main network, and its parameters are slowly updated during training through a soft update mechanism or by periodically replicating the main network parameters to improve the stability of value estimation and accelerate policy convergence.

[0070] S2. After the robot system initializes, it begins operation. During this process, the robot samples and executes actions from the action distribution output by the Actor network based on the current policy, applying the selected actions to the simulated or real environment, thereby triggering state transitions. Throughout this process, the robot control system collects environmental feedback information in real time, including the current state observation, the specific action executed, the immediate reward returned by the environment, and the updated next state, forming complete experience samples (state transition tuples). These experience samples are written into the experience replay buffer for subsequent policy optimization and value function updates. Through continuous multi-round interactions with the environment, the robot can accumulate diverse experience data, providing ample learning basis for subsequent deep reinforcement learning-based control policy optimization.

[0071] S3. Neural Network Update (Training Phase): The robot control system randomly samples a small batch of experience samples from the experience replay buffer and uses these samples to update the parameters of the Actor network and Critic network. Specifically, the Critic network parameters are adjusted by calculating the Time Difference Error (TD error) using the small batch of experience samples, enabling it to more accurately estimate the value of the current state-action pair. Simultaneously, based on the actions generated by the current strategy and their corresponding estimated values ​​by the Critic network, the Actor network parameters are updated to generate a better action strategy. This update process allows the robot to continuously learn from historical experience, gradually improving its decision-making ability and control performance in the environment. Through continuous iteration of this update process, the system gradually improves the decision quality of the policy network and the prediction accuracy of the value network, thereby achieving efficient learning and dynamic optimization of the robot's motion control strategy.

[0072] The following detailed description of this implementation method will be provided through a specific example.

[0073] Example 1

[0074] This embodiment provides a producer-consumer parallel architecture based on the aforementioned humanoid robot control optimization method, enabling real-time interaction and control strategy optimization between the humanoid robot and its simulation environment, thereby improving the motion control performance of the humanoid robot. Figure 6 As shown, in the control loop, the producer thread is responsible for interacting with the environment (real or simulated), collecting the robot's current state (such as joint angles, speed, IMU data, etc.), and storing the experience data in a shared buffer pool. Subsequently, the consumer thread samples data from the buffer pool to train and update the Q network, policy network (Actor network), and temperature network in the CAC control algorithm, thereby generating better control commands (such as torque or speed of each joint). These commands drive the robot's actual movement through the actuator interface layer, forming a closed-loop control process, and continuously iterating and optimizing in high-frequency loops to improve the stability, adaptability, and autonomous decision-making ability of the robot's movement.

[0075] S1. After the Humanoid robot control system starts, it initializes, including: initializing the Actor network (policy network) so that it can output action distribution according to the current state; initializing the Critic network (N networks) to estimate the value of state-action pairs; initializing the temperature network to adjust the balance between exploration and exploitation; initializing the target network parameters to ensure the stability of the training process; clearing the experience replay buffer and setting priority weights to support efficient sample replay; setting hyperparameters such as learning rate, entropy coefficient, update frequency (UTD), batch size, and network parameter thresholds to further ensure training efficiency and policy performance. These parameters are updated at a slower rate during training to stabilize the learning process of the Humanoid robot's motion.

[0076] S101. When controlling the Humanoid robot in MuJoCo simulation, the Actor network parameters ψ are first initialized. The function of this policy network is to output the distribution of joint motion actions a based on the current joint states s of the Humanoid robot. Initializing ψ aims to give the policy network a random initial parameter, enabling the Humanoid robot to have learning capabilities.

[0077] S102. Initialize the temperature network parameter α. The temperature network parameter α is used to control the degree of policy exploration. In the output distribution of the policy network, a higher α value makes the Humanoid robot's action distribution more uniform and its exploration more active; a lower α value makes the policy more inclined to output high-probability actions. Initializing α is to give the temperature network an initial parameter value and adjust the balance between exploration and utilization during the subsequent learning process of the Humanoid robot.

[0078] S103, Initialize N Qs function Network parameters φ i , i = 1, ..., N. Q function The network is used to estimate the expected cumulative reward of a Humanoid robot taking a motion action a in joint state s. These network parameters φ are initialized. i Make each Q function Networks can learn the value of state-action pairs (transitions), and multiple networks can provide different value estimation perspectives, assisting algorithms to more accurately evaluate the quality of actions. Humanoid robots improve their movement by continuously learning high-value actions during the learning process.

[0079] S104. Clear the replay buffer D. The experience replay buffer stores transition samples generated during the interaction between the Humanoid robot and the simulation environment, including information such as state, action, reward, and next state. It is cleared and initialized before the algorithm begins a new iteration to prevent old experiences from interfering with the new learning process.

[0080] S105, Initialize the priority experience replay weight p t Priority experience replay assigns weights to experiences based on their importance (e.g., time difference error, TD-error), and initializes the weights p. t This is to give each experience an initial measure of importance. In the motion control of the Humanoid robot, as learning progresses, those experiences that are more critical to motion control (such as experiences of standing stably or moving rapidly) will be given higher weight.

[0081] S106. Initialize N Q's. function Target network parameters v i,target The target network is used to stabilize the training process, and its structure is similar to Q. function The network is the same, but the parameters update more slowly. Initialize φ. i,target To ensure the target network can provide a relatively stable reference target at the beginning, the parameters of the target network are usually set initially to be similar to those of Q. function With identical network parameters, the target network can provide a value based on the current Q during the first learning iteration. functionThe initial target value of the network. This helps the Humanoid robot have a stable value estimation basis in the initial stage, thus allowing it to start the learning process more smoothly.

[0082] S107. Initialize the neural network parameter weight threshold β. This threshold β is used to determine whether the neural network parameters are outside the normal range. During training, the parameters may become too large or too small due to problems such as gradient explosion or vanishing gradients. Initializing β aims to detect whether the parameters exceed the threshold range in a timely manner during subsequent training. If they do, a response mechanism—neuron parameter initialization—is triggered to ensure normal network training. In Humanoid robot control, this can prevent joint motion strategies from exhibiting extreme or unexecutable movements due to abnormal parameters.

[0083] S108. Initialize target entropy H T The value is -dimA. dimA is the action space dimension, and the target entropy H is... T This serves as a guide for optimizing robot control. For example, if a Humanoid robot has a motion space dimension of 4 (corresponding to 4 independently movable joints), then the initial target entropy is -4. The target entropy acts as a reference for the temperature network's learning, enabling the policy to select actions under certain constraints (such as approaching the target entropy), balancing exploration and utilization. This helps the Humanoid robot maintain motion diversity in motion control, allowing it to explore new movement methods while utilizing known effective actions.

[0084] S109. Define the mini-batch size, which is a parameter used to sample experience from the replay buffer for network training, representing the number of experiences sampled for each network update. Set the batch size to 256, randomly selecting 256 experiences from the replay buffer to update the network parameters. Setting an appropriate batch size is crucial for balancing training efficiency and model update stability. In Humanoid robot motion control, this means randomly selecting 256 experiences from a large stored pool to update the network parameters each time, ensuring both data diversity and improved training efficiency.

[0085] S110. In Humanoid robot control, initialize the target network soft update coefficient ρ. The soft update coefficient ρ controls the update magnitude of the target network parameters. Each update increments the current Q value by a proportion of ρ. function The network parameters are merged into the target network parameters. The initial update ratio ρ is determined by initializing ρ to 0.001. Each update of the target network parameters absorbs the current Q... function A change of 0.001 in network parameters helps the target network adapt slowly to new motion strategies, thus providing a more stable reference target for updating the policy network.

[0086] S111. Initialize the update rate UTD. UTD represents the number of times a set of state transition tuples is used, that is, the number of times it is used to update network parameters during a training process; it is initialized to k, that is, UTD = k. Setting UTD appropriately can balance data utilization efficiency and network update frequency.

[0087] Initialize the temperature network learning rate λ α . λ α The step size is determined when updating the temperature network parameter α, which is used to determine the update magnitude of the temperature network parameter α. Initialize λ. α Given the initial learning step size λ of the temperature network. α The initial value is 0.0003, indicating that the temperature network parameters move in 0.0003 steps in the gradient direction with each update. This allows the temperature parameters to slowly adjust the Humanoid robot's exploration behavior.

[0088] S112, Initialize the policy network learning rate λ ψ . λ ψ The update magnitude of the network parameter ψ of the control policy. Initialization of λ. ψ Given the initial step size λ for the policy network learning. ψ Initialized to 0.0003, the policy network parameters are updated in steps of 0.0003 along the gradient direction of the policy network loss function. The learning rate affects the learning speed and stability of the policy network; too large a rate leads to excessively rapid updates and unstable learning, while too small a rate results in excessively slow learning. In other words, it controls the adjustment speed of the Humanoid robot's motion policy, enabling it to smoothly learn a better motion pattern.

[0089] S113. Initialize the target network learning rate λ φ . λ φ Used to control target network parameters Q function Target network parameters φ i,target The new amplitude ensures that the target network can track Q at an appropriate speed. function Network parameter changes provide a stable target value for the target algorithm, λ. φ Initialized to 0.0003, the target network parameters are updated in steps of 0.0003 according to the gradient direction of the target network loss function each time, providing a more accurate and stable target value estimate for the update of the motion strategy.

[0090] S114, Initialize Q function Number of networks M. M is updating Q. function When calculating the target value using the network, if M is initialized to 2, then two Q values ​​are considered when calculating TD-target. functionThe minimum value in the target network. Initializing M helps avoid overestimating the value of certain motion actions, thus enabling the Humanoid robot to more accurately evaluate the merits of different motion strategies and learn better motion control strategies.

[0091] S2. The Humanoid robot interacts with the simulation environment, selects actions based on the current policy, and executes those actions. During this process, the Humanoid robot collects observation data, executed actions, rewards received, and new state information. This data is then stored in an experience replay buffer for subsequent neural network training, enabling the algorithm to learn from past experiences and improve learning efficiency and stability.

[0092] S201, Action Selection and Execution, such as Figure 2 :

[0093] At each time step t, the Humanoid robot selects action a based on the current policy π (determined by the Actor network parameter ψ). t Specifically, from the policy network in joint states s t The output distribution π(·|s) under t Action a is obtained by sampling from ) t The input to this policy network is the current state s of each joint of the Humanoid robot. t (e.g., the angles and velocities of joints such as the hip, knee, and ankle), the output is the probability distribution of different actions taken in this state (e.g., the probability of actions such as leg extension, arm swing, or trunk twist). Then, the selected action a is executed. t This propels the Humanoid robot into a new state. t+1 For example, it can perform a leg-extending motion, causing the Humanoid robot to take a step forward.

[0094] S202 Data Collection:

[0095] In the environment, action a t After responding, the Humanoid robot collects the following key information:

[0096] Observational data: New joint states t+1 (such as new angles and speeds in joints like the hip, knee, and ankle).

[0097] Reward value r t The simulation environment is based on the current joint states of the Humanoid robot. t Next, execute action a t The given reward value r tThe reward reflects the quality of the Humanoid robot's current behavior; for example, a positive reward is given if the Humanoid robot stands steadily, and a negative reward is given if it falls over.

[0098] Current joint state s t and action a t Record the action selected when choosing action a. t The state of each joint at that time s t and the action a performed t This information itself constitutes an important part of the experience replay sample.

[0099] Experience replay sampling weight p t Based on the time difference error (TD-error) and reward r t Determine the empirical playback sampling weight p t This weight is crucial in subsequent experience replay sampling, determining the probability of a sample being selected. Higher-weighted samples receive more attention during training, helping the Humanoid robot focus on more valuable data for learning (such as stable and fast-moving motion experiences), thus improving learning efficiency. This is determined by the temporal difference error (TD-error) and the reward r. t Determine the empirical playback sampling weight p t Specifically:

[0100]

[0101] Where TD-error is the time difference error, and y is the target value of the time difference. Is the i-th Q function Network for joint states s t and action a t The estimated value, ε, is a non-zero positive number used to provide a non-zero lower bound for the TD-error of all samples, ensuring that each experience has a non-zero probability of being selected in the sampling, r t The reward is the amount received, and softmax() is the normalization function.

[0102] S203. Construct and store the transition pairs:

[0103] Collect the current joint states s t Action a to be performed t The reward r obtained t New joint states t+1 and the empirical replay sampling weight p t Integrate into a single transition pair (s t ,at ,r t ,s t+1 ,p t (State transition tuple). Then, add this transition pair to the experience replay buffer D, i.e., D←D∪(s t ,a t ,r t ,s t+1 ,p t The experience replay buffer is used to store a large number of historical data samples generated during the robot's interaction with the environment. These samples will be repeatedly used in the subsequent neural network training process to help the Humanoid robot learn and summarize experience, thereby continuously optimizing the strategy and value function estimation.

[0104] S3. Perform neural network updates by randomly sampling mini-batches of samples from the experience replay buffer. These samples are used to calculate the TD error, and the parameters of the Actor and Critic networks are updated accordingly. The Actor network update aims to optimize the policy to maximize the cumulative reward, while the Critic network update aims to more accurately estimate the value of each state-action pair.

[0105] S301, Sampling, such as Figure 3 :

[0106] According to p t Perform sampling of a batch size B from the experience replay buffer D, according to the sampling weight p. t A small batch of samples of size B related to the motion control of the Humanoid robot is selected to form a small batch sample set B.

[0107] S302, Forward Propagation and Parameter Weight Determination in Neural Networks:

[0108] A neural network forward propagation is performed on the sampled mini-batch set B. During the forward propagation, it is determined whether the parameter weights exceed a threshold β. If they do, neuron initialization is performed to ensure that the network parameters are within a reasonable range and to maintain stable network training. Specifically, neuron initialization involves initializing the neural network parameters and initializing the weights of parameters that exceed the preset threshold β.

[0109] S303, TD target (TDtarget, time difference target) calculation, such as Figure 4 :

[0110] Randomly select M Q function The minimum value of the target network is used to calculate the TD target. The specific calculation formula is as follows:

[0111]

[0112] Where r is the reward value (e.g., the reward the Humanoid robot receives for successfully completing a designated task), and γ is the discount factor (e.g., setting it to 0.9 indicates a certain discount on future rewards). Represents the i-th Q function Target network, s t+1 and a t+1 These refer to the new state (such as new joint angles and velocities) and the action (such as new changes in joint motion), where α is the temperature parameter and π is the temperature parameter. ψ (·∣s t+1 ) indicates that in state s t+1 The output distribution of the policy network.

[0113] S304, Update Q function Network parameters:

[0114] Update Q according to the following formula function Network parameter φ i (i = 1, ... N):

[0115]

[0116] in, It is Q function The learning rate of the network The loss function J(φ) represents i ) for φ i gradient, Is the i-th Q function Network on state s t and action a t Let y be the estimated value, TDtarget value, and B be the sampled mini-batch set. In this way, gradient descent is used to minimize the difference between the estimated value and the TDtarget value, thereby optimizing Q. function Network parameters.

[0117] S305, Update Q function Target network parameters:

[0118] Update Q using a soft update strategy function Target network parameters φ target,i The formula is:

[0119] φ target,i ←ρφ target,i +(1-ρ)φ i

[0120] Where ρ is the target network soft update coefficient, controlling the magnitude of the target network parameter update. This allows the target network parameters to gradually update towards the current Q while maintaining relative stability. function The convergence of network parameters stabilizes the entire training process, providing a more reliable estimate of target value for updating the motion strategy of the Humanoid robot.

[0121] S306. Update the sampling weights for experience replay:

[0122] According to the updated Q function Network parameters, update the sampling weights p of the experience replay buffer. t That is, p t ←p t+1 The new sampling weight p t+1 This reflects the importance of the updated experience samples, enabling more rational selection of samples that are more valuable for learning in subsequent sampling processes.

[0123] S307, Update the temperature network, such as Figure 5 :

[0124] Update the temperature network parameter α according to the following formula:

[0125]

[0126] Where, λ α It is the temperature network learning rate. H represents the gradient of the loss function J(α) with respect to α. t It is the target entropy, π ψ (a t |s t ) indicates that in state s t Choose action a t The probability of this is determined by optimizing the temperature network parameter α. This adjusts the probability distribution of the policy network output, enhancing the Humanoid robot's exploratory capabilities, balancing exploration and exploitation, and improving the algorithm's global optimization ability and decision diversity. For example, if the current policy entropy is -6, which is less than the target entropy of -5, the temperature parameter α will increase, making the policy network's output distribution more uniform. This enhances the Humanoid robot's exploratory nature, potentially leading it to try new movement patterns, such as changing its oscillation frequency or amplitude.

[0127] S308, Update policy network parameters:

[0128] Using the updated parameters, update the policy network parameters ψ according to the following formula:

[0129]

[0130] Where, λ ψIt is the learning rate of the policy network. Let J(ψ) represent the gradient of the loss function with respect to ψ. This update process aims to optimize the policy network so that, under certain entropy constraints, it can maximize the expected cumulative reward, thereby improving the performance of the policy. For example, if the parameters of the current policy network make the Humanoid robot's action selection too limited in some situations, making it unable to effectively explore the environment, then by updating the parameters, it can be made to favor actions that yield more rewards, such as more stable and faster movements, while maintaining a certain diversity of actions. N is the number of Q-networks. It is the network parameter φ i The corresponding Q network, π ψ (a t |s t ) is in state s t Choose action a t The probability of.

[0131] S309, Cyclic Update:

[0132] Steps S302 to S307 are repeated k times, meaning each mini-batch sample B is used to update the network parameters k times, followed by step S308. This fully extracts information from these experience samples, enabling the Humanoid robot to learn the optimal motion control strategy more effectively.

[0133] S4. After completing step S3 above, continue with the next iteration, that is, return to step S2 to interact with the simulation environment, and repeat this process continuously so that the Humanoid robot can continuously optimize its strategy and value estimation in continuous interaction with the simulation environment, thereby achieving better control.

[0134] To verify the effectiveness of the deep reinforcement learning-based robot control optimization method in this embodiment, it was implemented in the MuJoCo simulation environment using a Humanoid robot model. The algorithm was compared with several other popular reinforcement learning algorithms to demonstrate its performance in robot control tasks.

[0135] This implementation compares five commonly used robot control algorithms—TD3, SAC, REDQ, TQC, and TQC20—with the Continuous Actor Critics Algorithm (CAC). All algorithms were run under the same simulation environment and parameter settings to ensure the fairness and comparability of the experimental results. We used reward as the evaluation metric, specifically the cumulative reward the robot receives for completing tasks in the simulation environment.

[0136] First, all network parameters for the algorithms, such as learning rate, discount factor, and target entropy, are initialized. This implementation allows the robot to interact with the environment in a simulation, selecting and executing actions based on the current policy. During this process, the robot collects observation data, actions, rewards, and new state information, storing them in an experience replay buffer.

[0137] During the neural network update phase, this implementation randomly extracts a small batch of samples from the experience replay buffer. These samples are used to calculate the TD error, and the parameters of the Actor network and Critic network are updated accordingly. This process is repeated until a preset number of training steps is reached.

[0138] This implementation periodically records the average reward for each algorithm at different time steps. This data is used to evaluate the algorithm's performance and performance curves are plotted as follows: Figure 7 And a comparison chart of average rewards for different time step intervals, as shown in the figure. Figure 8 By analyzing these graphs, we can observe the performance of different algorithms during the training process, including their convergence speed, stability, and final performance.

[0139] Experimental results show that the proposed algorithm exhibits rapid learning and convergence characteristics during training. Figure 7 It can be seen that, compared with other algorithms, the CAC algorithm can reach a higher average reward faster and remain stable over a long period of time. Figure 8 This advantage is further confirmed, with the CAC algorithm showing significantly higher average rewards than other algorithms across all time steps (especially in the early training phase). This demonstrates that the CAC algorithm can effectively handle complex, dynamic, and nonlinear control problems, providing a novel solution for robot control.

[0140] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A robot control optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Initialize robot control system parameters and clear the experience playback buffer pool; S2. The robot control system generates an action according to the Actor network policy function. After executing the action, it collects environmental feedback information and confirms the experience replay sampling weight. It integrates the environmental feedback information and the experience replay sampling weight into a state transition tuple as an experience sample. The experience sample is written into the experience replay buffer pool. S3. Extract samples from the experience replay buffer pool, determine the time difference error based on the samples, update the network parameters based on the error value to minimize the value estimation bias, and simultaneously update the Actor network parameters and the Critic network weights to maximize the expected value of the state-action value function. S4. Repeat S2 to S3 to optimize robot control.

2. The robot control optimization method based on deep reinforcement learning according to claim 1, characterized in that, The robot control system parameters in S1 include Actor network parameters, Critic network weights, temperature network parameters, hyperparameters, and target network parameters. Among them, the Actor network parameters include the policy network parameter ψ, which is the policy function used to generate actions. Critic network weights, including N Q-factors function Network parameter φ i , i = 1, ..., N; φ i The expected cumulative reward for the robot to take motion action a in joint state s; The temperature network parameter α is used to adjust the diversity of exploration behaviors; The target network parameters include N Q parameters. function Target network parameters φ i,target The Q function Target network parameters φ i,target Used to provide a stable estimate of the target value.

3. The robot control optimization method based on deep reinforcement learning according to claim 2, characterized in that, In S1, the hyperparameters include: The target network soft update coefficient ρ is used to control the update speed of the target network parameters; Mini-batch size (batch) is used to set the number of samples used in each training session. The update rate UTD is used to determine the number of times network parameters are updated during a single training session. Priority experience replay weight p t This is used to adjust the sampling probability of samples in the experience replay buffer; The weight threshold β of the neural network parameters is used to control the conditions or range for updating the network parameters. Temperature network learning rate λ α , used to determine the update magnitude of the temperature network parameter α; Policy network learning rate λ ψ , used to control the update magnitude of the policy network parameter ψ; Target network learning rate λ φ Used to control the target network parameter Q function Target network parameters φ i,target New range; Target entropy H T Initialize target entropy H T = -dimA, where dimA is the action space dimension and the target entropy H is... T Used to guide the direction of robot control optimization; Q function Number of networks M, in updating Q function The network is used to calculate the target value.

4. The robot control optimization method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of S2 includes: S201, at each time step t, the policy network at the current joint s t Output distribution π(·|s) under the state t Action a is obtained by sampling from ) t And perform the action to push the robot into a new joint state. t+1 ; S202, In the context of action a t After responding, the robot collects environmental feedback information, including new joint states. t+1 Reward value r t and the robot's current joint state s t and action a t Simultaneously, based on the time difference error and the reward value r t Determine the empirical playback sampling weight p t And store; S203, The collected environmental feedback information and experience playback sampling weight p t Integrate into a state transition tuple (s) t ,a t ,r t ,s t+1 ,p t Then it is written to the experience replay buffer pool.

5. The robot control optimization method based on deep reinforcement learning according to claim 4, characterized in that, The empirical playback sampling weight p t The method for determining it is as follows: Where TD-error is the time difference error, and y is the target value of the time difference. Is the i-th Q function Network for joint states s t and action a t The estimated value, ε, is a non-zero positive number used to provide a non-zero lower bound for the TD-error of all samples, ensuring that each experience has a non-zero probability of being selected in the sampling, r t It represents the reward value obtained, and softmax() is the normalization function.

6. The robot control optimization method based on deep reinforcement learning according to claim 1, characterized in that, The specific process of S3 includes: S301. From the experience playback buffer D, according to the sampling weight p t Samples related to robot motion control are selected to form a sample set B, with a sample size of batch. S302. Determine the forward propagation and parameter weights of the neural network; perform forward propagation of the neural network on the sample set B obtained by sampling, and determine whether the parameter weights exceed the preset threshold β during the forward propagation process; if they exceed the threshold β, perform neuron initialization operation. S303. Determine the time difference target value y: Where, r t γ is the reward value, and γ is the discount factor. Represents the i-th Q function Target network, s t+1 a t+1 These represent the new joint state and motion, respectively; α is the temperature network parameter; and π... ψ (·∣s t+1 ) indicates that in state s t+1 Output distribution of the lower policy network; S304. Update the Critic network weights; S305, Update target network parameters; S306. Update the empirical replay sampling weights based on the updated target network parameters; S307, Update temperature network parameters; S308, Update policy network parameters; S309. The parameters of the Actor network, the weights of the Critic network, the parameters of the temperature network, the hyperparameters, and the parameters of the target network are updated cyclically, with an update rate UTD.

7. The robot control optimization method based on deep reinforcement learning according to claim 5, characterized in that, S304, which updates the Critic network weights, specifically involves updating the Q value in the Critic network weights. function Network parameter φ i : in, It is Q function The learning rate of the network, The loss function J(φ) represents i ) for φ i gradient, Is the i-th Q function Network on state s t and action a t The estimated value is y, which is the time difference target value, B is the sample set obtained by sampling, and s is a sample in the sample set B.

8. The robot control optimization method based on deep reinforcement learning according to claim 5, characterized in that, The S305 update of the target network parameters specifically involves updating Q using a soft update strategy. function Target network parameters φ target,i : f target,i ←rf target,i +(1-r)φ i Where ρ is the target network soft update coefficient, φ i Q is the weight in the Critic network. function Network parameters.

9. The robot control optimization method based on deep reinforcement learning according to claim 6, characterized in that, The method for updating the temperature network parameter α in S307 is as follows: Where, λ α It is the temperature network learning rate. H represents the gradient of the loss function J(α) with respect to α. t It is the target entropy, π ψ (a t |s t ) indicates that in state s t Choose action a t The probability is given by s, where s is a sample in the sample set B, and B is the sample set obtained by sampling.

10. The robot control optimization method based on deep reinforcement learning according to claim 6, characterized in that, The method for updating the policy network parameter ψ in S308 is as follows: Where, λ ψ It is the learning rate of the policy network. Let s represent the gradient of the loss function J(ψ) with respect to ψ, s be a sample in the mini-batch sample set B, B be the sample set obtained from sampling, α be the temperature network parameters, and N be the Q value. function Number of target networks Q represents function Network, π ψ (a t |s t ) is in state s t Choose action a t The probability, where i is the index symbol.

Images