Multi-joint robot motion posture adaptive control method and system based on deep reinforcement learning

Abstract

The application discloses a kind of multi-joint robot motion posture adaptive control method and system based on deep reinforcement learning, belongs to robot intelligent control technical field.The method includes: constructing state space and action space;Double policy network architecture consisting of main policy network and auxiliary policy network is constructed, both share value network;Design layered reward function, set the difference reward of stability and disturbance adaptation for main and auxiliary policy respectively;Adaptive disturbance compensation module is constructed to estimate external disturbance and generate compensation torque online;Adopt near-end policy optimization algorithm combined with collaborative constraint factor to train network;Main policy network is deployed and selectively accesses auxiliary policy network to carry out online inference.The application combines double policy collaborative decision and disturbance compensation, effectively improves the motion posture stability and adaptive capacity of multi-joint robot under complex terrain and external disturbance.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control technology, specifically relating to an adaptive control method for the motion posture of a multi-joint robot based on deep reinforcement learning, which is suitable for stable motion control of multi-joint robots under complex terrain and external disturbance conditions. Background Technology

[0002] With the rapid development of robotics technology, articulated robots (including bipedal humanoid robots, quadruped robots, and multi-degree-of-freedom robotic arms) are increasingly widely used in industrial production, disaster relief, military reconnaissance, and domestic services. However, articulated robots face the following technical challenges in practical deployment:

[0003] (1) Poor terrain adaptability. Traditional control methods (such as Model Predictive Control (MPC) and Zero Moment Point (ZMP) control) rely on accurate dynamic models. Their control performance drops significantly under unknown or changing terrain conditions, making it difficult to guarantee the robot's posture stability on rugged ground, slopes, steps and other complex terrains.

[0004] (2) Insufficient anti-disturbance capability. During the movement of the robot, it is often subjected to disturbances such as external impact, load change, and ground friction coefficient change. Traditional fixed parameter controllers lack adaptive adjustment capability and are difficult to quickly restore a stable posture after the disturbance occurs.

[0005] (3) It is difficult to balance multiple conflicting objectives. Robot posture control needs to simultaneously consider multiple conflicting objectives such as stability, energy efficiency, joint torque smoothness, and trajectory tracking accuracy. Traditional heuristic methods are difficult to achieve effective collaborative optimization of multiple objectives.

[0006] (4) High-dimensional continuous control is difficult. Multi-joint robots usually have dozens of degrees of freedom, with high dimensionality and strong continuity in their motion space. Traditional reinforcement learning methods are difficult to apply directly due to the discretization of the motion space, while existing continuous control algorithms still have shortcomings in terms of training stability and sample efficiency.

[0007] In recent years, deep reinforcement learning has shown great potential in the field of robot control, among which the proximal policy optimization (PPO) algorithm has attracted widespread attention due to its training stability and good continuous control performance. However, the standard PPO algorithm still has the following limitations when applied to the posture control of multi-joint robots: a single policy network cannot simultaneously take into account stable walking and perturbation response; the exploration efficiency decreases as training progresses, and it is prone to getting trapped in local optima; and there is a lack of online estimation and compensation mechanisms for external perturbations.

[0008] Therefore, there is an urgent need for a motion posture control method for multi-joint robots that can balance stability, adaptability, and multi-objective optimization. Summary of the Invention

[0009] This invention addresses the shortcomings of existing multi-joint robot posture control methods in terms of terrain adaptability, disturbance resistance, and multi-objective optimization by providing a deep reinforcement learning-based adaptive motion posture control method for multi-joint robots. This method achieves stable and efficient motion control of the robot under complex terrain and external disturbance conditions by constructing a dual-policy network architecture and an adaptive disturbance compensation module.

[0010] To achieve the above objectives, the following technical solution is adopted:

[0011] In a first aspect, the present invention provides an adaptive control method for the motion posture of a multi-joint robot based on deep reinforcement learning, comprising the following steps:

[0012] S1: Construct the state space and action space for the motion posture control of a multi-joint robot. The state space includes the robot joint space state and the environmental perception state. The action space is defined as the target angle increment of each joint.

[0013] S2: Construct a dual-strategy network architecture, which includes a primary strategy network and an auxiliary strategy network, wherein the primary strategy network and the auxiliary strategy network share the same value network;

[0014] S3: Design a hierarchical reward function, setting different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation;

[0015] S4: Construct an adaptive disturbance compensation module to estimate external disturbances online and generate compensation torque. The compensation torque is then superimposed on the basic control torque output by the policy network to form the final control torque applied to each joint of the robot.

[0016] S5: Employ the near-end policy optimization algorithm and train the main policy network, the auxiliary policy network, and the value network based on the hierarchical reward function. Introduce a collaborative constraint factor during training to coordinate the update step size of the main policy network and the auxiliary policy network.

[0017] S6: Deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis based on the detected external disturbance intensity.

[0018] Secondly, the present invention also provides a multi-joint robot motion posture adaptive control system based on deep reinforcement learning, comprising:

[0019] The state perception module is used to collect the robot's state information and construct the state space and action space for the motion posture control of the multi-joint robot. The state space includes the robot's joint space state and the environmental perception state, and the action space is defined as the target angle increment of each joint.

[0020] The dual-strategy network module includes a primary strategy network and an auxiliary strategy network, which share the same value network. The primary strategy network aims to generate stable gait, while the auxiliary strategy network aims to adjust perturbation response and terrain adaptability.

[0021] The hierarchical reward function module is used to set different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation.

[0022] An adaptive disturbance compensation module is used to estimate external disturbances online and generate compensation torques.

[0023] The training module is used to train the main policy network, the auxiliary policy network and the value network based on the hierarchical reward function using a proximal policy optimization algorithm. During the training process, a collaborative constraint factor is introduced to coordinate the update step size of the main policy network and the auxiliary policy network.

[0024] The online inference and control module is used to deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis according to the intensity of the detected external disturbance. The basic control torque output by the policy network is superimposed with the compensation torque to form the final control torque applied to each joint of the robot.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) Significantly improved terrain adaptability: Through a dual-policy network architecture, the main policy network learns a stable basic gait, while the auxiliary policy network makes adaptive adjustments specifically for terrain changes and obstacles. The two work together to enable the robot to maintain a stable motion posture in various scenarios such as flat ground, slopes, steps, and rugged terrain. Under typical simulation verification conditions, this method can obtain a better average cumulative reward than the standard PPO algorithm and shows better adaptability in complex terrain.

[0027] (2) Enhanced anti-disturbance capability: The adaptive disturbance compensation module estimates external disturbances in real time based on the generalized momentum method and generates compensation torque, enabling the robot to effectively recover a stable posture after being subjected to external shocks, load changes and other disturbances. Under typical simulation verification conditions, the controller equipped with the disturbance compensation module has a significant advantage in posture recovery time under pulse-type external disturbances compared to the baseline configuration without the module.

[0028] (3) Achieve multi-objective collaborative optimization: The hierarchical reward function incorporates multiple objectives such as stability, energy efficiency, torque smoothness, and gait symmetry into a unified optimization framework, avoiding the subjectivity and limitations of manual weighting of multiple objectives in traditional methods, enabling the robot to achieve energy-efficient movement while ensuring stability.

[0029] (4) High training stability: The collaborative constraint factor, combined with the pruning mechanism of PPO, effectively constrains the update step size of the dual-policy network, avoiding the performance collapse problem caused by excessive policy updates in the standard PPO algorithm. The shared value network, combined with reward normalization processing, reduces the computational cost of training and improves the efficiency of sample utilization.

[0030] (5) Online adaptive capability: The deployed policy network, in conjunction with the disturbance compensation module, can adapt to unknown terrain and sudden disturbances without retraining, exhibiting good generalization ability and real-time performance. The typical industrial robot control cycle is on the order of 1 to 10 milliseconds, and the policy network inference latency of this method can meet this requirement in actual tests on embedded platforms with GPU acceleration capabilities.

[0031] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0032] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0033] Figure 1 This is a flowchart illustrating the steps of an adaptive control method for the motion posture of a multi-joint robot based on deep reinforcement learning, according to an embodiment of the present invention.

[0034] Figure 2 This is a diagram of the overall system architecture of the present invention;

[0035] Figure 3 This is a flowchart of the dual-strategy network training process according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of a module of an adaptive control system for the motion posture of a multi-joint robot based on deep reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Example 1:

[0040] Figure 1 This is a flowchart illustrating the steps of an adaptive control method for the motion posture of a multi-joint robot based on deep reinforcement learning, according to Embodiment 1 of the present invention. Figure 2 This is the overall process framework diagram. Figure 2 This paper demonstrates the complete control process from state awareness, dual-policy network decision-making, disturbance compensation to torque synthesis and application, as well as the transmission path of environmental feedback reward signals during the training phase. Specifically, a method for adaptive motion posture control of a multi-joint robot based on deep reinforcement learning includes the following steps:

[0041] S1: Construct the state space and action space for the motion posture control of a multi-joint robot. The state space includes the robot joint space state and the environmental perception state. The action space is defined as the target angle increment of each joint.

[0042] Step S1: Construct the state space and motion space for the motion posture control of the multi-joint robot.

[0043] The state space includes the robot's joint space state and its environmental perception state. Specifically, at time... The state vector is defined as:

[0044]

[0045] in, This represents the current angle position of each joint. The number of degrees of freedom of the robot's joints; For the angular velocity of each joint; The joint torque applied at the previous moment; The output is the action from the previous moment; This is the position of the robot's center of mass. The linear velocity of the center of mass; The angular velocity of the center of mass; Projected along the direction of gravity; Information on contact force at the sole or distal end of the foot. This refers to the number of contact force sensors; This represents the total dimension of the state space.

[0046] The motion space is defined as the target angle increment for each joint. The motion vector is:

[0047]

[0048] in, For the first The target angle increment of each joint Actual applied joint torque Calculated using the impedance control law:

[0049]

[0050] in, For the target joint angle, and The first The proportions and differential gain coefficients of each joint.

[0051] :time The motion vector includes the target angle increments for each joint; : No. The target angle increment of each joint; Number of degrees of freedom of the robot's joints; : Joint index, with a value range of 1 to ; : No. The actual torque applied to each joint; : No. The proportional gain coefficient of each joint; : No. Differential gain coefficients of each joint; : No. The target angle of each joint is equal to the sum of the current angle and the target angle increment; : No. The current angle position of each joint; : No. The current angular velocity of each joint.

[0052] S2: Construct a dual-strategy network architecture, which includes a primary strategy network and an auxiliary strategy network, wherein the primary strategy network and the auxiliary strategy network share the same value network;

[0053] Step S2: Design a dual-strategy network architecture, including a primary strategy network and an auxiliary strategy network, both of which share the same value network.

[0054] like Figure 3 The diagram shown illustrates the training flowchart of the dual-policy network of this invention, demonstrating the complete training process from network parameter initialization, hierarchical reward calculation, advantage estimation, PPO loss optimization to joint fine-tuning and eventual convergence. Key training steps of this invention are highlighted in bold red boxes. (Main Policy Network) With stable gait generation as the primary objective, parameterization is performed as follows: Auxiliary policy network With disturbance response and terrain adaptation as the main objectives, parameterization is performed as follows: .

[0055] Both policy networks employ a multilayer perceptron architecture, comprising an input layer, hidden layers, and an output layer. The hidden layers use modified linear units (MRUs) or Gaussian error linear units (GAUs) as activation functions, while the output layer uses a hyperbolic tangent function to map actions to the effective range.

[0056] The shared value network The function used to estimate state values ​​is parameterized as follows: The system is trained using mixed trajectory data generated by two policy networks. To reconcile value estimations under two different reward scales, the two reward signals are normalized separately before training the shared value network.

[0057] The two policy networks employ a collaborative update mechanism during training, defining the first... Probability ratio of each policy network for:

[0058]

[0059] in, For the first The old policy before the policy network update. j=1 is the primary policy, and j=2 is the secondary policy.

[0060] To coordinate the update step size of the two policy networks, a cooperative constraint factor is introduced. :

[0061]

[0062] in, The probability ratio of the main policy network at time t. This represents the probability ratio of the auxiliary policy network at time t.

[0063] The design principle of this constraint factor is as follows: when the update rate of any policy network exceeds the pruning interval, a minimum operation is used to restrict the overall update step size, in accordance with the trust region constraint mechanism of PPO. This min operation ensures that the collaborative constraint term participates in gradient propagation only from the side with the smaller update rate—that is, when... hour, On this sample The gradient contribution of one policy network is zero, and vice versa. The gradient updates of the two policy networks at each data point are gated by the ratio of the other, forming an asymmetric update constraint mechanism: the same data point can only provide an effective constraint term gradient for the party with the more conservative update ratio, and the gradient of the other party on that term is not propagated. This coupling mechanism enables the two policy networks to mutually limit the update magnitude during joint optimization, preventing the single policy network from having an excessively large single-step update due to the high variance of the perturbation task during training, which would consequently affect the stability of the state distribution on which the shared value network depends.

[0064] To verify the independent contribution of the collaborative constraint factor itself, two ablation configurations were set up for comparison: (1) dual-policy network + the collaborative constraint factor of this invention (i.e., complete coupling mechanism); (2) dual-policy network but each is independently pruned and not shared. (That is, the two policy networks independently calculate the probability ratios and perform pruning separately, with no coupling of update amplitude between them). Comparative experiments show that the reward curve fluctuation amplitude of configuration (1) during training is significantly smaller than that of configuration (2), and the standard deviation of the policy network parameter update step size is significantly reduced, indicating that the collaborative constraint factor has an effective role in suppressing the instability of policy network updates.

[0065] S3: Design a hierarchical reward function, setting different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation;

[0066] Step S3: Design a hierarchical reward function and set differentiated reward signals for the main policy network and the auxiliary policy network respectively.

[0067] The primary and secondary strategies set differentiated reward signals for gait stability and perturbation / terrain adaptability, respectively. They are coupled and trained through a shared value network and collaborative constraint factors, so that the two strategies mutually constrain each other at the gradient level, rather than optimizing each other independently.

[0068] Reward function of the main policy network Emphasis on gait stability and energy efficiency:

[0069]

[0070] The definitions of each reward item are as follows:

[0071] Weighting coefficient This is an adjustable positive coefficient.

[0072] Progress Rewards Encouraging the robot to move towards the target direction is defined as the decrease in distance to the target point at adjacent time steps:

[0073]

[0074] in, For a moment The target location For a moment The position of the robot's torso.

[0075] Posture Rewards Encourage robots to maintain an upright posture:

[0076]

[0077] in, For a moment Unit vector with torso pointing upwards. The preset threshold value ranges from 0.85 to 0.95. This represents the vector dot product.

[0078] Joint torque smoothing bonus Punishment for drastic changes in joint torque:

[0079]

[0080] in, For a moment No. The torque of each joint.

[0081] Gait symmetry reward item This encourages coordinated movement of the bilaterally symmetrical joints. To avoid conflict between this reward and normal alternating gait, the angles of the bilaterally symmetrical joints are phase-aligned before calculating the difference—based on the gait cycle. Half of the value is the phase offset, compared with the current moment of the left leg joint. The right leg joint corresponding to the time:

[0082]

[0083] in, For a set of symmetrical joint pairs, This is the current gait cycle estimate. For a moment left leg Each joint angle For a moment Right leg Joint angles. Gait cycle. The online estimation method is as follows: Based on the detection of ground contact / remote events of the left and right feet using plantar contact force sensors, the time interval between two consecutive ground contact events of the same side foot is recorded. This time interval is then averaged using a sliding window (the window size is the size of the most recent 5 gait cycles) to obtain the current gait cycle estimate. This estimate is updated each time a ground contact event occurs. In the initial training phase ( Since sufficient historical data has not yet been accumulated, this reward item is set to 0. The window size is taken from the most recent 5 gait cycles, which can effectively filter out noise from single gait cycle measurements and respond promptly to changes in gait cycles, achieving a balance between estimation smoothness and response speed.

[0084] In the initial training phase, before a complete ground-leave-ground event sequence has been detected, the initial value of the gait period T is set to a preset empirical value. (For example =0.5s), during which the gait symmetry reward item is mentioned. Set to 0, and after accumulating enough ground contact event data, switch to online estimation based on sensor data.

[0085] Foot contact bonus Optimize plantar contact force:

[0086]

[0087] in, To support a sufficient quantity, For the first The contact force vector of each supporting foot. This is the contact force threshold.

[0088] Angular velocity tracking bonus :

[0089]

[0090] in, For the target yaw rate, The actual yaw rate of the torso. This is the bandwidth parameter.

[0091] Torque energy consumption penalty term :

[0092]

[0093] Reward function of auxiliary policy network Focus on disturbance response and terrain adaptation:

[0094]

[0095] Among them, the weighting coefficient This is an adjustable positive coefficient.

[0096] In-flight time bonus Encourage appropriate gait cycles to overcome obstacles.

[0097]

[0098] in, and The duration of the left and right feet off the ground, respectively. For indicator functions, and These represent the Boolean events for the first contact of the left and right feet with the ground.

[0099] Contact adaptation reward items Encourage adjusting contact force according to terrain:

[0100]

[0101] in, For the first The reference contact force vector of the supporting foot.

[0102] Disturbance recovery reward items Encourage a rapid return to a stable posture after being disturbed.

[0103]

[0104] in, For a moment Torso attitude angle vector As the reference attitude angle vector, For a moment Torso attitude angular velocity vector.

[0105] S4: Construct an adaptive disturbance compensation module to estimate external disturbances online and generate compensation torque. The compensation torque is then superimposed on the basic control torque output by the policy network to form the final control torque applied to each joint of the robot.

[0106] Step S4: Design an adaptive disturbance compensation module to estimate external disturbances online and generate compensation torque.

[0107] The adaptive disturbance compensation module is designed based on the generalized momentum method. The joint space dynamics equation is defined as follows:

[0108]

[0109] in, The joint angle vector. The joint angular velocity vector. The joint angular acceleration vector. The inertia matrix, The matrix of Coriolis force and centrifugal force. For gravity, To control the torque vector, This is the vector of external disturbance torque.

[0110] Define generalized momentum for:

[0111]

[0112] in, It is a generalized momentum vector.

[0113] A disturbance observer is constructed to feed back the estimation error of generalized momentum to generate an estimated disturbance torque. The estimated disturbance torque is then filtered to generate the compensation torque. The filtering process uses an exponential moving average filter. The compensation torque at the current moment is obtained by weighted summation of the estimated disturbance torque at the current moment and the compensation torque at the previous moment. The final control torque is formed by superimposing the basic control torque output by the policy network and the compensation torque.

[0114] Specifically, the disturbance observer is designed as follows:

[0115]

[0116] in, To estimate the momentum vector, To estimate the disturbance moment vector, This is the observer gain matrix (usually a diagonal matrix). For the generalized momentum estimation error. Observer gain matrix. The range of values ​​for each diagonal element is [5, 20] within the control period sampling rate. This is given under conditions of 1000Hz. When the actual deployed sampling rate differs from the example, it should be adjusted proportionally according to the sampling rate. The value of is chosen to maintain the same dynamic response characteristics of the observer, i.e. .in, The sampling rate of the control cycle used during training or debugging (1000Hz in the example). The sampling rate for the control cycle during actual deployment. The observer gain value is adjusted to match the actual sampling rate.

[0117] The accuracy of the disturbance estimation in this module depends on the accuracy of the robot dynamics model parameters. To enhance the robustness of the module to model errors, the present invention takes the following measures: (1) Observer gain The design needs to strike a balance between estimated response speed and noise suppression, with a value range of [5, 20] (diagonal elements). A lower gain can effectively attenuate the interference of high-frequency model errors on the estimated value; (2) A first-order low-pass filter is connected in series at the compensation torque output terminal, and the cutoff frequency of the filter can be configured according to the actual control requirements. In the preferred embodiment of the present invention, when the control cycle sampling rate At 1000Hz, the cutoff frequency of the first-order low-pass filter is... The value range is from 5Hz to 50Hz.

[0118] Compensating torque Generated from the estimated disturbance after being filtered by exponential moving average:

[0119]

[0120] in, These are the filter coefficients (compensation coefficients). This represents the compensation torque vector from the previous moment. Filter coefficients. With cutoff frequency The conversion relationship is as follows ,in, To control the periodic sampling rate. In a preferred embodiment of the invention (controlling the periodic sampling rate) To obtain a faster disturbance tracking response, take In scenarios requiring stronger noise suppression, a smaller [size] can be selected. Value (e.g.) (The corresponding cutoff frequency at the above sampling rate is approximately 10Hz). Both parameter configurations described above are optional parameters of the same EMA filtering formula under different requirements, and are not cascaded. When the actual deployed control cycle sampling rate differs from the example, it should be adjusted accordingly based on the above conversion relationship. The value is used to maintain the same cutoff frequency characteristics.

[0121] Ultimately, the control torque applied to the robot for:

[0122]

[0123] in, This is the basic control torque vector output by the policy network.

[0124] S5: Employ the near-end policy optimization algorithm and train the main policy network, the auxiliary policy network, and the value network based on the hierarchical reward function. Introduce a collaborative constraint factor during training to coordinate the update step size of the main policy network and the auxiliary policy network.

[0125] Step S5: Train the dual-policy network and the value network using the proximal policy optimization algorithm.

[0126] To improve the robustness of the policy network from simulation to reality, domain randomization technology is introduced during the training phase: ±20% uniformly distributed noise is injected into the robot's environmental dynamic parameters (including robot link mass, center of mass position, and ground friction coefficient); at the same time, random communication delay (1 to 3 control cycles) is introduced into the motion output of the joint motors.

[0127] Define generalized advantage estimation for:

[0128]

[0129] Where L is the total length of the trajectory. As a discount factor, For GAE attenuation parameters, timing differential error Defined as:

[0130]

[0131] in, For a moment Instant rewards received For value network state The value estimate.

[0132] For the two policy networks, calculate their respective advantage estimates. and ,in, Rewards based on master policy network calculate, Rewards based on auxiliary policy networks calculate.

[0133] The pruning / substitution objective functions for the two policy networks are as follows:

[0134]

[0135] in, The main policy network's pruning substitution objective function; The pruning alternative objective function of the auxiliary policy network; : time step The expected value (empirical mean); Take the minimum of the two; Collaborative constraint factor, the probability ratio shared by two policy networks, is defined as follows: ; The main policy network at time 1 Advantage estimation; : The auxiliary policy network at time Advantage estimation; :Will Cut off to interval The clipping function within; : Trimming hyperparameter, value range is .

[0136] The two policy networks share the same collaborative constraint factor. However, their strengths differ.

[0137] The loss function of the value network is:

[0138]

[0139] in, Loss function of value network; Parameters of the value network; Value network for state Value estimation; :time The state vector; :time Discount cumulative rewards Accumulate rewards for discounts; Total length of the trajectory; Discount factor, range of values ; :time The instant reward received; : Current time step index; : Time step index, used for accumulation, value range arrive .

[0140] Overall loss function for:

[0141]

[0142] in, For strategy In state The entropy below the threshold is used to encourage exploration; The value network loss is calculated based on normalized returns, representing a mixed value loss. Overall loss function; : Parameters of the main policy network; Parameters of the auxiliary policy network; Parameters of the value network; The main policy network's pruning substitution objective function; The pruning alternative objective function of the auxiliary policy network; Weighting coefficients for value loss items; : Weighting coefficients of the entropy regularization term; : time step The expected value (empirical mean); Policy Network In state Select action The probability of; Action variables (summation variables, iterating through all possible actions); :time The state vector; Strategy Index These correspond to the primary strategy and the secondary strategy, respectively.

[0143] During training, the value network Updates are performed based on a hybrid trajectory collected by two policy networks. To reconcile the scale differences between the two reward signals, the reward value is normalized before calculating the value loss. The indices of the two policy networks are consistent with the indices of the trajectory categories: in the probability ratio formula... This represents the policy network number, where j indicates the trajectory category collected by the corresponding policy network. Specifically, j=1 corresponds to trajectory data from the main policy network, and j=2 corresponds to trajectory data from the auxiliary policy network. The normalization formula is:

[0144]

[0145] in, and These are the first in the current batch The mean and standard deviation of the trajectory returns (j=1 corresponds to the main policy trajectory, j=2 corresponds to the auxiliary policy trajectory), and the current batch refers to a mini-batch of data sampled from the currently collected trajectory data each time the policy is updated. : No. Trajectory at time Normalized cumulative return; : No. Trajectory at time The cumulative return (original dimension), specifically, and The trajectories of the primary and secondary strategies at time points are respectively... The cumulative returns.

[0146] Then, the normalized reward value is used as the training target; the normalized state value estimate output by the value network needs to be converted back to the original dimension through inverse normalization before it is used for the calculation of temporal difference error.

[0147] The value network output To estimate the state values ​​under normalized dimensions, and to calculate the GAE advantage corresponding to the immediate reward, it is necessary to perform the following in each training iteration: Through inverse normalization ( (After converting the corresponding strategy type back to the original dimensions, it can be used again.) The calculation is performed instead of waiting for all training to complete before executing the transformation. : Normalized state value estimation of value network output; State value estimation after inverse normalization, used for calculate, This is the time-series difference error, used for generalized advantage estimation (GAE) calculation.

[0148] The mixed value loss is:

[0149]

[0150] in, Trajectory dataset collected by the main policy network. Trajectory datasets collected by the auxiliary strategy network.

[0151] S6: Deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis based on the detected external disturbance intensity.

[0152] Step S6: Execute policy network deployment and online adaptive inference.

[0153] After training, the main policy network will be... Deployed in the robot controller. When running online, it performs the following operations in each control cycle:

[0154] (1) Collect the current status This includes information on joint angles, angular velocities, center of mass status, and contact forces.

[0155] (2) Master policy network Output basic actions ;

[0156] (3) The adaptive disturbance compensation module estimates the external disturbance and generates the compensation torque according to step S4. ;

[0157] (4) Calculate the final control torque And applied to each joint;

[0158] (5) Auxiliary policy network It runs continuously in the background, and its output does not participate in the synthesis of the final control command under the default state (the disturbance intensity is below the threshold), but is only used for monitoring and strategy evaluation.

[0159] (6) When a significant external disturbance is detected ( , To trigger the auxiliary policy network's output to participate in control command synthesis, a preset threshold (ideally 10%~20% of the nominal torque amplitude) is set. To prevent command conflicts that might result from directly mixing the output torques of the two policy networks, a joint fine-tuning stage is introduced during the training phase: after basic training is completed, the shared value network parameters are fixed, and the two policy networks continue to be trained synchronously for a small number of iterations in a mixed scenario containing random perturbations and obstacles (typically a value of...). (Environmental step), during which the learning rate of the strategy network decays to 20% of that during the basic training phase (i.e., The base learning rate is The batch size remains unchanged, and the loss function is consistent with step S5. The criterion for fine-tuning convergence is: in continuous... Within the environmental step, the hybrid reward curve ( and The moving average rate of change of the weighted sum is less than 1%. Smooth switching based on perturbation strength is used during online inference.

[0160]

[0161] in, The switching coefficient; To switch the steepness factor, switch the steepness factor. The dimension of torque is the reciprocal of the torque, and its value is determined according to the unit system of torque. In the embodiment using the International System of Units (SI, where the unit of torque is N·m), The value range is 0.5 to 2.0. When the disturbance intensity is below the threshold, Control is entirely dominated by the main strategy; when the disturbance intensity exceeds the threshold, The output of the auxiliary strategy network increases monotonically according to an exponential curve and remains less than 1 under finite disturbance strength, approaching 1 as the disturbance strength increases. This allows the output of the auxiliary strategy network to gradually participate in the synthesis of control commands, but under finite disturbance strength... It will not be strictly equal to 1, thus ensuring that the output of the main policy always retains a certain weight, maintaining the continuity and stability of control instructions. In actual numerical calculations, an upper limit value less than 1 (such as 0.99) can be set for β to ensure that the main policy network always retains at least 1% control weight.

[0162] Steps S1 to S6 together constitute a complete adaptive control closed loop: Step S1 establishes the interaction interface (state space and action space) between the robot and the environment; Steps S2 and S3 construct dual-strategy decision-making and differentiated optimization objectives, enabling the robot to simultaneously learn a stable basic gait and adaptive adjustment strategies for disturbances / terrain; Step S4 introduces model-based disturbance compensation at the execution layer to compensate for the response delay of pure data-driven strategies under sudden disturbances; Step S5 ensures the stability of the training process and the ability to transfer from simulation to reality through collaborative constraint factors and domain randomization techniques; Step S6 realizes online adaptive inference after deployment. Each of these steps addresses different technical issues at different levels of the control system, complementing and synergistically achieving stable motion control of a multi-joint robot under complex terrain and external disturbance conditions.

[0163] Figure 4 This is a schematic diagram of a module of a multi-joint robot motion posture adaptive control system based on deep reinforcement learning, according to an embodiment of the present invention. The present invention also provides a multi-joint robot motion posture adaptive control system based on deep reinforcement learning, comprising:

[0164] The state perception module is used to collect the robot's state information and construct the state space and action space for the motion posture control of the multi-joint robot. The state space includes the robot's joint space state and the environmental perception state, and the action space is defined as the target angle increment of each joint.

[0165] The dual-strategy network module includes a primary strategy network and an auxiliary strategy network, which share the same value network. The primary strategy network aims to generate stable gait, while the auxiliary strategy network aims to adjust perturbation response and terrain adaptability.

[0166] The hierarchical reward function module is used to set different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation.

[0167] An adaptive disturbance compensation module is used to estimate external disturbances online and generate compensation torques.

[0168] The training module is used to train the main policy network, the auxiliary policy network and the value network based on the hierarchical reward function using a proximal policy optimization algorithm. During the training process, a collaborative constraint factor is introduced to coordinate the update step size of the main policy network and the auxiliary policy network.

[0169] The online inference and control module is used to deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis according to the intensity of the detected external disturbance. The basic control torque output by the policy network is superimposed with the compensation torque to form the final control torque applied to each joint of the robot.

[0170] The multi-joint robot motion posture adaptive control system based on deep reinforcement learning provided in this embodiment of the invention can execute the multi-joint robot motion posture adaptive control system method based on deep reinforcement learning provided in any of the above embodiments of the invention, and has the corresponding functions and beneficial effects of executing the method. For detailed process, please refer to the relevant operations of the multi-joint robot motion posture adaptive control system method based on deep reinforcement learning in the foregoing embodiments.

[0171] Example 2: Walking control of a bipedal humanoid robot in complex terrain

[0172] This embodiment uses a 20-DOF bipedal humanoid robot model (12 DDOF for legs, 6 DDOF for arms, and 2 DDOF for neck) to verify the effectiveness of the method of the present invention in a GPU-accelerated physics simulation environment (Isaac Gym). The following experimental data are the average values ​​obtained from 5 random seeded repeated experiments.

[0173] (1) State space configuration: Dimension of state vector It includes 20 joint angles, 20 joint angular velocities, 20 torques from the previous moment, 20 movements from the previous moment, center of mass position (3D), center of mass linear velocity (3D), center of mass angular velocity (3D), gravity direction projection (3D), and 4 plantar contact force information.

[0174] (2) Motion space configuration: The motion vector dimension is 20, and each motion component corresponds to the target angle increment of a joint, with a value range of [value range missing]. Radius (independent for each joint). Joint torque is calculated using a PD control law with proportional gain. Each dimension is set to 50, differential gain Each dimension is set to 5.

[0175] (3) Network structure: Both the main policy network and the auxiliary policy network are three-layer MLPs, with the number of hidden layer neurons being respectively The activation function is GELU. The shared value network structure is identical. Network parameters are updated using the Adam optimizer, with a learning rate of... .

[0176] (4) Training hyperparameters: The number of parallel environments is 2048, and the discount factor is... GAE parameters Cutting parameters Value loss coefficient Entropy coefficient Perturbation observer gain Each diagonal element is set to 10, and the filter coefficients are... Disturbance detection threshold Take the nominal torque amplitude Switch steepness coefficient Each training iteration collects trajectory data over 24 time steps, followed by 5 rounds of policy updates. The total number of training steps is approximately... Environmental step. All experiments were repeated 5 times, using different random seeds. The control cycle sampling rate is 1000 Hz.

[0177] Domain randomization configuration: During training, the mass of the robot link is uniformly disturbed within ±20% of the nominal value, the ground friction coefficient is uniformly disturbed within ±20%, and the joint motion command is injected with a random communication delay of 1 to 3 control cycles.

[0178] (5) Comparison with the baseline configuration: The standard PPO algorithm used for comparison employs the same state / action space, single-policy network + single-value network structure, and training hyperparameters (number of parallel environments, total number of training steps, learning rate, domain randomization configuration, etc. are all consistent), and the reward function is... and The weighted summaries of all items (with weights consistent with the corresponding coefficients in the tiered versions) are used to exclude the impact of reward differences on the comparison results.

[0179] (6) Ablation experiment configuration: To verify the independent contribution of each module of the present invention, the following three configurations were set up for comparative experiments (all under the same parallel environment with 2048 elements and the same total number of training steps). (Environmental step, under the same 5 random seed conditions)

[0180] Configuration A (Dual-policy network only, no disturbance compensation): Employs a dual-policy network architecture and a hierarchical reward function, but does not include the disturbance compensation module in step S4. The final control torque... ;

[0181] Configuration B (Disturbance compensation only, single policy): Employs a single-policy network + disturbance compensation module. The policy network optimization objective is... and Weighted combined rewards;

[0182] Configuration C (Complete Solution): Includes both a dual-strategy network and a disturbance compensation module, which is the complete technical solution of this invention.

[0183] To independently verify the contribution of the collaborative constraint factors, additional configurations are set based on configuration A above. Dual-strategy network, but each strategy is independently tailored and not shared. (That is, the two policy networks independently calculate probability ratios and perform pruning separately, with no coupling in update magnitude between them), all other conditions are exactly the same as configuration A. The reward curve fluctuation of configuration A (including the co-constraint factor) during training (measured by the standard deviation of the reward over the most recent 100 iterations) is as follows: Configuration for The former is numerically lower than the latter by approximately Meanwhile, the standard deviation of the network parameter update step size (gradient norm) for policy A is... Configuration for This indicates that the collaborative constraint factor plays a positive role in suppressing update instability.

[0184] (7) Results of the experiment on flat terrain: Under flat terrain, the average cumulative reward of this method (configuration C) is (Number of repeated experiments, mean ± std), standard PPO algorithm (single policy + merged reward, no perturbation compensation) is ( In complex terrain (containing random elevation changes and obstacles), the average cumulative reward of this method is... ( The standard PPO algorithm is .

[0185] (8) Disturbance recovery experiment: During the robot's stable walking process, at The duration is 0.5 s, and the amplitude is [value missing]. The pulsed external disturbance torque (applied in the torso yaw direction). The disturbance recovery time is defined as the time from the end of the disturbance until the robot's torso attitude angle converges back to its normal value. And the duration required to last 0.5 seconds. The recovery time for the baseline state (open-loop PD control, no policy network, no disturbance compensation) without any compensation measures is set to... (Secondary perturbation experiment).

[0186] The recovery times for each configuration are as follows: Configuration A (dual-policy network only) is... Configuration B (disturbance compensation only, single strategy) is as follows: Configure C (complete solution) as follows: All three configurations outperformed the baseline, with configuration C achieving the best average recovery time among the three. An independent samples t-test (Welch's test) was used to compare the recovery time of configuration C with the baseline. This indicates that configuration C's improvement relative to the baseline state is statistically significant.

[0187] (9) Summary of ablation experiments: The average cumulative rewards for the three configurations in complex terrain are as follows: Configuration A Configuration Configuration Configuration C is numerically superior to both Configuration A and Configuration B.

[0188] Explanation regarding model dependency and simulation-to-reality transfer:

[0189] The policy network control part of this invention adopts a model-free reinforcement learning framework. Its policy network inference process does not depend on the analytical expression of the robot's precise dynamic model. It can output control actions only based on the current observation state, which is the main source of the generalization ability of the method of this invention.

[0190] The disturbance compensation module, as an auxiliary enhancement component, has its disturbance estimation accuracy limited by the parameters of the robot dynamics model. The accuracy. To enhance robustness in actual deployment: (1) The generalized momentum perturbation observer used in this invention can be improved through gain. The design adjusts the estimated response characteristics, and the lower gain value can effectively suppress the high-frequency noise pollution caused by model error; (2) A first-order low-pass filter is connected in series at the output of the observer to further attenuate the estimation spikes caused by modeling error. That is, the engineering robustness of the disturbance compensation module comes from the coordinated design of the low-pass filter and the observer gain, rather than the dependence on the high-precision dynamic model.

[0191] Regarding the migration from simulation to reality, this invention improves the policy network's tolerance to model uncertainties by introducing domain randomization technology in step S5 (±20% disturbance of link mass, ±20% disturbance of friction coefficient, and random communication delay of 1-3 control cycles). It should be noted that the performance verification experiments in Embodiments 1 and 2 of this invention were completed in a physical simulation environment, and the domain randomization technology is used to improve the robustness of the policy network's migration from simulation to reality. In actual hardware deployment, adjustments to control cycles, sensor noise parameters, etc., still need to be made according to the specific platform characteristics.

[0192] In summary, this invention provides a method and system for adaptive motion posture control of multi-joint robots based on deep reinforcement learning. Through the organic combination of a dual-policy network architecture, a hierarchical reward function, a cooperative constraint factor, and an adaptive perturbation compensation module, it effectively improves the robot's motion stability and adaptability under complex terrain and external perturbation conditions. Specifically, the dual-policy network is an innovation at the decision layer, solving the problem that a single policy cannot adequately address multiple objectives; the hierarchical reward function is an innovation at the training signal layer, providing differentiated optimization objectives for the dual-policy network; the cooperative constraint factor is an innovation at the training algorithm layer, ensuring that the dual-policy networks do not interfere with each other during joint optimization; and the adaptive perturbation compensation module is an auxiliary enhancement at the execution layer, compensating for the response lag of purely data-driven policies under sudden perturbations.

[0193] The above description is only a preferred embodiment of the present invention and is used only to explain the present invention. It is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0194] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0195] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. A method for adaptive motion posture control of a multi-joint robot based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Construct the state space and action space for the motion posture control of a multi-joint robot. The state space includes the robot joint space state and the environmental perception state. The action space is defined as the target angle increment of each joint. S2: Construct a dual-strategy network architecture, which includes a primary strategy network and an auxiliary strategy network, wherein the primary strategy network and the auxiliary strategy network share the same value network; S3: Design a hierarchical reward function, setting different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation; S4: Construct an adaptive disturbance compensation module to estimate external disturbances online and generate compensation torque. The compensation torque is then superimposed on the basic control torque output by the policy network to form the final control torque applied to each joint of the robot. S5: Employ the near-end policy optimization algorithm and train the main policy network, the auxiliary policy network, and the value network based on the hierarchical reward function. Introduce a collaborative constraint factor during training to coordinate the update step size of the main policy network and the auxiliary policy network. S6: Deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis based on the detected external disturbance intensity.

2. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the state space at time t includes: the current angular position of each joint, the angular velocity of each joint, the joint torque applied at the previous time, the action output at the previous time, the position of the robot's center of mass, the linear velocity of the center of mass, the angular velocity of the center of mass, the projection of the gravity direction, and the contact force information of the foot or end. The motion space is defined as the target angle increment vector of each joint. The torque actually applied to each joint is calculated by the impedance control law, which calculates the joint torque based on the deviation between the target angle and the current angle of each joint and the angular velocity information.

3. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S2, both the main policy network and the auxiliary policy network are multilayer perceptron structures, each containing an input layer, a hidden layer and an output layer. The activation function of the hidden layer is a modified linear unit or a Gaussian error linear unit, and the output layer uses a hyperbolic tangent function to map the action to the effective range.

4. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S2, the main policy network and the auxiliary policy network adopt a collaborative update mechanism during training, defining the first... The probability ratio of each policy network at time t for: ； in, For the first The old strategy before the strategy network update; For a moment The action vector; For a moment The state vector; Introducing a collaborative constraint factor: ;in, The probability ratio of the main policy network at time t. The probability ratio of the auxiliary policy network at time t; When the update ratio of either policy network exceeds the pruning interval, the overall update step size is restricted by taking the minimum value operation. The gradient updates of the two policy networks at each data point are subject to the gating constraint of the other's ratio, forming an asymmetric update constraint mechanism.

5. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the reward function of the main policy network for: ； in, For progress rewards; For posture rewards; For joint torque smoothing bonus items; This is a reward item for gait symmetry. This is a foot contact bonus item; This is a reward item for angular velocity tracking. This is a torque energy consumption penalty term; The reward function of the auxiliary policy network for: ； in, For in-flight time rewards; For contact force adaptation bonus items; For disturbance recovery rewards; Weighting coefficient All are adjustable positive coefficients.

6. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 5, characterized in that, The gait symmetry reward item for: ； in, For a set of symmetrical joint pairs, This is the current gait cycle estimate. For a moment left leg Each joint angle For a moment Right leg The online estimation method for the gait cycle T is as follows: based on the detection of the ground contact and ground departure events of the left and right feet by the plantar contact force sensor, the time interval between two consecutive ground contact events of the same side foot is recorded, and the current gait cycle estimate is obtained by averaging through a sliding window.

7. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S4, the adaptive disturbance compensation module is designed based on the generalized momentum method: defining the generalized momentum of the robot joint space, constructing a disturbance observer, feeding back the estimation error of the generalized momentum through the disturbance observer to generate an estimated disturbance torque, and generating the compensation torque after filtering the estimated disturbance torque. The filtering process uses exponential moving average filtering, and the compensation torque at the current moment is obtained by weighted summation of the estimated disturbance torque at the current moment and the compensation torque at the previous moment. The final control torque is formed by superimposing the base control torque output by the policy network and the compensation torque.

8. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S5, domain randomization technology is introduced during the training process: random noise is injected into the robot's environmental dynamic parameters, and random communication delays are introduced into the motion output of the joint motors. The environmental dynamic parameters include the robot link mass, center of mass position, and ground friction coefficient, with the random noise amplitude of each parameter being ±20% of the nominal value; the communication delay is 1 to 3 control cycles.

9. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S5, the value network is updated based on the mixed trajectory data collected by the main policy network and the auxiliary policy network. Before calculating the value loss, the cumulative returns of the two types of trajectories from the two policy networks are normalized respectively, and the normalized return values ​​are used as training targets. The normalized state value estimate output by the value network needs to be converted back to the original dimensions through inverse normalization before it is used for the calculation of temporal difference error.

10. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 1, characterized in that, In step S6, during online inference, current state information is collected in each control cycle, the main strategy network outputs basic actions, and the adaptive disturbance compensation module estimates external disturbances and generates compensation torques. The basic control torques output by the main strategy network and the compensation torques are superimposed and applied to each joint. The auxiliary strategy network runs continuously in the background. When the intensity of external disturbance is lower than a preset threshold, the output of the auxiliary strategy network does not participate in the synthesis of the final control command.

11. The adaptive motion posture control method for multi-joint robots based on deep reinforcement learning according to claim 10, characterized in that, When the detected external disturbance intensity exceeds the preset threshold, a switching coefficient is introduced. The switching coefficient is zero when the disturbance intensity is lower than or equal to the threshold, and the control is completely dominated by the main strategy network. When the disturbance intensity exceeds the threshold, the switching coefficient increases monotonically with the increase of the disturbance intensity and is always less than 1 under finite disturbance intensity. It approaches 1 as the disturbance intensity increases, so that the output of the auxiliary strategy network gradually participates in the synthesis of control commands, while the output of the main strategy network always retains a certain weight.

12. A multi-joint robot motion posture adaptive control system based on deep reinforcement learning, characterized in that, include: The state perception module is used to collect the robot's state information and construct the state space and action space for the motion posture control of the multi-joint robot. The state space includes the robot's joint space state and the environmental perception state, and the action space is defined as the target angle increment of each joint. The dual-strategy network module includes a primary strategy network and an auxiliary strategy network, which share the same value network. The primary strategy network aims to generate stable gait, while the auxiliary strategy network aims to adjust perturbation response and terrain adaptability. The hierarchical reward function module is used to set different reward signals for the main policy network and the auxiliary policy network respectively, wherein the reward function of the main policy network focuses on gait stability and energy efficiency, and the reward function of the auxiliary policy network focuses on perturbation response and terrain adaptation. An adaptive disturbance compensation module is used to estimate external disturbances online and generate compensation torques. The training module is used to train the main policy network, the auxiliary policy network and the value network based on the hierarchical reward function using a proximal policy optimization algorithm. During the training process, a collaborative constraint factor is introduced to coordinate the update step size of the main policy network and the auxiliary policy network. The online inference and control module is used to deploy the trained main policy network in the robot controller for online adaptive inference, and selectively incorporate the output of the auxiliary policy network into the control command synthesis according to the intensity of the detected external disturbance. The basic control torque output by the policy network is superimposed with the compensation torque to form the final control torque applied to each joint of the robot.

Images