A multi-joint robot cooperative control method and system based on reinforcement learning

Abstract

The application discloses a multi-joint robot cooperative control method and system based on reinforcement learning, relates to the technical field of robot control, and comprises the following steps: defining constraint conditions according to a target task; constructing a reward function according to the target task and the constraint conditions; training parameters of a policy network by using a proximal policy optimization algorithm until the reward function is maximized, so as to obtain a trained policy network; inputting state data of a multi-joint robot into the trained policy network to obtain constraint parameters; obtaining control instructions of the multi-joint robot based on the constraint parameters by using a model predictive control algorithm; and driving the multi-joint robot to execute the target task according to the control instructions, and dynamically adjusting the constraint parameters according to state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot. The application can improve the control precision and stability of the multi-joint robot and can complete the adaptation of multi-task scenes.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method and system for cooperative control of multi-joint robots based on reinforcement learning. Background Technology

[0002] As robotics technology expands into highly dynamic and dexterous scenarios, complex tasks such as humanoid robot backflips, quadruped robot obstacle crossings, and precision robotic arm operations place stringent demands on the accuracy and robustness of control systems. Model predictive control (MPC), with its forward-looking trajectory planning capabilities, has become one of the core technologies for complex motion control. Its control effect is highly dependent on preset constraints, which need to be precisely matched with the physical laws of the task (such as the conservation of angular momentum during a backflip) and external forces (such as the ground reaction force during the push-off phase).

[0003] However, traditional MPC control systems have significant limitations: On the one hand, constraints need to be manually derived and modeled based on physical laws. For the complex mechanical structure of robots (multi-joint coupling, uneven mass distribution), manual modeling is difficult to accurately capture physical laws and external force characteristics. For example, when a humanoid robot does a backflip, the mathematical expression of angular momentum conservation needs to be adapted to the dynamic changes in limb posture, and the threshold setting of ground return force needs to match ground characteristics and mechanical load limits. Manually set fixed constraints are prone to modeling errors, which directly lead to a decrease in control accuracy. On the other hand, the external environment and robot state in complex tasks are dynamic and changeable. Manually preset static constraints cannot be adaptively adjusted and are difficult to cope with sudden disturbances (such as sudden changes in ground friction coefficient and load changes), which limits the environmental adaptability of the system.

[0004] While reinforcement learning (RL) technology can adapt to complex tasks through autonomous learning, it suffers from insufficient robustness when applied alone: ​​the strategies trained by reinforcement learning are highly dependent on the consistency between the simulation environment and the real scene. Small differences in physical parameters (such as the friction coefficient deviation between the simulation and the real environment) may cause the strategy to fail, and it is prone to action instability when dealing with sudden situations, making it difficult to meet the safe execution requirements of complex tasks.

[0005] In existing technologies, the combination of MPC and reinforcement learning mainly focuses on the pattern of "reinforcement learning outputs control commands, and MPC optimizes trajectories," failing to address the core pain point of MPC constraints' reliance on manual intervention and thus unable to achieve autonomous optimization and dynamic adaptation of constraints. Therefore, there is an urgent need for a new type of collaborative control system that can utilize the autonomous learning capabilities of reinforcement learning to achieve accurate modeling and adaptive adjustment of MPC constraints, while also leveraging the stability of MPC to compensate for the robustness deficiencies of reinforcement learning, thereby meeting the high precision and high reliability requirements of complex tasks for control systems. Summary of the Invention

[0006] In view of this, embodiments of this application provide a cooperative control method and system for multi-joint robots based on reinforcement learning, so as to improve the control accuracy and stability of multi-joint robots.

[0007] One aspect of this application provides a cooperative control method for a multi-joint robot based on reinforcement learning, the method comprising the following steps:

[0008] Define constraints based on the target task;

[0009] Construct a reward function based on the target task and the constraints;

[0010] The parameters of the policy network are trained using a proximal policy optimization algorithm until the reward function is maximized, thus obtaining the trained policy network.

[0011] The state data of the multi-joint robot is input into the trained policy network to obtain the constraint parameters output by the trained policy network.

[0012] The control commands for the multi-joint robot are obtained by solving the constraint parameters using a model predictive control algorithm.

[0013] The control command drives the multi-joint robot to perform the target task, and the constraint parameters are dynamically adjusted based on the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot.

[0014] In some embodiments, before inputting the state data of the multi-joint robot into the trained policy network to obtain constraint parameters, the method further includes a step of collecting the state data of the multi-joint robot, wherein collecting the state data of the multi-joint robot includes the following steps:

[0015] The multi-jointed robot is defined as a spinal quadruped robot;

[0016] A simulation robot model of the aforementioned quadrupedal robot was built using MuJoCo;

[0017] The physical interactions of the simulated robot model are simulated in the MuJoCo simulation environment, and then the state data of the simulated robot model is collected.

[0018] In some embodiments, obtaining the control commands for the multi-joint robot based on the constraint parameters using a model predictive control algorithm includes the following steps:

[0019] set up Let be the joint angle vector of the multi-joint robot. The joint angular velocity vector. The joint angular acceleration vector. Let be the three-dimensional position vector of the centroid of the multi-joint robot. , The center-of-mass attitude rotation matrix includes pitch angles. Yaw angle Roll angle , This is the joint control input vector;

[0020] The optimization problem of the model predictive control algorithm is defined as minimizing the trajectory tracking error and the cost of control input smoothness; wherein, the optimization problem is expressed as:

[0021] ;

[0022] ;

[0023] ;

[0024] in, To predict the time domain, To control the time domain, For the expected position of the center of mass, For the center of gravity, the expected posture, R is the state tracking weight matrix, and R is the control weight matrix. This represents matrix vectorization operations; MPC constraints include joint angle limitations. and angular velocity limit , and the dynamic constraints of the n-dimensional parameters output by the near-end strategy optimization algorithm;

[0025] The optimization problem is solved by the model predictive control algorithm, and the target angular velocity of each joint of the multi-joint robot is then output as the control command.

[0026] In some embodiments, the target task is a backflip task, which includes a take-off acceleration phase, an in-flight flip phase, and a posture adjustment landing phase.

[0027] The process of defining constraints based on the target task includes the following steps:

[0028] The constraint parameters output by the trained policy network are matched and mapped with the ground return force constraint and angular momentum conservation constraint of the backflip task.

[0029] in:

[0030] The generation steps of the ground return force constraint include: the constraint parameters include The ground reaction force is defined as being evenly distributed on the leg in contact with the ground. Based on the joint states of the multi-joint robot relative to the position of the center of mass and speed The estimated real-time ground return force, combined with the dynamic model of the multi-joint robot, yields the following ground return force constraint:

[0031] ;

[0032] The steps for generating the angular momentum conservation constraint include: the constraint parameters further include... ; Determine that angular momentum is equal to the product of moment of inertia and angular velocity. The angular momentum conservation constraint is defined as follows: When flipping in the air, the total angular momentum of the center of mass rotation and the angular momentum of the joint rotation remains constant.

[0033] ;

[0034] .

[0035] In some embodiments, constructing the reward function based on the target task and the constraints includes the following steps:

[0036] Define task completion rewards This is used to evaluate the achievement of the aforementioned somersault task; wherein, the task completion reward... A segmented reward logic is adopted: during the take-off phase, if the vertical velocity of the center of mass is... Reward +30; During the aerial flip phase, reward +20 for each 90° flip completed. Full flip bonus +80; during landing phase, if the center of gravity pitch angle Roll angle And the peak value of the ground return force is ≤ If the landing is deemed smooth, the reward is +200; if the 360° flip is not completed or the landing posture deviation exceeds the limit, the reward is 0.

[0037] Define constraints and adapt rewards The constraint parameters used to guide the policy network output to adapt to the physical requirements of the backflip task; wherein, the constraint adaptation reward It is divided into ground return force adaptation bonus and angular momentum conservation adaptation bonus: the ground return force adaptation bonus is... This ensures that the ground return force is stably output within the threshold range;

[0038] Define security penalties and rewards To avoid dangerous movement states: if the joint angle exceeds the limit range, each joint exceeding the limit is penalized by -15; if the multi-joint robot collides with the environment, the penalty is -50 and the current training round is terminated; if the control input is greater than the maximum angular velocity of the joint, the penalty is -20.

[0039] Define step reward This is used to reward survival, so that the multi-joint robot is rewarded +1 for each simulated step.

[0040] In some embodiments, training the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized to obtain the trained policy network includes the following steps:

[0041] Define the strategy network output layer to use Activation functions are used to map constraint parameters to a preset range;

[0042] The parameter configuration includes the total number of training rounds, the maximum number of steps per round, the batch size, and the policy update after collecting the first set number of samples;

[0043] The training process employs a segmented training and gradient pruning strategy. The first phase focuses on optimizing constraints during the take-off phase, amplifying the reward weight for ground return to fit the reward, and guiding the policy network to learn a ground return threshold that meets physical conditions. The second phase optimizes constraints during the in-flip phase, amplifying the reward weight for attitude tracking. The third phase is a full-process collaborative optimization, restoring the balance of reward weights, setting the step reward weight to 0, and adjusting constraint parameters to adapt to the requirements of the entire backflip phase. During training, the average reward is calculated every second set number of rounds. When the average reward for a consecutive second set number of rounds reaches a preset reward threshold, the policy network is considered converged, resulting in the trained policy network.

[0044] In some embodiments, inputting the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network includes the following steps:

[0045] The trained policy network is deployed to the computing unit mounted on the multi-joint robot;

[0046] The computing unit inputs the state data of the multi-joint robot into the trained policy network and runs the trained policy network to obtain the constraint parameters output by the trained policy network.

[0047] Another aspect of this application embodiment provides a multi-joint robot cooperative control system based on reinforcement learning, the system comprising:

[0048] Constraint definition unit, used to define constraints according to the target task;

[0049] A reward construction unit is used to construct a reward function based on the target task and the constraints.

[0050] The network training unit is used to train the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized, thereby obtaining the trained policy network.

[0051] The state constraint unit is used to input the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network.

[0052] The instruction generation unit is used to obtain the control instructions for the multi-joint robot by solving the constraint parameters using a model predictive control algorithm.

[0053] A robot drive unit is used to drive the multi-joint robot to perform the target task according to the control instructions, and to dynamically adjust the constraint parameters according to the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot.

[0054] Another aspect of this application embodiment provides an electronic device, including a processor and a memory;

[0055] The memory is used to store programs;

[0056] The processor executes the program to implement any of the methods described above.

[0057] Another aspect of this application provides a computer-readable storage medium storing a program that is executed by a processor to implement the method described in any of the above embodiments.

[0058] This application includes at least the following beneficial effects:

[0059] This application defines constraints based on the target task; constructs a reward function based on the target task and constraints; trains the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized, obtaining the trained policy network; inputs the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network; uses a model predictive control algorithm to solve for the control commands of the multi-joint robot based on the constraint parameters; drives the multi-joint robot to execute the target task according to the control commands, and dynamically adjusts the constraint parameters based on the dynamic feedback state data of the multi-joint robot to dynamically drive the multi-joint robot. This application defines constraints and reward functions specifically for the target task, and then uses the trained policy network to enable the policy network to output accurate constraint parameters, and then solves for the control commands based on these constraint parameters, improving the control accuracy and stability of the multi-joint robot and enabling adaptation to multi-task scenarios. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a cooperative control method for multi-joint robots based on reinforcement learning, provided as an embodiment of this application;

[0062] Figure 2 An example diagram of a quadruped robot provided in an embodiment of this application;

[0063] Figure 3 This is a structural block diagram of a multi-joint robot cooperative control system based on reinforcement learning, provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] Reference Figure 1 This application provides a cooperative control method for multi-joint robots based on reinforcement learning, specifically including the following steps S100~S150:

[0066] S100: Define constraints based on the target task;

[0067] S110: Construct a reward function based on the target task and the constraints;

[0068] S120: The parameters of the policy network are trained using the proximal policy optimization algorithm until the reward function is maximized, thus obtaining the trained policy network;

[0069] S130: Input the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network;

[0070] S140: The control commands for the multi-joint robot are obtained by solving the constraint parameters using a model predictive control algorithm;

[0071] S150: Drive the multi-joint robot to perform the target task according to the control command, and dynamically adjust the constraint parameters according to the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot.

[0072] Optionally, before inputting the state data of the multi-joint robot into the trained policy network to obtain constraint parameters, the method further includes a step of collecting the state data of the multi-joint robot, wherein collecting the state data of the multi-joint robot includes the following steps:

[0073] The multi-jointed robot is defined as a spinal quadruped robot;

[0074] A simulation robot model of the aforementioned quadrupedal robot was built using MuJoCo;

[0075] The physical interactions of the simulated robot model are simulated in the MuJoCo simulation environment, and then the state data of the simulated robot model is collected.

[0076] Optionally, the step of using a model predictive control algorithm to obtain the control commands for the multi-joint robot based on the constraint parameters includes the following steps:

[0077] set up Let be the joint angle vector of the multi-joint robot. The joint angular velocity vector. The joint angular acceleration vector. Let be the three-dimensional position vector of the centroid of the multi-joint robot. , The center-of-mass attitude rotation matrix includes pitch angles. Yaw angle Roll angle , This is the joint control input vector;

[0078] The optimization problem of the model predictive control algorithm is defined as minimizing the trajectory tracking error and the cost of control input smoothness; wherein, the optimization problem is expressed as:

[0079] ;

[0080] ;

[0081] ;

[0082] in, To predict the time domain, To control the time domain, For the expected position of the center of mass, For the center of gravity, the expected posture, R is the state tracking weight matrix, and R is the control weight matrix. This represents matrix vectorization operations; MPC constraints include joint angle limitations. and angular velocity limit , and the dynamic constraints of the n-dimensional parameters output by the near-end strategy optimization algorithm;

[0083] The optimization problem is solved by the model predictive control algorithm, and the target angular velocity of each joint of the multi-joint robot is then output as the control command.

[0084] Optionally, the target task is a backflip task, which includes a take-off acceleration phase, an in-air flip phase, and an attitude adjustment landing phase.

[0085] The process of defining constraints based on the target task includes the following steps:

[0086] The constraint parameters output by the trained policy network are matched and mapped with the ground return force constraint and angular momentum conservation constraint of the backflip task.

[0087] in:

[0088] The generation steps of the ground return force constraint include: the constraint parameters include The ground reaction force is defined as being evenly distributed on the leg in contact with the ground. Based on the joint states of the multi-joint robot relative to the position of the center of mass and speed The estimated real-time ground return force, combined with the dynamic model of the multi-joint robot, yields the following ground return force constraint:

[0089] ;

[0090] The steps for generating the angular momentum conservation constraint include: the constraint parameters further include... ; Determine that angular momentum is equal to the product of moment of inertia and angular velocity. The angular momentum conservation constraint is defined as follows: When flipping in the air, the total angular momentum of the center of mass rotation and the angular momentum of the joint rotation remains constant.

[0091] ;

[0092] .

[0093] Optionally, constructing the reward function based on the target task and the constraints includes the following steps:

[0094] Define task completion rewards This is used to evaluate the achievement of the aforementioned somersault task; wherein, the task completion reward... A segmented reward logic is adopted: during the take-off phase, if the vertical velocity of the center of mass is... Reward +30; During the aerial flip phase, reward +20 for each 90° flip completed. Full flip bonus +80; during landing phase, if the center of gravity pitch angle Roll angle And the peak value of the ground return force is ≤ If the landing is deemed smooth, the reward is +200; if the 360° flip is not completed or the landing posture deviation exceeds the limit, the reward is 0.

[0095] Define constraints and adapt rewards The constraint parameters used to guide the policy network output to adapt to the physical requirements of the backflip task; wherein, the constraint adaptation reward It is divided into ground return force adaptation bonus and angular momentum conservation adaptation bonus: the ground return force adaptation bonus is... This ensures that the ground return force is stably output within the threshold range;

[0096] Define security penalties and rewards To avoid dangerous movement states: if the joint angle exceeds the limit range, each joint exceeding the limit is penalized by -15; if the multi-joint robot collides with the environment, the penalty is -50 and the current training round is terminated; if the control input is greater than the maximum angular velocity of the joint, the penalty is -20.

[0097] Define step reward This is used to reward survival, so that the multi-joint robot is rewarded +1 for each simulated step.

[0098] Optionally, the step of training the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized to obtain the trained policy network includes the following steps:

[0099] Define the strategy network output layer to use Activation functions are used to map constraint parameters to a preset range;

[0100] The parameter configuration includes the total number of training rounds, the maximum number of steps per round, the batch size, and the policy update after collecting the first set number of samples;

[0101] The training process employs a segmented training and gradient pruning strategy. The first phase focuses on optimizing constraints during the take-off phase, amplifying the reward weight for ground return to fit the reward, and guiding the policy network to learn a ground return threshold that meets physical conditions. The second phase optimizes constraints during the in-flip phase, amplifying the reward weight for attitude tracking. The third phase is a full-process collaborative optimization, restoring the balance of reward weights, setting the step reward weight to 0, and adjusting constraint parameters to adapt to the requirements of the entire backflip phase. During training, the average reward is calculated every second set number of rounds. When the average reward for a consecutive second set number of rounds reaches a preset reward threshold, the policy network is considered converged, resulting in the trained policy network.

[0102] Optionally, the step of inputting the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network includes the following steps:

[0103] The trained policy network is deployed to the computing unit mounted on the multi-joint robot;

[0104] The computing unit inputs the state data of the multi-joint robot into the trained policy network and runs the trained policy network to obtain the constraint parameters output by the trained policy network.

[0105] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0106] This embodiment relates to the field of robot control technology, specifically to an MPC constraint self-optimization control method that integrates physical laws and external force estimation. It aims to solve the problems of traditional MPC relying on manual modeling of physical constraints and the difficulty in accurately estimating external forces. This embodiment uses a reinforcement learning strategy to simultaneously estimate the physical law parameters required for the task (such as the angular momentum conservation coefficient) and external forces (such as ground reciprocation force), automatically generating them as dynamic constraints for the MPC. Using the robot's joint states and center of mass pose as observations, the reinforcement learning strategy outputs constraint parameters. The MPC solves for the optimal joint control commands based on these constraints, completing closed-loop training in conjunction with a physics engine. A task-specific reward function guides the strategy to optimize constraints, ensuring that the constraints adapt to the robot's mechanical characteristics and the task's physical laws (such as angular momentum conservation and ground reciprocation force utilization). This embodiment avoids the problem of difficulty in accurately modeling constraints manually due to the robot's complex mechanical structure, and leverages the stability of MPC to improve the sim-to-real robustness of reinforcement learning. It can be widely applied to scenarios such as complex movements of legged robots and precision operations of multi-joint robotic arms. This embodiment can be used for complex tasks requiring quadrupedal and humanoid robots, such as inspection and disaster relief tasks in high-risk environments.

[0107] This embodiment provides an MPC collaborative control method based on reinforcement learning self-optimization constraints. By training the constraints of the MPC through reinforcement learning, it achieves autonomous adaptation and dynamic optimization of constraints. Specifically, it includes the following steps:

[0108] Step 1: Construct a training environment of "reinforcement learning - MPC - physics engine" using MuJoCo or Gazebo physics engine as the simulation platform, and build a training process of "observation - policy - constraint - MPC - execution". The observation data includes robot joint state, center of mass position and posture information. MPC solves for the optimal control command based on the constraint conditions and robot dynamics model.

[0109] Step 2: Design a task-oriented policy and reward function. Define the constraint type and dimension of the policy output according to the specific task requirements. If the task is a robot backflip, the constraints include ground return force constraints and angular momentum conservation equation constraints. The policy output corresponds to the value of F on the left side of the equation constraint (e.g., the ground return force constraint satisfies FG=acc). Design a multi-dimensional reward function around the rationality of the task objective and constraints. Use the proximal policy optimization (PPO) algorithm for offline training and iteratively update the policy network parameters until the reward function is maximized.

[0110] Step 3: Deploy the trained policy network to the MPC system of the real robot, collect robot state data in real time and input it into the policy network to obtain dynamic constraint parameters. The MPC solves the joint control commands based on the constraints, drives the robot to perform tasks and dynamically adjusts the constraints according to real-time feedback.

[0111] Step 4: Based on different task requirements and robot platforms, repeat steps 1 to 3 to redefine the constraint type, dimension, and reward function to complete the adaptation for multi-task scenarios.

[0112] The specific implementation method is as follows:

[0113] Reference Figure 2 The following uses a quadruped robot backflip task as an example to explain in detail the specific implementation steps of this embodiment:

[0114] Step 1: Build a closed-loop training system that includes a spinal quadruped robot model, the MuJoCo physics engine, the open-source MPC solver library, and the PPO reinforcement learning algorithm. Clarify the input-output mapping relationship between each module to lay the foundation for subsequent constrained self-optimization training.

[0115] First, the structure of the spinal quadruped robot is defined. This robot adopts an integrated "spine-limbs" structure, in which the spine module contains two tandem joints. Each joint has two degrees of freedom: pitch and yaw. The range of joint bending angles is [not specified]. °, deflection angle range It is used to coordinate the robot's overall posture to adapt to high-dynamic movements such as flipping; each limb has 3 driven joints (hip, knee, and ankle joints), each joint has only pitch freedom, for a total of 12 limb joints, with the hip joint pitch angle range being - Knee joint flexion / extension range Ankle flexion / extension range The sole features a flexible contact design with a radius of... To ensure stable contact with the ground.

[0116] The physics engine uses MuJoCo to build a high-fidelity robot simulation model and simulate real physical interactions. The simulation step size is 10ms, the gravitational acceleration is set to 9.8m / s², the ground material is set to hard rubber, and the friction coefficient is 0.8. MuJoCo's built-in contact force detection module is also enabled to acquire ground return force data in real time. The MPC solution uses the open-source ocs2 library, which has the advantages of fast solution speed and strong numerical stability, adapting to the robot's real-time control requirements. The MPC prediction time domain is configured to be 0.5s (50 simulation steps), and the control time domain to be 0.1s (10 simulation steps). The reinforcement learning algorithm uses Proximal Policy Optimization (PPO), and the environment is set up using a custom Gymnasium. The policy network adopts a fully connected neural network structure for autonomous learning and outputting MPC constraint parameters.

[0117] Finally, a complete process of "observation-policy-constraint-MPC-execution-feedback" is constructed, and the input-output correspondence of each stage is clearly defined:

[0118] 1. Data Acquisition: The robot's state information is collected in real time through the MuJoCo simulation environment as the observation input for the PPO algorithm. The observation dimensions total 40 dimensions, including the angles and angular velocities of 12 limb joints (24 dimensions), the pitch / yaw angles and corresponding angular velocities of 2 spinal joints (4 dimensions), the three-dimensional position and velocity of the robot's center of mass (6 dimensions), and the pitch / yaw / roll angles and angular velocities of the center of mass (6 dimensions).

[0119] 2. PPO policy output: The PPO policy network takes 42-dimensional observation data as input, and after three fully connected layers (42-dimensional input layer, 256-dimensional hidden layer, and n-dimensional output layer), it outputs n-dimensional MPC constraint parameters as the constraint input for MPC.

[0120] 3. MPC Solving and Output: First, define the core variable symbols for MPC:

[0121] set up The robot joint angle vector (14×1, including 12 limb joints and 2 spinal degrees of freedom). The joint angular velocity vector is (14×1). The joint angular acceleration vector is (14×1). The three-dimensional position vector of the centroid (3×1, i.e.) ), The center-of-mass attitude rotation matrix (3×3, determined by pitch angle) Yaw angle Roll angle constitute), The joint control input vector (i.e., the target angular velocity of the joint) is 14×1; the optimization objective of MPC is to minimize the trajectory tracking error and the cost of control input smoothness, and the optimization problem is expressed as:

[0122] ;

[0123] ;

[0124] ;

[0125] in For predicting the time domain (50 steps). To control the time domain (10 steps). For the expected position of the center of mass, For the center of gravity, the expected posture, R is the state tracking weight matrix (a 3+9 order diagonal matrix with diagonal elements set to 10), and R is the control weight matrix (a 14 order diagonal matrix with diagonal elements set to 0.1). Represents matrix vectorization operations; constraints include basic joint angle limitations. and angular velocity limit The dynamic constraints are the n-dimensional parameters output by PPO (detailed in step two); the above constrained quadratic programming problem is solved by MPC, and the target angular velocities of 14 joints (12 limb joints + 2 spinal degrees of freedom) are output as the robot's control commands.

[0126] 4. Simulation Execution and Feedback: The 14 joint target angles output by MPC are input into the MuJoCo simulation environment to drive the robot to complete the corresponding actions. Simultaneously, the MuJoCo environment outputs new robot state information (updated observation data) and interactive physical quantities (realistic return force, real angular momentum), feeding back to the observation input and reward calculation module of the PPO algorithm, forming a complete closed loop. The data flow transmission frequency throughout the training process is consistent with the MuJoCo simulation step size, i.e., a complete data interaction and calculation is completed every 10ms, ensuring real-time control.

[0127] Step 2: Based on the physical characteristics of the backflip task, clarify the MPC constraint type and generation logic of the strategy output, design a reward function adapted to the task, and complete the parameter configuration and training logic construction of the PPO algorithm to achieve autonomous optimization of constraints.

[0128] First, let's analyze the backflip task, which consists of three phases: the takeoff and acceleration phase, the mid-air flip phase, and the posture adjustment and landing phase. The core requirement of the takeoff and acceleration phase is to gain sufficient initial velocity by pushing off the ground with all four limbs. The power source for this phase is the ground reaction force (ground return force), and it's necessary to ensure that the peak ground return force provides sufficient initial velocity. vertical take-off speed and The horizontal backward speed must not exceed the load-bearing limit of the robot's mechanical structure; during the aerial flip phase, there is no external force (ignoring air resistance), and the flip posture must be maintained by relying on the law of conservation of angular momentum. This is achieved by adjusting the relative angle between the spinal joints and limb joints to change the moment of inertia. Stable rollover; during the landing phase, precise control of joint attitude is required to minimize deviation of the center of gravity pitch angle. Roll angle deviation Ensure a smooth landing and avoid collision damage.

[0129] Next, the mapping definition between the strategy output and MPC constraints is completed. The 6-dimensional parameters output by the strategy network need to accurately match the two core physical requirements of the backflip task, namely the ground return force constraint and the angular momentum conservation constraint. The specific mapping relationship and constraint generation logic are as follows:

[0130] First, the ground return force constraint is generated, and the strategy outputs 3D inequality constraint parameters. Assuming the ground reaction force is evenly distributed on the leg in contact with the ground, For robot joint state-based , with the position of the center of mass and speed The estimated real-time ground return force, combined with the constraints on the ground obtained from the robot dynamics model, is as follows:

[0131] ;

[0132] Second, angular momentum conservation constraints are generated, and the strategy outputs 3D equality constraint parameters. Once the body leaves the ground and begins to rotate in the air, the principle of conservation of angular momentum comes into play. Angular momentum is equal to the product of moment of inertia and angular velocity. In the absence of external torque, the angular momentum of a body remains constant during rotation in the air. For example, when a robot begins to rotate in the air, it changes its moment of inertia by adjusting its body posture. If the body is curled up, the moment of inertia decreases, and according to the conservation of angular momentum, the angular velocity increases, thus increasing the rotational speed; conversely, if the body is extended, the moment of inertia increases, the angular velocity decreases, and the rotational speed decreases. Therefore, during a somersault in the air, the total value of the angular momentum of the center of mass rotation and the angular momentum of the joint rotation remains constant, that is:

[0133] ;

[0134] ;

[0135] After the above constraints are generated, they are synchronously input into the MPC controller built in step one, which together with the dynamic constraints constitutes the complete constraint system for the backflip task.

[0136] Next, a multi-dimensional reward function is designed. The reward function needs to take into account task completion, constraint adaptability, and motion safety, and guide the PPO strategy to optimize constraint parameters. The specific design of each sub-reward is as follows: First, task completion reward. It is used to evaluate the achievement of the core objective of the backflip task, and adopts a segmented reward logic: in the take-off phase, if the vertical velocity of the center of gravity is high... Reward +30; During the aerial flip phase, reward +20 for each 90° flip completed. Full flip bonus +80; during landing phase, if the center of gravity pitch angle Roll angle And the peak value of the ground return force is ≤ A successful landing earns a reward of +200; failure to complete a 360° flip or exceeding the landing posture limit results in a reward of 0. Secondly, there is a constraint adaptation reward. The constraint parameters used to guide the strategy output are adapted to the physical requirements of the task, and are divided into ground return force adaptation rewards and angular momentum conservation adaptation rewards: the ground return force adaptation reward is... The system encourages stable output of the return force within a threshold range, avoiding insufficient take-off speed due to excessive force or damage to the mechanical structure due to excessive force. Thirdly, it includes safety penalties and rewards. To avoid dangerous movement states: if a joint angle exceeds the limit range defined in step one, a penalty of -15 is applied to each joint exceeding the limit; if the robot collides with the environment (e.g., the torso touches the ground), a penalty of -50 is applied and the current training round is terminated; if the control input is greater than the maximum angular velocity of the joint, a penalty of -20 is applied to ensure the safety of the training process. Finally, a step reward needs to be added. The reward is for survival; the robot receives +1 for each simulated step it takes.

[0137] Finally, the design and configuration of the PPO algorithm were completed. The PPO algorithm, implemented using the PyTorch framework, is used to learn the optimization strategy for the aforementioned constraint parameters. Specific design details are as follows: First, the network structure; the output layer of the policy network adopts... The activation function maps the constraint parameters to a reasonable range, where the ground return force threshold is one such parameter. Second, regarding core parameter configuration, the total number of training rounds is set to 100,000,000, with a maximum number of steps per round of 400 (corresponding to a 4-second simulation time, covering the backflip process); the batch size is 1000, and the policy is updated every 1000 samples collected. Third, regarding training logic design, a training strategy of "segmented training + gradient pruning" is adopted: the first 60,000 rounds focus on optimizing the constraints of the take-off phase, amplifying the reward weight of the ground return adaptation reward (weight coefficient 1.5), and guiding the policy to learn a reasonable ground return threshold; The round focuses on optimizing the constraints of the aerial flip phase and amplifies the reward weight of attitude tracking reward (weight coefficient 1.5). The rounds involve full-process collaborative optimization, restoring the balance of reward weights, setting the step reward weight to 0, and fine-tuning constraint parameters to adapt to the full-stage requirements of the backflip. During training, the average reward is calculated every 100 rounds. The average reward is determined after 100 consecutive rounds. When the policy convergence is reached, the network parameters of the policy at this time are saved for subsequent deployment.

[0138] Step 3:

[0139] 1. The converged PPO policy network was solidified and deployed to the edge computing unit (NVIDIA Jetson AGX Orin, with a computing power of ≥200 TOPS, meeting the real-time requirement of 10ms data processing) on ​​the spinal quadruped robot. The MPC controller was ported to the real-time operating system and maintained a 10ms control cycle. The joint encoder, IMU and other sensors were calibrated to ensure that the accuracy of the 40-dimensional observation data matched the simulation, and the parameters of the MPC dynamic model were corrected.

[0140] 2. Real-time acquisition of robot state data is input into the strategy network, and the 6-dimensional constraint parameters are obtained and then transmitted to the MPC controller. The MPC combines the real-time state and constraints to solve for the optimal joint target angular velocity, driving the robot to perform a backflip. The constraint parameters are dynamically adjusted through real-time feedback data such as ground return force and angular velocity to adapt to environmental disturbances.

[0141] 3. After completing the somersault maneuver in three stages—take-off, flip, and landing—the system monitors the operational status of each module in real time. If any data anomalies or command timeouts occur, the system immediately triggers a safety braking mechanism to ensure the safe completion of the mission.

[0142] Step 4: Based on the "Reinforcement Learning-MPC" collaborative framework built in the above steps, the output dimension, constraint type and reward function of the policy can be fine-tuned to achieve rapid adaptation to different highly dynamic tasks without reconstructing the overall control architecture.

[0143] In summary, this embodiment includes the following key technical solutions:

[0144] 1. Constraint Self-Optimization Mechanism: The core innovation is to autonomously generate and optimize MPC constraints through reinforcement learning, which solves the problem of the difficulty in accurately modeling physical laws and external forces in traditional MPC, and significantly improves task execution efficiency. The scope of protection covers the method of constructing the mapping relationship between reinforcement learning and MPC constraints, the autonomous generation and dynamic optimization logic of constraint parameters, and the technical implementation path of this mechanism to improve the accuracy of MPC modeling and optimize task execution efficiency.

[0145] 2. PPO Learning Constraints Instead of Direct Learning Strategies: The core innovation is to use the PPO algorithm to learn MPC constraints instead of directly outputting control strategies. This avoids the environmental sensitivity of reinforcement learning's direct output of control strategies, significantly reduces sim-to-real transfer errors, and improves policy robustness. The scope of protection includes the PPO algorithm and the adaptation learning method for MPC constraints, the control logic design of "learning constraints rather than policies", and the technical path of this method to reduce sim-to-real errors and improve robustness.

[0146] 3. Collaborative Control Enables High-Dynamic Motion: The core innovation is the integration of the stable control characteristics of MPC and the self-optimization capability of PPO to enable flexible and stable acrobatic movements (such as flips) of multi-joint robots such as quadruped robots. The scope of protection includes the collaborative control architecture of MPC and PPO, the interaction logic between the two in the high-dynamic motion scenario of multi-joint robots, and the technical implementation of this architecture to enable robots to complete acrobatic movements such as flips.

[0147] Reference Figure 3 This application provides a reinforcement learning-based multi-joint robot cooperative control system, including:

[0148] Constraint definition unit, used to define constraints according to the target task;

[0149] A reward construction unit is used to construct a reward function based on the target task and the constraints.

[0150] The network training unit is used to train the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized, thereby obtaining the trained policy network.

[0151] The state constraint unit is used to input the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network.

[0152] The instruction generation unit is used to obtain the control instructions for the multi-joint robot by solving the constraint parameters using a model predictive control algorithm.

[0153] A robot drive unit is used to drive the multi-joint robot to perform the target task according to the control instructions, and to dynamically adjust the constraint parameters according to the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot.

[0154] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0155] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0156] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0157] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0159] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0160] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0161] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0162] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0163] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A cooperative control method for multi-joint robots based on reinforcement learning, characterized in that, The method includes the following steps: Define constraints based on the target task; Construct a reward function based on the target task and the constraints; The parameters of the policy network are trained using a proximal policy optimization algorithm until the reward function is maximized, thus obtaining the trained policy network. The state data of the multi-joint robot is input into the trained policy network to obtain the constraint parameters output by the trained policy network. The control commands for the multi-joint robot are obtained by solving the constraint parameters using a model predictive control algorithm. The control command drives the multi-joint robot to perform the target task, and the constraint parameters are dynamically adjusted based on the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot. The process of obtaining control commands for the multi-joint robot based on the constraint parameters using a model predictive control algorithm includes the following steps: set up Let be the joint angle vector of the multi-joint robot. The joint angular velocity vector. The joint angular acceleration vector. Let be the three-dimensional position vector of the centroid of the multi-joint robot. , The center-of-mass attitude rotation matrix includes pitch angles. Yaw angle Roll angle , This is the joint control input vector; The optimization problem of the model predictive control algorithm is defined as minimizing the trajectory tracking error and the cost of control input smoothness; wherein, the optimization problem is expressed as: ； ； ； in, To predict the time domain, To control the time domain, For the expected position of the center of mass, For the center of gravity, the expected posture, R is the state tracking weight matrix, and R is the control weight matrix. This represents matrix vectorization operations; MPC constraints include joint angle limitations. and angular velocity limit , and the dynamic constraints of the n-dimensional parameters output by the near-end strategy optimization algorithm; The optimization problem is solved by the model predictive control algorithm, and the target angular velocity of each joint of the multi-joint robot is then output as the control command. The process of training the policy network using a proximal policy optimization algorithm until the reward function is maximized to obtain the trained policy network includes the following steps: Define the strategy network output layer to use Activation functions are used to map constraint parameters to a preset range; The parameter configuration includes the total number of training rounds, the maximum number of steps per round, the batch size, and the policy update after collecting the first set number of samples; The training process employs a segmented training and gradient pruning strategy. The first phase focuses on optimizing constraints during the take-off phase, amplifying the reward weight for ground return to fit the reward, and guiding the policy network to learn a ground return threshold that meets physical conditions. The second phase optimizes constraints during the in-flip phase, amplifying the reward weight for attitude tracking. The third phase is a full-process collaborative optimization, restoring the balance of reward weights, setting the step reward weight to 0, and adjusting constraint parameters to adapt to the requirements of the entire backflip phase. During training, the average reward is calculated every second set number of rounds. When the average reward for a consecutive second set number of rounds reaches a preset reward threshold, the policy network is considered converged, resulting in the trained policy network.

2. The multi-joint robot cooperative control method based on reinforcement learning according to claim 1, characterized in that, Before inputting the state data of the multi-joint robot into the trained policy network to obtain constraint parameters, the method further includes a step of collecting the state data of the multi-joint robot. Collecting the state data of the multi-joint robot includes the following steps: The multi-jointed robot is defined as a spinal quadruped robot; A simulation robot model of the aforementioned quadrupedal robot was built using MuJoCo; The physical interactions of the simulated robot model are simulated in the MuJoCo simulation environment, and then the state data of the simulated robot model is collected.

3. The multi-joint robot cooperative control method based on reinforcement learning according to claim 1, characterized in that, The target task is a backflip task, which includes a take-off and acceleration phase, an in-air flip phase, and a posture adjustment and landing phase. The process of defining constraints based on the target task includes the following steps: The constraint parameters output by the trained policy network are matched and mapped with the ground return force constraint and angular momentum conservation constraint of the backflip task. in: The generation steps of the ground return force constraint include: the constraint parameters include The ground reaction force is defined as being evenly distributed on the leg in contact with the ground. Based on the joint states of the multi-joint robot relative to the position of the center of mass and vertical velocity of the center of mass The estimated real-time ground return force, combined with the dynamic model of the multi-joint robot, yields the following ground return force constraint: ； The steps for generating the angular momentum conservation constraint include: the constraint parameters further include... ; Determine that angular momentum is equal to the product of moment of inertia and angular velocity. The angular momentum conservation constraint is defined as follows: When flipping in the air, the total angular momentum of the center of mass rotation and the angular momentum of the joint rotation remains constant. ； 。 4. The multi-joint robot cooperative control method based on reinforcement learning according to claim 3, characterized in that, The step of constructing the reward function based on the target task and the constraints includes the following steps: Define task completion rewards This is used to evaluate the achievement of the aforementioned somersault task; wherein, the task completion reward... A segmented reward logic is adopted: during the take-off phase, if the vertical velocity of the center of mass is... Reward +30; During the aerial flip phase, reward +20 for each 90° flip completed. Full flip bonus +80; during landing phase, if the center of gravity pitch angle Roll angle And the peak value of the ground return force is ≤ If the landing is deemed smooth, the reward is +200; if the 360° flip is not completed or the landing posture deviation exceeds the limit, the reward is 0. Define constraints and adapt rewards The constraint parameters used to guide the policy network output to adapt to the physical requirements of the backflip task; wherein, the constraint adaptation reward It is divided into ground return force adaptation bonus and angular momentum conservation adaptation bonus: the ground return force adaptation bonus is... This ensures that the ground return force is stably output within the threshold range; Define security penalties and rewards To avoid dangerous movement states: if the joint angle exceeds the limit range, each joint exceeding the limit is penalized by -15; if the multi-joint robot collides with the environment, the penalty is -50 and the current training round is terminated; if the control input is greater than the maximum angular velocity of the joint, the penalty is -20. Define step reward This is used to reward survival, so that the multi-joint robot is rewarded +1 for each simulated step.

5. A cooperative control method for multi-joint robots based on reinforcement learning according to any one of claims 1 to 4, characterized in that, The process of inputting the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network includes the following steps: The trained policy network is deployed to the computing unit mounted on the multi-joint robot; The computing unit inputs the state data of the multi-joint robot into the trained policy network and runs the trained policy network to obtain the constraint parameters output by the trained policy network.

6. A multi-joint robot cooperative control system based on reinforcement learning, characterized in that, The system is used to implement the reinforcement learning-based cooperative control method for multi-joint robots as described in claim 1, and the system includes: Constraint definition unit, used to define constraints according to the target task; A reward construction unit is used to construct a reward function based on the target task and the constraints. The network training unit is used to train the parameters of the policy network using a proximal policy optimization algorithm until the reward function is maximized, thereby obtaining the trained policy network. The state constraint unit is used to input the state data of the multi-joint robot into the trained policy network to obtain the constraint parameters output by the trained policy network. The instruction generation unit is used to obtain the control instructions for the multi-joint robot by solving the constraint parameters using a model predictive control algorithm. A robot drive unit is used to drive the multi-joint robot to perform the target task according to the control instructions, and to dynamically adjust the constraint parameters according to the state data dynamically fed back by the multi-joint robot, so as to dynamically drive the multi-joint robot.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images