A quadruped robot control method with natural action and system safety

Abstract

The application discloses a kind of quadruped robot control method with action naturalness and system security, comprising the following steps: constructing network architecture, including strategy network, task value network, multi-head constraint value network and adversarial motion prior discriminator network;The motion control problem of robot is described as a constrained Markov decision process, the safety constraint condition is defined, and the log barrier term is constructed based on the principle of interior point method, and the constrained optimization objective is converted into an unconstrained optimization objective;The current state of robot is input into the strategy network to obtain the target joint angle, and the task reward, constraint value and style reward of environmental feedback are collected;Calculate the task advantage function and the constraint advantage function, and update the strategy network and the discriminator network jointly using the limited proximal policy optimization algorithm;The strategy network is deployed in the physical robot for control, realizing the dual guarantee of action naturalness and system security, and solving the problem of easy violation in the initial stage of traditional soft constraint exploration.

Description

Technical Field

[0001] This invention relates to the field of robot motion control technology, specifically to a control method for a quadruped robot that combines natural movement with system safety. Background Technology

[0002] Quadruped robots, as high-degree-of-freedom mobile platforms with discrete foot-to-ground characteristics, exhibit remarkable locomotion potential and environmental adaptability in unstructured and complex terrains such as steps, ruins, and mountainous areas. However, quadruped robot systems are essentially typical strongly coupled, underactuated nonlinear dynamic systems with a high-dimensional continuous state-action space. Considering their complex kinematic structure and variability in interaction with the environment, developing motion controllers that combine stability, agility, and safety remains a major technical challenge in the field of robot control.

[0003] Current mainstream control strategies mostly employ model-based control methods, such as model predictive control (MPC) or whole-body control (WBC). These methods heavily rely on accurate dynamic mathematical models to predict the system state and solve for the optimal control input. However, in real physical environments, the contact dynamics between a robot's foot and the ground are extremely difficult to model accurately, and the system often faces structured and unstructured uncertainties such as time-varying friction coefficients, mechanical wear errors, and sensor noise. When actual physical parameters deviate from theoretical models, or when the robot encounters unmodeled external impacts, model-based controllers often experience performance degradation or even failure, making it difficult to meet the high robustness requirements for motion in complex and varied terrains.

[0004] In recent years, data-driven methods, represented by deep reinforcement learning, have provided a new paradigm for solving model dependency problems through end-to-end trial-and-error learning. However, they still have limitations in the motion control of quadruped robots. First, traditional reinforcement learning relies heavily on manually designed reward functions, making it difficult to balance multiple objectives. This often leads to robots generating unnatural gaits with high-frequency oscillations, stiffness, or excessive energy consumption, lacking the fluidity and agility of biological motion. Second, standard reinforcement learning is essentially an unconstrained reward maximization process, lacking strict guarantees of physical limits. During the exploration process, the policy network is prone to outputting "dangerous" actions that violate joint position limits, motor torque saturation, or posture safety thresholds. This not only increases the risk of deployment on actual machines and may cause hardware damage, but also makes it difficult to achieve the optimal trade-off between task performance and safety through simple reward and penalty terms (soft constraints). Summary of the Invention

[0005] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a quadruped robot control method that combines natural movement with system safety. This invention utilizes generative adversarial networks, restricted Markov decision processes, and interior-point optimization techniques to study the safe stylized motion control problem of quadruped robots. Within a reinforcement learning framework that constrains robot posture, joint position, and joint torque, a style reward mechanism based on adversarial motion priors and a constraint optimization algorithm based on logarithmic obstacle functions are designed to achieve safe motion control of the quadruped robot. This ensures that while mimicking natural biological gait, the robot's motion state strictly converges within a preset safety constraint range.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A control method for a quadruped robot that combines natural movement with system safety includes the following steps:

[0008] Step 1: Construct and initialize the reinforcement learning network architecture, including the policy network, task value network, multi-head constraint value network, and adversarial motion prior discriminator network;

[0009] Step 2: Describe the motion control problem of the quadruped robot as a constrained Markov decision process. Based on the multi-head constrained value network, define the robot's expected cumulative discount cost, safety constraints and corresponding constraint cost functions to obtain the policy optimization objective with safety constraints. Then, based on the principle of interior point method, construct a logarithmic obstacle term to transform the policy optimization objective with safety constraints into an unconstrained optimization objective, thereby limiting the expected cumulative discount cost of each type of constraint to below the threshold.

[0010] Step 3: Obtain the robot's current state and input it into the policy network. The policy network outputs the robot's target action to interact with the environment. The target action is the target position of multiple motor joints. Collect the task reward from the environment, the constraint cost value based on the constraint cost function, the style reward output by the adversarial motion prior discriminator network, the motion trajectory state of the robot after executing the target action, and the motion trajectory state of the preset reference motion dataset. Based on the collected data, calculate the task advantage function and constraint advantage function using the task value network and the multi-head constraint value network, respectively. Solve the unconstrained optimization objective using the task advantage function, constraint advantage function, and constrained proximal policy optimization algorithm, and train the adversarial motion prior discriminator network.

[0011] Step 4: Deploy the converged policy network as the motion controller for the quadruped robot to achieve motion control of the quadruped robot.

[0012] Furthermore, both the policy network and the task value network employ a multilayer perceptron consisting of three hidden layers, with the neuron size of both the policy network and the task value network being [512, 256, 128]. The multi-head constraint value network consists of a shared multilayer perceptron of size [512, 256, 128] and multiple functionally independent output branches, with each output branch corresponding to a class of security constraints. The adversarial motion prior discriminator network employs a multilayer perceptron consisting of two hidden layers, with its neuron size being [1024, 512].

[0013] Furthermore, a constrained Markov decision process is defined as:

[0014] ;

[0015] in Representing the state space, Represents the action space. Represents the state transition model. Represents the reward function, , , express A cost function, Represents the initial state distribution. Indicates the discount factor;

[0016] The expected cumulative discount cost is:

[0017] ;

[0018] in For the first A constraint cost function, Representation strategy;

[0019] The optimization objective of the policy with safety constraints is:

[0020] ;

[0021] ;

[0022] ;

[0023] in This indicates the strategy at the current moment. This indicates the updated strategy. Represents the set of all strategies. For strategy The distribution of discount states, This indicates the constraint cost limit value. Indicates the first Class constraints, This represents the reward advantage function. Represents the constrained advantage function. , , representing the temporal difference errors of the reward and the cost, respectively. For hyperparameters, and These represent the output values ​​of the task value network and the multi-head constraint value network, respectively. and These represent the reward value and the cost value, respectively. The average KL divergence is used to measure the performance of a new strategy. Compared to the old strategy Range of change This represents the threshold for the difference in KL divergence between the old and new strategies.

[0024] Furthermore, based on the principle of interior point method, a logarithmic barrier term is constructed, specifically in the form of:

[0025] ;

[0026] in The hyperparameter that determines the steepness of the logarithmic barrier function;

[0027] The final unconstrained optimization objective takes the form of:

[0028] ;

[0029] .

[0030] Furthermore, the robot's current observation state is represented as:

[0031] ;

[0032] in Indicates the angular velocity of the base. This represents the projected gravity vector of the base. Indicates user control commands, Indicates the position of the joint motor. Indicates the speed of the joint motor. Indicates the action at the previous moment;

[0033] The motion trajectory state is specifically represented as follows:

[0034] ;

[0035] in Indicates the position of the joint motor. This indicates the position of the foot tip in the base coordinate system. Indicates the linear velocity of the base. Indicates the angular velocity of the base. Indicates the speed of the joint motor. This indicates the height of the base relative to the ground.

[0036] Furthermore, the target action of the robot output by the policy network includes mapping the target position of the motor joints to the joint torque of the motors, specifically expressed as follows:

[0037] ;

[0038] in, Indicates the target torque. , These are the proportional and derivative coefficients of the PD controller, respectively. This is the default joint position.

[0039] Furthermore, the reward function includes a target velocity tracking reward. Base Z-axis speed bonus Base attitude stability reward Joint torque reward Joint acceleration reward Collision Rewards Smoothness of motion reward Style Rewards ;

[0040] The specific form of the total reward is as follows:

[0041] ;

[0042] in It is a scalar used to balance the gap between task rewards and style rewards.

[0043] Furthermore, the target velocity tracking reward is represented as:

[0044] ;

[0045] in Indicates the sensitivity to target velocity tracking. This command indicates the speed of the fuselage along the x and y axes. This command represents the fuselage's z-axis angular velocity. Indicates the velocity along the x and y axes of the fuselage. This represents the angular velocity of the fuselage along the z-axis. This indicates an encouragement for the robot to output actions that meet the target instructions;

[0046] The base z-axis velocity bonus is represented as follows:

[0047] ;

[0048] in Indicates the speed along the z-axis of the fuselage. This indicates the speed of the robot's z-axis.

[0049] The fuselage attitude stability bonus is represented as:

[0050] ;

[0051] in and These represent the angular velocities along the x-axis and y-axis of the fuselage, respectively. This indicates the angular velocity of the robot's body along the x and y axes as a penalty.

[0052] Joint torque reward is expressed as:

[0053] ;

[0054] in Indicates the joint torque of the robot. This represents the total number of joints in the robot. This indicates excessive joint torque output as a penalty;

[0055] Joint acceleration reward is represented as:

[0056] ;

[0057] in Indicates joint acceleration. This indicates excessive joint acceleration as a punishment.

[0058] Collision bonus is represented as:

[0059] ;

[0060] in,

[0061] ,

[0062] Indicates the first Time of the first The contact force vector of each component. Indicates the contact force threshold. This indicates that the robot is penalized for collisions based on the number of parts it contacts.

[0063] The reward for smoothness of motion is represented as:

[0064] ;

[0065] in and These represent the current action and the action at the previous moment, respectively. This indicates a drastic change in the action used to punish;

[0066] Style rewards are represented as follows:

[0067] ;

[0068] in This represents the output of the adversarial motion prior discriminator network. This indicates an encouragement for the robot to output a movement style similar to the reference dataset.

[0069] Furthermore, the safety constraint functions include fuselage xy-axis attitude constraints:

[0070] ;

[0071] ;

[0072] in, Indicates the aircraft's roll angle. Indicates the aircraft's pitch angle;

[0073] Joint position limit constraints:

[0074] ;

[0075] in,

[0076] ,

[0077] This represents the total number of joints in the robot. Indicates the first The feasible domain of each joint;

[0078] Foot contact force constraint:

[0079] ;

[0080] in,

[0081] ,

[0082] express Time of the first The contact force vector at the foot end, This indicates the preset maximum bearing capacity threshold of the foot.

[0083] Joint torque constraints:

[0084] ;

[0085] in,

[0086] ,

[0087] express Time of the first The output torque of each joint is limited by the peak torque of the motor.

[0088] Furthermore, the input to the policy network is defined as the robot's current observed state. The output is The specific form of the optimization objective for the limited proximal strategy, considering the joint position, is as follows:

[0089] ;

[0090] in, Importance sampling represents the ratio of the probability of the new and old strategies for the current action. The function represents the Limit at Within the interval, This represents the pruning hyperparameter, used to control the magnitude of policy updates;

[0091] The input to the task value network is defined as the robot's current observed state. Output It is a scalar representing the expected value of the robot's current state, and its training loss function is set as follows:

[0092] ;

[0093] The input to a multi-head constrained value network is defined as the robot's current observed state. Output for A scalar representing the current state of the robot. The expected cost of the class, and its loss function are set as follows:

[0094] .

[0095] Furthermore, the input to the adversarial motion prior discriminator network is defined as the state transition-based feature observations of two consecutive frames. Its training loss function is set as follows:

[0096] ;

[0097] The total training loss function is:

[0098] ;

[0099] in Indicates a state transition pair. Indicates the reference motion dataset, Indicates parameterization Adversarial motion prior discriminator network, Indicates parameterization The policy network, Represents the gradient penalty coefficient. This indicates that the discriminator output is relative to the parameters. The gradient.

[0100] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0101] 1. Natural and realistic movements with strong generalization: This invention introduces adversarial motion prior (AMP) technology, which uses an adversarial motion prior discriminator network to distinguish the style differences between generated actions and reference actions. It can guide the robot to learn natural biological gait (such as trotting, jumping, etc.) without the need for manually designing complex reward functions, thus avoiding the jitter and unnatural movements common in traditional reinforcement learning, reducing mechanical wear and improving motion efficiency.

[0102] 2. High safety and reliable hardware protection: This invention utilizes Constrained Markov Decision Process (CMDP) and interior point method to incorporate physical limits such as joint position, motor torque, and foot impact force into a strict mathematical constraint framework. By constructing logarithmic obstacle terms, hard constraints are transformed into strong penalty barriers in the optimization objective. Compared with traditional soft constraint methods, this can more effectively prevent the strategy from outputting dangerous actions in the early stages of exploration, significantly reducing the risk of hardware damage during the migration from simulation to real machine.

[0103] 3. Robustness and adaptability: The multi-head constrained value network used in this invention can independently estimate different types of safety risks. Combined with the constrained proximal strategy optimization algorithm, the robot can dynamically adjust its control strategy under the premise of strictly meeting multiple physical safety constraints such as joint torque and anti-tipping posture when facing sudden external force disturbances or uncertainties in dynamic parameters. This achieves the unity of highly dynamic agile motion and system operation safety. Attached Figure Description

[0104] Figure 1 This is a schematic diagram of the control method for a quadruped robot that combines natural movement and system safety according to the present invention. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0106] like Figure 1As shown, this embodiment proposes a quadruped robot control method that combines natural movement with system safety. This method aims to address the problems of stiff movements, unnatural behavior, and lack of strict safety constraints in existing quadruped robot motion control technologies. This embodiment introduces adversarial motion priors to guide the robot to learn and mimic natural biological gait, and utilizes constrained Markov decision processes and interior point methods to incorporate physical limits into a strict mathematical constraint framework, thereby achieving high-performance, highly robust, natural, and safe motion control. The policy network in this embodiment takes a 45-dimensional body state as input and outputs the desired angles of twelve joints. The desired torques of the twelve joints are calculated by a PD controller. This invention simultaneously trains the adversarial motion prior discriminator network and the policy network using the Isaac Gym simulation platform, ultimately obtaining the trained policy network and deploying it into the physical robot. The method of this invention specifically includes the following steps.

[0107] Step 1: Use the Isaac Gym simulation tool to create a simulation training environment for deep reinforcement learning of quadruped robots and import the relevant quadruped robot URDF files; build and initialize the reinforcement learning network architecture, which includes a policy network, a task value network, a multi-head constraint value network, and an adversarial motion prior discriminator network.

[0108] Step 2: Describe the motion control problem of the quadruped robot as a constrained Markov decision process, define the robot's expected cumulative discount cost, safety constraints and corresponding constraint cost functions, obtain the policy optimization objective with safety constraints, and then construct a logarithmic obstacle term based on the interior point method principle to transform the policy optimization objective with safety constraints into an unconstrained optimization objective, thereby limiting the expected cumulative discount cost of each type of constraint to below the threshold.

[0109] Step 3: Obtain the robot's current state and input it into the policy network. The policy network outputs the robot's target action to interact with the environment. The target action is the target position of multiple motor joints. Control the robot to execute the target action and collect the task reward from the environment, the constraint cost value based on the constraint cost function, the style reward based on the output of the adversarial motion prior discriminator network, the motion trajectory state of the robot after executing the target action, and the motion trajectory state of the preset reference motion dataset. Based on the collected data, the task advantage function and constraint advantage function are calculated using the task value network and the multi-head constraint value network, respectively. The above unconstrained optimization objective is solved using the task advantage function, constraint advantage function, and constrained proximal policy optimization algorithm, and the adversarial motion prior discriminator network is trained.

[0110] Step 4: Set the number of training iterations. After meeting the required number of iterations, deploy the converged policy network as the motion controller for the quadruped robot, achieving safe and stylized motion control. Furthermore, both the policy network and the task value network employ a multilayer perceptron consisting of three hidden layers. The multilayer perceptron is a general function approximator that maps complex sensor data into specific joint control commands or evaluates the quality of the current state through linear combinations of multiple neurons and nested nonlinear activation functions. The neuron size of both the policy network and the task value network is [512, 256, 128]. The multi-head constraint value network consists of a shared multilayer perceptron of size [512, 256, 128] and multiple functionally independent output branches. The multi-head structure aims to achieve risk decoupling estimation of physical constraints. Each independent output branch corresponds to a specific type of physical constraint. Common features are extracted through the shared layer, and the cumulative discounted cost prediction value for that type of constraint is output through the independent branches. The adversarial motion prior discriminator network employs a multilayer perceptron consisting of two hidden layers, with a neuron size of [1024, 512]. The network uses a progressively decreasing neuron size design to gradually compress and nonlinearly map the high-dimensional robot perception state, thereby accurately extracting the core features supporting complex gait control while ensuring real-time computation.

[0111] In this embodiment, the constrained Markov decision process is defined as a tuple. ,in Representing the state space, Represents the action space. Represents the state transition model. Represents the reward function, , , express A cost function, Represents the initial state distribution. This represents the discount factor.

[0112] Furthermore, the expected cumulative discount cost is expressed as:

[0113] ;

[0114] in For the first A constraint cost function, Representation strategy;

[0115] Furthermore, the specific form of the policy optimization objective with safety constraints is as follows:

[0116] ;

[0117] ;

[0118] ;

[0119] in This indicates the strategy at the current moment. This indicates the updated strategy. Represents the set of all strategies. For strategy The distribution of discount states, This indicates the constraint cost limit value. Indicates the first Class constraints, This represents the reward advantage function. Represents the constrained advantage function. , , representing the temporal difference errors of the reward and the cost, respectively. For hyperparameters, and These represent the output values ​​of the task value network and the multi-head constraint value network, respectively. and These represent the reward value and the cost value, respectively. The average KL divergence is used to measure the performance of a new strategy. Compared to the old strategy Range of change This represents the threshold for the difference in KL divergence between the old and new strategies;

[0120] In this embodiment, the logarithmic barrier term is constructed using the interior-point method principle, specifically in the following form:

[0121] ;

[0122] in The hyperparameters that determine the steepness of the logarithmic barrier function are ultimately used to determine the unconstrained optimization objective:

[0123] ;

[0124] .

[0125] Furthermore, the robot's current observed state includes the base angular velocity, the base projected gravity vector, user control commands, joint motor positions, joint motor speeds, and the action from the previous moment, specifically represented as follows:

[0126] ;

[0127] The motion trajectory includes the position of the articulated motor, the position of the foot in the base coordinate system, the linear velocity of the base, the angular velocity of the base, the velocity of the articulated motor, and the height of the base relative to the ground. Its specific form is as follows:

[0128] ;

[0129] Furthermore, the reward function is in a combined form, specifically including a target velocity tracking reward. Base Z-axis speed bonus Base attitude stability reward Joint torque reward Joint acceleration reward Collision Rewards Smoothness of motion reward Style Rewards The specific form of the total reward is as follows:

[0130] ;

[0131] in It is a scalar value used to balance the gap between task rewards and style rewards;

[0132] Furthermore, the target velocity tracking reward is represented as:

[0133] ;

[0134] in Indicates the sensitivity to target velocity tracking. This command indicates the speed of the fuselage along the x and y axes. This command represents the fuselage's z-axis angular velocity. Indicates the velocity along the x and y axes of the fuselage. This represents the angular velocity of the fuselage along the z-axis. This indicates that the robot is encouraged to perform actions that meet the target instructions.

[0135] The base z-axis velocity bonus is represented as follows:

[0136] ;

[0137] in Indicates the speed along the z-axis of the fuselage. This indicates the speed of the robot's z-axis.

[0138] The fuselage attitude stability bonus is represented as:

[0139] ;

[0140] in and These represent the angular velocities along the x-axis and y-axis of the fuselage, respectively. This indicates the angular velocity of the robot's body along the x and y axes.

[0141] Joint torque reward is expressed as:

[0142] ;

[0143] in Indicates the joint torque of the robot. This represents the total number of joints in the robot. This indicates that excessive joint torque output is being penalized.

[0144] Joint acceleration reward is represented as:

[0145] ;

[0146] in Indicates joint acceleration. This indicates excessive joint acceleration as a punishment.

[0147] Collision bonus is represented as:

[0148] ;

[0149] in,

[0150] ,

[0151] This indicates that in the Isaac Gym simulation, the first... Time of the first The contact force vector of each component. Indicates the contact force threshold. This indicates that the robot is penalized for collisions based on the number of parts it contacts.

[0152] The reward for smoothness of motion is represented as:

[0153] ;

[0154] in and These represent the current action and the action at the previous moment, respectively. This indicates a drastic change in the action of punishment.

[0155] Style rewards are represented as follows:

[0156] ;

[0157] in This represents the output of the adversarial motion prior discriminator network. This indicates an encouragement for the robot to output a movement style similar to the reference dataset.

[0158] The constraint cost function designed in this invention covers the key physical constraints of quadruped robots, specifically including: body posture constraints, which are expressed as follows:

[0159] ;

[0160] ;

[0161] in, Indicates the aircraft's roll angle. Indicates the aircraft's pitch angle.

[0162] Joint position limit constraints:

[0163] ;

[0164] in,

[0165] ,

[0166] This represents the total number of joints in the robot. Indicates the first The feasible domain of each joint;

[0167] Foot contact force constraint:

[0168] ;

[0169] in,

[0170] ,

[0171] express Time of the first The contact force vector at the foot end, This indicates the preset maximum bearing capacity threshold of the foot.

[0172] Joint torque constraints:

[0173] ;

[0174] in,

[0175] ,

[0176] express Time of the first The output torque of each joint is limited by the peak torque of the motor.

[0177] Furthermore, the input to the policy network is defined as the robot's current observed state. The output is For joint position, the Proximal Policy Optimization (PPO) algorithm, while ensuring fast robot learning, limits the update magnitude to prevent the robot from completely breaking down its movements or causing hardware damage due to excessive update magnitude. Therefore, the objective of the constrained proximal policy optimization is:

[0178] ;

[0179] in, Importance sampling represents the ratio of the probability of the new and old strategies on the current action. The function represents the Limit at Within the interval, This represents the pruning hyperparameter, used to control the magnitude of policy updates.

[0180] The input to the task value network is defined as the robot's current observed state. Output It is a scalar representing the expected value of the robot's current state, and its training loss function is set as follows:

[0181] ;

[0182] The input to a multi-head constrained value network is defined as the robot's current observed state. Output for A scalar representing the current state of the robot. The expected cost of the class, and its loss function are set as follows:

[0183] ;

[0184] Furthermore, the input to the adversarial motion prior discriminator network is defined as the state transition-based feature observations of two consecutive frames. Its training loss function is set as follows:

[0185] ;

[0186] in Indicates a state transition pair. Indicates the reference motion dataset, Indicates parameterization Adversarial motion prior discriminator network, Indicates parameterization The policy network, Represents the gradient penalty coefficient. This indicates that the discriminator output is relative to the parameters. The gradient.

[0187] In summary, the final total training loss function is:

[0188] ;

[0189] Using the Isaac Gym simulation tool, a simulation training environment for deep reinforcement learning of a quadruped robot is created, and the relevant quadruped robot URDF file is imported. This example uses the Unitree Go1 quadruped robot. The URDF file specifically includes information such as linkage mechanisms, joint definitions, and actuator configurations. The physics engine backend is configured as PhysX, and the simulation time step is set. (50Hz), and enable GPU pipeline to support direct transfer of tensor format data in video memory, avoiding data copy latency.

[0190] In one embodiment of the present invention, the parameters of the converged policy network model are solidified and converted into a JIT format suitable for inference on an embedded platform. This JIT format is then deployed to the onboard computing unit of the quadruped robot, and the actual control loop is activated. Feedback data from the inertial measurement unit and joint encoders are collected in real time through the robot's underlying drive interface. An observation state vector is constructed that isomorphic to that used in the simulation training phase and is input into the policy network for inference. Finally, the joint target position command output by the network is calculated into a motor torque signal by the onboard PD controller and sent out for execution. This drives the quadruped robot to achieve stable and stylized motion in a real physical environment at a fixed control frequency, such as 50Hz.

[0191] This invention achieves a synergistic effect of naturalness and safety in the motion control of quadruped robots by deeply integrating adversarial motion prior (AMP) with interior point method safety constraints. On the one hand, the adversarial motion prior discriminator network guides the policy network to generate a smooth, biological-like gait through adversarial training, avoiding the motion stiffness problem caused by the complex reward engineering relied upon in traditional reinforcement learning. On the other hand, the logarithmic obstacle term constructed based on the interior point method transforms physical safety boundaries such as joint limits and torque constraints into strong penalty barriers, forming a hard constraint effect in the early stages of optimization and preventing the policy from exploring dangerous areas. The synergistic effect of the two is reflected in the following aspects: the natural motion priors provided by AMP reduce the search difficulty of the policy within the feasible domain of safety constraints, while the strict safety boundary guaranteed by the interior point method provides a stable training environment for AMP imitation learning, avoiding misjudgment by the discriminator due to dangerous samples; at the same time, the decoupled estimation of multiple types of safety risks by the multi-head constrained value network, combined with the conservative update strategy of constrained proximal policy optimization, ensures that the robot can maintain biological agility and energy efficiency in high-dynamic motion, while strictly converging within the preset safety constraints. Ultimately, it achieves the technical effect of obtaining high-fidelity motion style and hardware-level safety protection simultaneously without fine parameter tuning.

[0192] The above is the content of the present invention. However, the implementation of the present invention is not limited to the above embodiments. Any changes made in accordance with the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A quadruped robot control method with natural motion and system safety, characterized in that, Includes the following steps: Step 1: Construct and initialize the reinforcement learning network architecture, including the policy network, task value network, multi-head constraint value network, and adversarial motion prior discriminator network; Step 2: Describe the robot's motion control problem as a constrained Markov decision process, define the robot's expected cumulative discount cost, safety constraints and corresponding constraint cost functions, obtain the policy optimization objective with safety constraints, construct a logarithmic obstacle term based on the interior point method, and transform the policy optimization objective with safety constraints into an unconstrained optimization objective, so as to limit the expected cumulative discount cost of each type of constraint to below the threshold. Step 3: Obtain the robot's current state and input it into the policy network. The policy network outputs the robot's target action and interacts with the environment. The target action is the target position of multiple motor joints. Collect the task reward from the environment, the constraint cost value based on the constraint cost function, the style reward output by the adversarial motion prior discriminator network, the motion trajectory state of the robot after executing the target action, and the motion trajectory state of the preset reference motion dataset. Based on the collected data, calculate the task advantage function and constraint advantage function using the task value network and the multi-head constraint value network, respectively. Solve the unconstrained optimization objective using the task advantage function, constraint advantage function, and constrained proximal policy optimization algorithm, and train the adversarial motion prior discriminator network. Step 4: Deploy the converged policy network as the motion controller for the quadruped robot to achieve motion control of the quadruped robot.

2. The quadruped robot control method according to claim 1, which combines natural movement and system safety, is characterized in that, A constrained Markov decision process is defined as follows: ； in Representing the state space, Represents the action space, Represents the state transition model. Represents the reward function, , , express A cost function, Represents the initial state distribution. Indicates the discount factor; The expected cumulative discount cost is: ； in For the first A constraint cost function, Representation strategy; The optimization objective of the policy with safety constraints is: ； ； ； in This indicates the strategy at the current moment. This indicates the updated strategy. Represents the set of all strategies. For strategy The distribution of discount states, This indicates the constraint cost limit value. Indicates the first Class constraints, This represents the reward advantage function. Represents the constrained advantage function. , , representing the temporal difference errors of the reward and the cost, respectively. For hyperparameters, and These represent the output values ​​of the task value network and the multi-head constraint value network, respectively. and These represent the reward value and the cost value, respectively. The average KL divergence is used to measure the performance of a new strategy. Compared to the old strategy Range of change This represents the threshold for the difference in KL divergence between the old and new strategies.

3. The quadruped robot control method according to claim 2, which combines natural movement and system safety, is characterized in that... Based on the principle of interior point method, a logarithmic barrier term is constructed, specifically in the following form: ； in The hyperparameter that determines the steepness of the logarithmic barrier function; The final unconstrained optimization objective takes the form of: ； 。 4. A quadruped robot control method that combines natural movement and system safety according to claim 3, characterized in that, The robot's current observation state is represented as: ； in Indicates the angular velocity of the base. This represents the projected gravity vector of the base. Indicates user control commands, Indicates the position of the joint motor. Indicates the speed of the joint motor. Indicates the action at the previous moment; The motion trajectory state is specifically represented as follows: ； in Indicates the position of the joint motor. This indicates the position of the foot tip in the base coordinate system. Indicates the linear velocity of the base. Indicates the angular velocity of the base. Indicates the speed of the joint motor. This indicates the height of the base relative to the ground.

5. The quadruped robot control method with both natural movement and system safety according to claim 4, characterized in that, The target actions of the robot output by the policy network include mapping the target positions of the motor joints to the joint torques of the motors, specifically expressed as follows: ； in, Indicates the target torque. , These are the proportional and derivative coefficients of the PD controller, respectively. This is the default joint position.

6. A quadruped robot control method that combines natural movement and system safety according to claim 5, characterized in that, The reward function includes target velocity tracking reward. Base Z-axis speed bonus Base attitude stability reward Joint torque reward Joint acceleration reward Collision Rewards Smoothness of motion reward Style Rewards ; The specific form of the total reward is as follows: ； in It is a scalar used to balance the gap between task rewards and style rewards.

7. A quadruped robot control method that combines natural movement and system safety according to claim 6, characterized in that, The target speed tracking reward is represented as: ； in Indicates the sensitivity to target velocity tracking. This command indicates the speed of the fuselage along the x and y axes. This command represents the fuselage's z-axis angular velocity. Indicates the velocity along the x and y axes of the fuselage. This represents the angular velocity of the fuselage along the z-axis. This indicates an encouragement for the robot to output actions that meet the target instructions; The base z-axis velocity bonus is represented as follows: ； in Indicates the speed along the z-axis of the fuselage. This indicates the speed of the robot's z-axis. body Attitude stability reward is represented as: ； in and These represent the angular velocities along the x-axis and y-axis of the fuselage, respectively. This indicates the angular velocity of the robot's body along the x and y axes as a penalty. Joint torque reward is expressed as: ； in Indicates the joint torque of the robot. This represents the total number of joints in the robot. This indicates excessive joint torque output as a penalty; Joint acceleration reward is represented as: ； in Indicates joint acceleration. This indicates excessive joint acceleration as a punishment. Collision bonus is represented as: ； in, ， Indicates the first Time of the first The contact force vector of each component. Indicates the contact force threshold. This indicates that the robot is penalized for collisions based on the number of parts it contacts. The reward for smoothness of motion is represented as: ； in and These represent the current action and the action at the previous moment, respectively. This indicates a drastic change in the action used to punish; Style rewards are represented as follows: ； in This represents the output of the adversarial motion prior discriminator network. This indicates an encouragement for the robot to output a movement style similar to the reference dataset.

8. A quadruped robot control method that combines natural movement and system safety according to claim 7, characterized in that, Safety constraint functions include fuselage xy-axis attitude constraints: ； ； in, Indicates the aircraft's roll angle. Indicates the aircraft's pitch angle; Joint position limit constraints: ； in, ， This represents the total number of joints in the robot. Indicates the first The feasible domain of each joint; Foot contact force constraint: ； in, ， express Time of the first The contact force vector at the foot end, This indicates the preset maximum bearing capacity threshold of the foot. Joint torque constraints: ； in, ， express Time of the first The output torque of each joint is limited by the peak torque of the motor.

9. A quadruped robot control method that combines natural movement and system safety according to claim 8, characterized in that, The input to the policy network is defined as the robot's current observed state. The output is The specific form of the optimization objective for the limited proximal strategy, considering the joint position, is as follows: ； in, Importance sampling represents the ratio of the probability of the new and old strategies for the current action. The function represents the Limit at Within the interval, This represents the pruning hyperparameter, used to control the magnitude of policy updates; The input to the task value network is defined as the robot's current observed state. Output It is a scalar representing the expected value of the robot's current state, and its training loss function is set as follows: ； The input to a multi-head constrained value network is defined as the robot's current observed state. Output for A scalar representing the current state of the robot. The expected cost of the class, and its loss function are set as follows: 。 10. A quadruped robot control method that combines natural movement and system safety according to claim 9, characterized in that, The input to the adversarial motion prior discriminator network is defined as the state transition-based feature observations of two consecutive frames. Its training loss function is set as follows: ； The total training loss function is: ； in Indicates a state transition pair. Indicates the reference motion dataset, Indicates parameterization Adversarial motion prior discriminator network, Indicates parameterization The policy network, Represents the gradient penalty coefficient. This indicates that the discriminator output is relative to the parameters. The gradient.

Images