A quadruped robot low-noise gait control method and system based on a soft landing reward function
By using a control method based on a soft landing reward function and combining it with a reinforcement learning algorithm to optimize the gait of a quadruped robot, the noise and stability problems of the quadruped robot are solved, achieving low-noise and stable gait control suitable for multiple application scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2025-11-26
- Publication Date
- 2026-07-07
AI Technical Summary
Existing gait control methods for quadruped robots have failed to effectively reduce the impact and noise when the feet contact the ground. Furthermore, traditional mechanical optimization methods are costly, have poor adaptability, and lack targeted control algorithms, which affect user experience and application scenarios.
A control method based on a soft landing reward function is adopted. By defining the foot contact judgment criteria, a soft landing reward function with negative vertical contact force penalty is designed. Combined with reinforcement learning algorithm and gait stability constraints, the policy network is trained to optimize the gait control of the quadruped robot.
It significantly reduces foot noise of quadruped robots without affecting gait stability, adapts to different ground materials, reduces noise by 15-20 decibels, and requires no mechanical modification, making it suitable for noise-sensitive scenarios.
Smart Images
Figure CN121209388B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of four-group robot motion control technology, and in particular to a low-noise gait control method and system for quadruped robots based on a soft landing reward function. Background Technology
[0002] Quadruped robots are bionic robots with four legs. They have advantages such as a more acceptable appearance and universal land mobility. As a type of multi-legged robot, they have a stronger load-bearing capacity and better stability than bipedal robots, and a simpler structure than hexapod or octagonal robots. Therefore, they have attracted the attention of researchers at home and abroad.
[0003] Quadruped robots, with their flexible movement capabilities, have significant advantages in adapting to complex terrains and operating in multiple scenarios, and in recent years have gradually penetrated into close-range human-computer interaction scenarios such as indoor services and medical care. However, the gait control of traditional quadruped robots focuses on movement stability and speed, neglecting the impact force and noise issues at the moment the feet contact the ground. When the robot's feet contact the ground at a high speed or with a large force, obvious impact noise is generated, which not only affects the user experience but may also interfere with the operation of medical equipment and affect the quietness of the indoor environment, limiting their application in noise-sensitive scenarios.
[0004] Existing methods for reducing robot gait noise mainly focus on optimizing the mechanical structure design, such as using elastic foot pads and shock-absorbing joints. However, this approach has the following drawbacks:
[0005] Mechanical optimization is costly: elastic floor mats need to be customized according to different floor materials (wood flooring, tile, carpet), and are prone to wear and tear with long-term use, resulting in high maintenance costs;
[0006] Poor adaptability: Once the mechanical structure is fixed, it cannot dynamically adjust the shock absorption effect according to environmental changes. For example, when switching between soft carpet and hard tile floor, the noise control effect fluctuates greatly.
[0007] Disconnected from motion control: Mechanical optimization is not combined with gait strategy adjustment, so it cannot reduce landing impact from the perspective of "active control", and the noise reduction is limited.
[0008] Furthermore, in the field of control algorithms, existing reinforcement learning gait control methods mostly aim for "lowest energy consumption" and "smooth gait," lacking design for "low noise." Although some methods attempt to add contact force penalty terms, the penalty range covers the entire contact phase (including the support phase after landing), which in turn affects the robot's gait support stability, leading to problems such as "soft gait" and "decreased load capacity." Summary of the Invention
[0009] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a low-noise gait control method and system for quadruped robots based on a soft landing reward function. This method overcomes the shortcomings of high mechanical optimization costs and insufficient targeting of control algorithms, while taking into account both noise control and gait stability of the robot.
[0010] The objective of this invention can be achieved through the following technical solution: a low-noise gait control method for a quadruped robot based on a soft landing reward function, comprising the following steps:
[0011] S1. Construct the motion control environment for the quadruped robot and define the foot contact determination criteria;
[0012] S2. Design a soft landing reward function based on vertical contact force to negatively penalize large contact forces at the moment of landing, guiding the quadruped robot to learn the landing mode with low contact force.
[0013] S3. Collect the real-time contact force and historical contact state of the quadruped robot's feet. Using the soft landing reward function as the core optimization objective, a reinforcement learning algorithm is adopted, combined with gait stability constraints, to iteratively train the policy network and obtain a low-noise gait policy network.
[0014] S4. Use a low-noise gait strategy network to output gait control signals and control the gait movements of the quadruped robot accordingly.
[0015] Furthermore, in step S1, the quadruped robot motion control environment is specifically a simulation control environment or a physical control environment, and the foot contact determination criterion is specifically:
[0016] If the vertical contact force of the foot along the Z-axis is greater than 1N, it is determined that the foot is in contact with the ground.
[0017] Furthermore, in step S2, the soft landing reward function specifically extracts the vertical contact force at the moment of landing, sums all the vertical contact forces at the moment of foot landing, and takes the negative number to obtain the final reward value.
[0018] Furthermore, the process of extracting the vertical contact force at the moment of impact includes:
[0019] Obtain the current vertical contact force data of the foot along the Z-axis, filter out foot contact states with vertical contact force > 1N, compare with the foot contact state at the previous moment, and determine whether the current moment is the moment of landing through logical operation. If the determination is yes, then determine the vertical contact force at the moment of landing = contact force × Boolean floating-point result at the moment of landing.
[0020] Furthermore, the reinforcement learning algorithm in step S3 is specifically the PPO (Proximal Policy Optimization) algorithm.
[0021] Furthermore, the gait stability constraints in step S3 include joint angle constraints and body posture constraints.
[0022] Furthermore, the joint angle constraint is used to limit the movement angle of the quadruped robot's leg joints within a corresponding safe range to prevent joint damage due to exceeding limits.
[0023] Furthermore, the body posture constraint ensures that the quadruped robot remains horizontal and stable during movement by adding penalty terms for body pitch angle and roll angle, thus preventing it from tipping over.
[0024] Furthermore, the iterative training process for the policy network in step S3 includes:
[0025] S31. Initialize the policy network parameters, set the number of training iterations and the number of training steps per round;
[0026] S32. In each round of training, collect the gait motion data of the quadruped robot, and calculate the soft landing reward and stability constraint penalty to obtain the total reward value;
[0027] S33. Update the policy network parameters based on the PPO algorithm, minimize the negative loss of the total reward value, and obtain a low-noise gait policy network.
[0028] A low-noise gait control system for a quadruped robot based on a soft landing reward function includes a data acquisition module, a data cache module, and a controller. The data acquisition module is used to acquire data on the angles, angular velocities, three-dimensional contact forces of the feet, and foot contact states of the quadruped robot's leg joints, and transmits the acquired data to the data cache module for storage.
[0029] The controller is equipped with a pre-trained low-noise gait strategy network, which retrieves data from the data cache module, updates and calculates the soft landing reward value and corresponding loss, and optimizes the output gait control signal of the quadruped robot to control and adjust the angle of the quadruped robot's leg joints.
[0030] Compared with the prior art, the present invention has the following advantages:
[0031] This invention defines a foot contact determination criterion and designs a soft landing reward function based on vertical contact force to negatively penalize large contact forces at the moment of landing, guiding the quadruped robot to learn a low-contact-force landing mode. By collecting real-time contact forces and historical contact states of the quadruped robot's feet, and using the soft landing reward function as the core optimization objective, a reinforcement learning algorithm is employed, combined with gait stability constraints, to iteratively train the policy network to obtain a policy network with low-noise gait capabilities. This network is used to control the gait movements of the quadruped robot, effectively reducing the impact force and contact noise at the moment of foot landing while ensuring the gait stability of the quadruped robot.
[0032] This invention defines the foot contact criterion as a vertical contact force of the foot along the Z-axis greater than 1N. When calculating the soft landing reward function, it extracts the vertical contact force of the foot at the moment of landing by judging the moment when there was no contact in the previous moment and contact is made now. The sum of the contact forces is taken as a negative number as the reward value, that is, the greater the contact force, the smaller the reward. This accurately determines the "moment of landing" and only penalizes the contact force at the "moment of landing". It does not affect the contact force in the support phase after landing (sufficient contact force is needed in the support phase to maintain body balance) and avoids gait instability due to over-optimization.
[0033] When iteratively training a policy network, this invention uses the PPO algorithm as a reinforcement learning algorithm. By limiting the policy update step size, the stability of the training process is ensured, and gait imbalance caused by excessive pursuit of low contact force is avoided. In addition, by combining soft landing rewards with joint angle constraints and body posture constraints, the contradiction between "noise" and "gait practicality" is resolved, and the dual goals of "low noise" and "high stability" can be achieved. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0035] Figure 2 This is a schematic diagram of the system structure of the present invention;
[0036] Figure 3 This is a schematic diagram of the application framework for Example 2;
[0037] The markings in the diagram are as follows: 1. Data acquisition module, 2. Data caching module, 3. Controller. Detailed Implementation
[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0039] Example 1
[0040] like Figure 1As shown, a low-noise gait control method for a quadruped robot based on a soft landing reward function includes the following steps:
[0041] S1. Construct the motion control environment for the quadruped robot and define the foot contact determination criteria;
[0042] S2. Design a soft landing reward function based on vertical contact force to negatively penalize large contact forces at the moment of landing, guiding the quadruped robot to learn the landing mode with low contact force.
[0043] S3. Collect the real-time contact force and historical contact state of the quadruped robot's feet. Using the soft landing reward function as the core optimization objective, a reinforcement learning algorithm is adopted, combined with gait stability constraints, to iteratively train the policy network and obtain a low-noise gait policy network.
[0044] S4. Use a low-noise gait strategy network to output gait control signals and control the gait movements of the quadruped robot accordingly.
[0045] This embodiment applies the above solution, and the specific process includes:
[0046] 1) Construct a motion simulation or physical control environment for the quadruped robot, set the foot contact judgment criteria (the vertical contact force of the foot along the Z-axis is considered to be in contact if it is greater than 1N), and collect real-time vertical contact force data and historical contact status of the robot's feet.
[0047] 2) Design a soft landing reward function. By determining the landing moment when "there was no contact in the previous frame and contact in the current frame", extract the vertical contact force of the feet at the moment of landing, and take the negative number of the total contact force as the reward value (the larger the contact force, the smaller the reward value).
[0048] Specifically: Obtain the Z-axis contact force data of the feet of the quadruped robot in the current frame, and filter out the foot contact states (current_contacts) with contact force > 1N.
[0049] Then, by comparing the foot contact state of the previous frame (last_foot_contacts), the moment of landing is determined by the logical operation - torch.logical_and(~last_foot_contacts, current_contacts);
[0050] Extract the vertical contact force at the moment of landing (landing_forces = contact force × floating-point result of Boolean value at the moment of landing), sum the contact forces at the moment of landing of all feet and take the negative number to obtain the final reward value - return-torch.sum(landing_forces,dim=1).
[0051] 3) A reinforcement learning algorithm is used, with the soft landing reward function as the core optimization objective, combined with robot gait stability constraints (including joint angle range, movement speed limit, etc.) to train the policy network.
[0052] In this embodiment, the reinforcement learning algorithm adopts the Proximal Policy Optimization (PPO) algorithm, which ensures the stability of the training process by limiting the policy update step size and avoids gait imbalance caused by excessive pursuit of low contact force.
[0053] The gait stability constraints specifically include:
[0054] Joint angle constraints: Limit the movement angles of each joint (hip, knee, and ankle) of the robot's legs within a safe range to prevent joint damage due to exceeding limits;
[0055] Body posture constraints: By adding penalties for body pitch and roll angles, we ensure that the robot's body remains horizontal and stable during movement, thus preventing it from tipping over.
[0056] 4) Iteratively adjust the reinforcement learning parameters and reward function weights, and obtain a policy network that can stably output low-noise gait through multiple training and verification processes, thereby achieving low-noise robot movement. The iterative training includes the following sub-steps:
[0057] 41) Initialize the policy network parameters, and set the number of training iterations (e.g., 1000 rounds) and the number of training steps per round (e.g., 200 steps).
[0058] 42) In each training round, collect robot gait motion data (contact force, joint angle, body posture), calculate soft landing reward and stability constraint penalty, and obtain the total reward value;
[0059] 43) Update the policy network parameters based on the PPO algorithm, minimize the negative loss of the total reward value, and obtain a policy network with low noise gait.
[0060] Based on the above method, a low-noise gait control system for a quadruped robot based on a soft landing reward function is implemented, such as... Figure 2 As shown, it includes a data acquisition module 1, a data cache module 2, and a controller 3. The data acquisition module 1 is used to acquire the angle, angular velocity, three-dimensional contact force of the foot, and foot contact state data of the quadruped robot's leg joints, and transmit the acquired data to the data cache module 2 for storage.
[0061] The controller 3 is equipped with a pre-trained low-noise gait strategy network, which is used to retrieve data from the data cache module 2, update and calculate the soft landing reward value and corresponding loss, optimize the output gait control signal of the quadruped robot, and control and adjust the angle of the quadruped robot's leg joints.
[0062] In this embodiment, the data acquisition module 1 collects the angles and angular velocities of each joint of the robot's leg in real time, the three-dimensional contact forces of the foot (X-axis: front-back direction, Y-axis: left-right direction, Z-axis: vertical direction), and the foot contact state (last_foot_contacts) of the previous frame (time interval ≤ 0.01 seconds), and stores them in the data cache module 2 for subsequent use by the controller 3 when calculating rewards.
[0063] Example 2
[0064] This embodiment applies the technical solution described in Embodiment 1 to build such a system. Figure 3 The application framework shown mainly includes:
[0065] Step 1: Environment Setup and Data Acquisition
[0066] A quadruped robot model was built using the Isaacgym simulation platform and existing open-source frameworks, with the leg joint parameters set as follows:
[0067] The range of hip joint pitch angle is -45° to 45°, and the range of hip roll angle is -30° to 30°.
[0068] The range of knee flexion angle is 0° to 90°;
[0069] The range of ankle joint pitch angle is -20° to 20°;
[0070] Step 2: Deployment of the soft landing reward function
[0071] To address the core issue that "the noise at the moment of landing is determined by the vertical contact force," a precise soft landing reward function is designed that penalizes the contact force only at the moment of landing, without affecting the contact force during the support phase after landing (sufficient contact force is needed during the support phase to maintain body balance), thus avoiding gait instability caused by over-optimization.
[0072] The designed soft landing reward function code is integrated into the reward calculation unit of the reinforcement learning environment to ensure real-time access to contact force data and historical contact states. The reward value is updated and calculated once per round and then passed to the loss calculation unit of the PPO algorithm.
[0073] Step 3: Strengthen learning and training
[0074] Initialize PPO algorithm parameters: learning rate = 1e-3, batch size = 64, GAE coefficient = 0.95, discount factor = 0.99, Clip threshold = 0.2;
[0075] Start training: Set the training rounds to 2000 rounds, and update the policy network parameters after each round of training.
[0076] Step 4: Physical Deployment
[0077] The trained low-noise gait strategy network is deployed to the embedded controller of a physical quadruped robot.
[0078] This embodiment also conducted field tests: on a wooden floor, the robot's average noise level using the native algorithm was 68.04 dB, and the maximum average noise level was 91.53 dB; after using the low-noise gait control strategy proposed in this solution, the average noise level was 65.34 dB, and the maximum average noise level was 87.63 dB.
[0079] In summary, this solution optimizes the vertical contact force at the moment the robot's feet touch the ground by designing a specific soft landing reward function. By accurately determining the landing moment, designing a negative reward for the contact force, and combining this with reinforcement learning to train the robot's gait strategy network, it achieves low-noise and stable gait movement, offering the following advantages:
[0080] 1) High noise control precision: By accurately determining the "moment of landing", optimization is only performed on the core stage of noise generation. Compared with traditional mechanical optimization methods, the noise reduction is greater (up to 15-20 decibels) and does not affect gait support stability.
[0081] 2) Low cost and strong adaptability: No need to modify the robot's mechanical structure, low-noise gait is achieved only through algorithm optimization, and different ground materials can be adapted by adjusting the reward weight, avoiding mechanical maintenance costs and adapting to multiple application scenarios;
[0082] 3) Multi-objective optimization: By combining soft landing rewards with stability and speed rewards, the contradiction between "low noise" and "gait practicality" is resolved. The trained gait is both quiet and has stable movement ability (speed ≥ 0.4 m / s, continuous exercise for 2 hours without tipping over).
[0083] 4) Easy deployment: The trained policy network can be directly deployed to different models of quadruped robots (only the joint angle mapping parameters need to be adjusted), which is highly versatile and suitable for large-scale applications.
[0084] Compared to traditional noise-free gait control methods, this solution has significant advantages in noise control and movement efficiency, and is better suited for noise-sensitive indoor service and medical assistance scenarios.
Claims
1. A low-noise gait control method for a quadruped robot based on a soft landing reward function, characterized in that, Includes the following steps: S1. Construct the motion control environment for the quadruped robot and define the foot contact determination criteria; S2. Design a soft landing reward function based on vertical contact force to negatively penalize large contact forces at the moment of landing, guiding the quadruped robot to learn the landing mode with low contact force. S3. Collect the real-time contact force and historical contact state of the quadruped robot's feet. Using the soft landing reward function as the core optimization objective, a reinforcement learning algorithm is adopted, combined with gait stability constraints, to iteratively train the policy network and obtain a low-noise gait policy network. The gait stability constraints include joint angle constraints and body posture constraints. S4. Use a low-noise gait strategy network to output gait control signals and control the gait movements of the quadruped robot accordingly. In step S1, the quadruped robot's motion control environment is specifically a simulation control environment or a physical control environment, and the foot contact determination criterion is specifically as follows: If the vertical contact force of the foot along the Z-axis is greater than 1N, it is determined that the foot is in contact with the ground. In step S2, the soft landing reward function specifically extracts the vertical contact force at the moment of landing. The sum of the vertical contact forces at the moment of foot landing is taken as a negative number to obtain the final reward value. The process of extracting the vertical contact force at the moment of landing includes: Obtain the current vertical contact force data of the foot along the Z-axis, filter out foot contact states with vertical contact force > 1N, compare with the foot contact state at the previous moment, and determine whether the current moment is the moment of landing through logical operation. If the determination is yes, then determine the vertical contact force at the moment of landing = contact force × Boolean floating-point result at the moment of landing. By determining the moment of landing when "there was no contact in the previous moment but contact is now", the vertical contact force of the foot at the moment of landing is extracted. The sum of the contact forces is negative as the reward value, that is, the greater the contact force, the smaller the reward, so as to accurately determine the "moment of landing" and only penalize the contact force at the "moment of landing", without affecting the contact force in the support phase after landing, thus avoiding gait instability due to over-optimization. In step S3, the reinforcement learning algorithm is specifically the PPO algorithm. By limiting the policy update step size, the stability of the training process is ensured, and gait imbalance caused by excessive pursuit of low contact force is avoided. The iterative training process for the policy network in step S3 includes: S31. Initialize the policy network parameters, set the number of training iterations and the number of training steps per round; S32. In each round of training, collect the gait motion data of the quadruped robot, calculate the soft landing reward and stability constraint penalty, obtain the total reward value, and combine the soft landing reward with joint angle constraints and body posture constraints to achieve the dual goals of "low noise" and "high stability". S33. Update the policy network parameters based on the PPO algorithm, minimize the negative loss of the total reward value, and obtain a low-noise gait policy network.
2. The low-noise gait control method for a quadruped robot based on a soft landing reward function according to claim 1, characterized in that, The joint angle constraint is used to limit the movement angle of the quadruped robot's leg joints within a corresponding safe range to prevent joint damage due to exceeding limits.
3. The low-noise gait control method for a quadruped robot based on a soft landing reward function according to claim 1, characterized in that, The body posture constraints, by adding penalties for body pitch and roll angles, ensure that the quadruped robot remains horizontal and stable during movement, thus preventing it from tipping over.
4. A low-noise gait control system for a quadruped robot based on a soft landing reward function, implemented using the low-noise gait control method for a quadruped robot based on a soft landing reward function as described in claim 1, characterized in that... It includes a data acquisition module (1), a data cache module (2) and a controller (3). The data acquisition module (1) is used to acquire the angle, angular velocity, three-dimensional contact force of the foot and the contact state data of the leg joint of the quadruped robot, and transmit the acquired data to the data cache module (2) for storage. The controller (3) is equipped with a trained low-noise gait strategy network, which is used to call data from the data cache module (2), and optimize the output gait control signal of the quadruped robot by updating and calculating the soft landing reward value and corresponding loss, so as to control and adjust the leg joint angle of the quadruped robot.