Quadruped robot motion control method based on adaptive deep reinforcement learning

Abstract

The invention discloses a quadruped robot motion control method based on adaptive deep reinforcement learning. The method comprises the following steps: S1, determining a network model, a composite reward function, a state space and a bionic action generation mechanism of a simulation training environment; the state space provides environment information input, the network model processes the input information and generates a decision, the bionic action mechanism executes a specific decision behavior, and the composite reward function evaluates a behavior effect and optimizes a decision direction; s2, constructing a simulation training environment of the quadruped robot, wherein the simulation environment comprises quadruped robot model information and simulation environment information; s3, training the network model by using a deep reinforcement learning algorithm based on robot model information and simulated environment information to obtain a trained motion control strategy; and S4, verifying the feasibility of utilizing the trained motion control strategy by controlling the motion of the quadruped robot in a real environment. According to the invention, the self-adaptive capability to a complex environment is obviously improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a motion control method for a quadruped robot based on adaptive deep reinforcement learning. Background Technology

[0002] Current traditional quadruped robot control methods mainly rely on pre-established dynamic models and manually designed control rules. This approach suffers from drawbacks such as cumbersome parameter adjustments, insufficient robustness, and slow response in changing environments. Furthermore, when quadruped robots move in complex environments, they not only need to maintain a stable body posture at all times, but also must possess the ability to quickly respond to various environmental obstacles, terrain changes, and unexpected events, and make reasonable behavioral decisions accordingly.

[0003] Meanwhile, quadruped robots contain numerous joints and sensors, resulting in highly complex dynamic models that involve intricate dynamic characteristics and multi-joint cooperative control. Achieving efficient motion control for quadruped robots is an extremely challenging task. In this process, quadruped robots need to effectively manage and regulate multi-dimensional degrees of freedom to achieve both stability and precision in their motion.

[0004] However, traditional model-based deep reinforcement learning methods have several limitations when dealing with such problems: on the one hand, they rely too much on accurate modeling of environmental dynamics, while there are a lot of uncertainties and variability in the actual environment, which limits the applicability of model-based methods in complex scenarios; on the other hand, the method is also insufficient in generalization ability, which often leads to unsatisfactory motion control performance of quadruped robots when facing complex environments they have never encountered before. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the present invention aims to provide a quadruped robot motion control method based on adaptive deep reinforcement learning. This method directly utilizes real-world interaction experience to train the quadruped robot, significantly improving its adaptability to complex environments. Through field learning, the quadruped robot can extract effective information from practical experience, optimize its coping strategies for complex terrain and sudden obstacles, thereby enhancing the stability and reliability of motion control.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A motion control method for a quadruped robot based on adaptive deep reinforcement learning includes the following steps;

[0008] S1: Determine the network model, composite reward function, state space, and biomimetic action generation mechanism for the simulation training environment;

[0009] The state space provides environmental information input, the network model processes the input information and generates decisions, the biomimetic action mechanism executes specific decision-making behaviors, and the composite reward function evaluates the effects of the behaviors and optimizes the decision direction.

[0010] S2: Construct a simulation training environment for the quadruped robot, wherein the simulation environment includes quadruped robot model information and simulation environment information;

[0011] S3: Based on the robot model information and the simulation environment information, the network model is trained using a deep reinforcement learning algorithm to obtain the trained motion control strategy;

[0012] S4: Verify the feasibility of using the trained motion control strategy by controlling the movement of the quadruped robot in a real environment.

[0013] In S1, the network model uses an improved SAC algorithm to construct a model-free controller. The improved SAC algorithm includes a network model that integrates the state value network and the policy network.

[0014] Among them, the state value network is used to estimate the long-term expected return in the current state, providing a stable value benchmark for policy optimization; the policy network aims to maximize the function and outputs continuous motion distribution parameters to guide the quadruped robot in motion control in uncertain environments.

[0015] The Huber loss function is introduced to evaluate the state value network; the state value network is used to predict the current state s. t (Current state s) t V(s) is an approximation of the robot's own joint angles, velocities, and other state information collected by sensors. t ):

[0016]

[0017] Where Q(s) t ,a t ) is the state value function, representing the state value in state s. t Take action a t The expected return, logπ(a) t |s t ) indicates that in state s t The entropy of the strategy to take action;

[0018] For the current state s t The state value function V(s) is calculated using the Bellman equation. t Target value:

[0019]

[0020] Where, r t In state s t Take action a t The immediate reward obtained afterward, γ is the discount factor, s t+1 Is taking action a t The next state obtained is then used to update the state value network so that it approximates the state value function.

[0021] The method of introducing the Huber loss function provides a clear optimization direction and quantitative evaluation standard for the state value network by improving the loss function, and outputs the action probability distribution to respond to the gradient signal of the loss function; moreover, the Huber loss function is a robust loss function that combines the advantages of mean squared error (MSE) and mean absolute error (MAE), and dynamically adjusts the error penalty method through the threshold parameter δ.

[0022]

[0023] Where δ is the switching point used to control between squared loss and absolute loss, and Er represents the error between the predicted value and the target value;

[0024] When the absolute value of the prediction error is less than δ, squared loss is used to maintain gradient smoothness and convergence efficiency; when the error exceeds δ, linear loss is switched to suppress the excessive influence of outliers. This piecewise mechanism allows it to converge quickly with small errors like MSE, while also possessing the robustness against outliers of MAE. At the same time, the model accuracy can be flexibly balanced by adjusting δ.

[0025] The composite reward function indirectly guides the training direction and behavioral preferences of the network model by influencing policy evaluation and policy optimization. It includes both direct and indirect rewards, providing an optimization objective for the network model and guiding the agent to learn the desired behavior. The specific details are as follows:

[0026] The direct reward is a speed reward R. v When the linear velocity v of the quadruped robot x Within the expected range [v t 1.5V t When the linear velocity v is within a certain range, a high reward is given; when the linear velocity v is within a certain range, a high reward x Within the range (1.5V) t 3v t Within a certain range, give it a gradually decreasing reward; when the linear velocity v x No reward is given if the value falls within the other ranges; the specific formula is as follows:

[0027]

[0028] Indirect rewards are introduced by using a counter called `stablesteps` to record the number of time steps it takes for the pose to remain stable. An additional reward is given when `stablesteps` reaches a preset time window T. This incentivizes sustained stable movement; conversely, if a posture deviation occurs, stablesteps are reset, and the current posture is evaluated in real time to determine if it is within the normal range. The specific formula for indirect rewards is as follows:

[0029]

[0030] The state space includes the position, posture, and movement speed of the quadruped robot; the angles and velocities of each joint, provided by the joint encoder; the plantar contact information, provided by the plantar pressure sensor; and the previous action, used to assist in decision-making.

[0031] The aforementioned biomimetic motion generation mechanism sets a desired position target, and then adjusts the joint positions of the quadruped robot through a PD controller based on the error between the current position and the desired position, as well as the rate of change of the error, to achieve precise control of the quadruped robot.

[0032] The biomimetic motion generation mechanism consists of the following steps:

[0033] Step (1): Set target position: Determine the target position of each joint according to the specific control task and motion requirements; the target position is the trajectory point calculated by the algorithm, which is a specific position set by the user.

[0034] Step (2): Calculate the error: Calculate the position error of each joint by comparing the current joint position with the target position;

[0035] Step (3): Calculate the error change rate: This is obtained by differentiating the position error. In the quadruped robot motion control task, this invention uses the speed information provided by the sensor to calculate it.

[0036] Step (4): PD controller adjustment: Using position error and error change rate, the control command of the joint is calculated by the PD controller;

[0037] Step (5): Execute control commands: Adjust the joints of the quadruped robot according to the control commands calculated by the PD controller so that it gradually approaches the target position.

[0038] In step (4), the PD controller uses the given target position θ target Current position θ current and speed Then output u, which is expressed as follows:

[0039]

[0040] Where K p It is the proportional gain, used to adjust the effect of position error on control commands; K d It is the differential gain, used to adjust the effect of the rate of change of position error on the control command; (θ) target -θ current ) indicates position error. This represents the rate of change of position error; by adjusting K... p and K d The value of is used to achieve precise control over the joint position of the quadruped robot, allowing it to gradually approach the target position.

[0041] In step S2, robot model information is constructed. SolidWorks is used to create a structural model of the quadruped robot and complete the joint types, inertia matrix, and material properties. Then, the model is imported into the MuJoCo simulation platform to construct simulation environment information that supports contact, friction, and joint actuation.

[0042] The structural model of the quadruped robot includes a central body and four symmetrically distributed mechanical legs. Each leg is composed of a hip link, a thigh link, and a calf link connected in series and connected to the body through the hip joint, thigh joint, and knee joint to form a three-degree-of-freedom kinematic chain. The joints are connected by single-axis rotation and the mounting positions for motors and sensors are reserved to facilitate subsequent simulation modeling and physical realization.

[0043] The mechanism described above utilizes this model to generate control commands to adjust the joint position by setting the target position, calculating the error between the current position and the target, and the rate of change of the error, and then executing them in the MuJoCo simulation environment.

[0044] In S3, the specific simulation training process enables the quadruped robot to achieve stable walking in complex environments such as irregular terrain, obstacle areas, and sloping surfaces, while possessing good anti-interference capabilities and posture recovery capabilities. On the MuJoCo platform, an unstructured complex scene containing elements such as irregular ground, grass, slopes, and mazes is constructed based on MJCF files, and different friction coefficients and slopes are set to simulate differences in the real environment.

[0045] Import the quadruped robot model information, including the motor control interfaces and sensor data channels of 12 joints (hip, thigh, and calf joints) on each leg of the quadruped robot, to realize a closed-loop structure of perception-control-feedback; deploy the model to the simulation environment for multiple rounds of testing to verify the performance of the strategy under different initial postures, terrain changes, and sudden disturbances;

[0046] Evaluation metrics include: average stable steps and distance traveled, attitude angle fluctuation range, travel speed and efficiency, control energy consumption and number of joint overloads; by comparing with traditional control strategies, it is demonstrated that the constructed strategy network has better stability, adaptability and generalization ability.

[0047] In step S4, the model of the policy network trained and converged in the simulation environment is exported and subjected to format conversion and lightweight processing to ensure that it can run on the actual robot embedded computing platform; the policy output action information is connected to the robot's low-level drive interface through ROS or middleware framework to form a control closed loop; the data collected by the real robot's sensors is filtered, synchronized and state extracted to ensure that it is consistent with the state input space used in the simulation training.

[0048] Construct test scenarios that are consistent with or equivalent to the simulation environment, including: a flat ground walking test area, unstructured terrain such as grass and slopes test sections, and external interference test areas such as thrust and foot interference; use IMU and other sensor systems to systematically perform auxiliary measurements and collect robot motion trajectory and posture data;

[0049] Start the robot system, initialize the posture, and ensure that the model is loaded successfully; then continuously run the strategy control module to collect the status input in real time and generate the action output; then execute the stable standing and starting test, uniform linear walking, non-flat ground movement, posture recovery under sudden disturbance, and crossing simple obstacles in sequence.

[0050] The beneficial effects of this invention are:

[0051] Compared to traditional model-based methods, this invention abandons the reliance on environmental modeling and directly drives strategy learning through interactive experience in real or high-fidelity simulation environments, thereby achieving rapid adaptation to dynamic environments and decision optimization.

[0052] The improved SAC algorithm of this invention integrates the state value function and the policy network, which effectively alleviates the problem of low learning efficiency of traditional SAC in high-dimensional continuous action space; and introduces the Huber loss function to improve the noise resistance and training convergence of the state value function, avoiding interference from large errors or abnormal states on network training.

[0053] The composite reward function designed in this invention combines direct rewards based on velocity and indirect rewards based on steady time, which not only strengthens the quadruped robot's intention to move forward, but also constrains the stability of its movements and the continuity of its posture.

[0054] By constructing a multi-element training environment on the MuJoCo platform and loading a 12-DOF quadruped robot model, this invention ensures the authenticity of the training data and the portability of the deployment. Attached Figure Description

[0055] Figure 1 This is a flowchart of a motion control method for a quadruped robot under uncertain conditions.

[0056] Figure 2 This is a diagram of the network model framework used in this invention.

[0057] Figure 3 This is a flowchart of the biomimetic motion generation mechanism involved in this invention.

[0058] Figure 4 This is a structural model diagram of the simple quadruped robot involved in this invention.

[0059] Figure 5 This describes the data acquisition and processing flow in actual motion control.

[0060] Figure 6 This is a diagram of the simulation environment constructed in this invention.

[0061] Figure 7 This is a diagram illustrating the effect of applying the control strategy of this invention to an actual robot. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings.

[0063] like Figures 1-7 As shown, the present invention provides a quadruped robot motion control method based on adaptive deep reinforcement learning, comprising the following steps:

[0064] Step S1: Determine the network model, composite reward function, state space, and biomimetic action generation mechanism of the simulation training environment.

[0065] The state space is used to extract the robot's own state and environmental perception information, which serves as the input to the neural network model. The network model is used to generate actions and evaluate action quality, and is the core of learning the control strategy. The bionic action generation mechanism is responsible for converting the high-level action parameters output by the strategy into actual joint control commands, ensuring motion coordination and physical feasibility. The composite reward function integrates multiple performance indicators, such as speed, posture stability, energy consumption, and joint impact, to guide the strategy to optimize towards the desired behavior.

[0066] The above network model framework is as follows Figure 2 As shown, a model-free controller is constructed using an improved SAC algorithm, which integrates the state value network and the policy network, and introduces the Huber loss function to comprehensively evaluate the "advantages" of the robot's actions, thereby achieving joint control of motion speed and balance.

[0067] Predict the current state s using a state value network model. tApproximate value of the state value function V(s) t ):

[0068]

[0069] Where Q(s) t ,a t ) is the state-action-value function, representing the state action value function in state s. t Take action a t The expected return. logπ(a) t |s t ) indicates that in state s t The strategy entropy for taking action.

[0070] Define the state value function V(s) t The target value for the current state s t The target value is calculated using the Bellman equation:

[0071]

[0072] Where, r t In state s t Take action a t The immediate reward obtained afterward, γ is the discount factor, s t+1 Is taking action a t The next state is then obtained. This objective value is used to update the state value function network to make it approximate the true state value function.

[0073] Secondly, Huber error is introduced into the network model for calculating the loss function:

[0074]

[0075] Here, δ is the switching point used to control the transition between squared loss and absolute loss. Er represents the error between the predicted and target values. By employing the Huber loss function, the motion control of quadruped robots can be made more precise and adaptable to complex environments.

[0076] Then, using gradient descent, the parameters of the state value function network are updated by minimizing the loss function of the state value function, so that the predicted state value function approximation V(s) is obtained. t Approaching the target value V target (s t ):

[0077]

[0078] In the parameter update formula of the above state value network, φ iLet represent the parameters of the value network at the i-th state, and η be the learning rate, used to control the step size of gradient descent. This indicates the parameter φ i The gradient derivative operation; V is a training batch sampled from the experience replay buffer, containing multiple quadruplets (s, a, r, s′), representing the current state, the action performed, the immediate reward obtained, and the next state after the transition, respectively; |B| is the number of samples in the batch; V(s) t ) is the current state value network for state s t The estimated value, V target (s t ) is the target state value used for supervised training.

[0079] Next, in step S1 above, a composite reward function is designed to indirectly guide the training direction and behavioral preferences of the network model by influencing policy evaluation and policy optimization. The specific content is as follows:

[0080] Direct Reward: Compared to traditional methods, this invention sets the desired speed within a range and uses the following speed reward R. v This means that when the quadruped robot's speed is within the desired speed range, the reward is a high positive value; as the quadruped robot's speed gradually approaches the desired speed boundary, the reward gradually decreases, indicating that the speed needs to be adjusted to maintain within the desired range. This setting not only allows the quadruped robot to dynamically adjust its speed according to the current environment and task requirements, better adapting to various complex situations, but also enables the quadruped robot to more reliably cope with various challenges in practical applications. Speed ​​reward function R v The mathematical formula is as follows:

[0081]

[0082] Among them, v t It is a target speed set by humans, v x It is the linear velocity of the quadruped robot in the forward direction;

[0083] Indirect reward: A counter `stablesteps` is introduced to record the number of time steps it takes for the pose to remain stable. When `stablesteps` reaches a preset time window T, an additional reward is given. This encourages sustained, stable movement over a long period of time;

[0084] Conversely, if a posture deviation occurs, stablesteps are reset, and the current posture is evaluated in real time to determine if it is within the normal range; the specific formula for indirect rewards is as follows:

[0085]

[0086] Where, ω bThis represents the basic reward, ω. i It is the indirect reward coefficient, stablesteps is the number of time steps, and T is the preset time window.

[0087] Next, the state space involved in step S1 above, including the robot's own motion state, terrain perception information, and multi-source features such as the direction of the task target, constitutes a high-dimensional vector input; the network model performs feature extraction and policy mapping based on this state input to achieve the learning of states to actions; the specific content is as follows:

[0088] The state space construction includes information on multiple aspects of the quadruped robot, such as position, posture, motion speed, joint angles, and velocities. Gyroscopes and accelerometers are used to acquire the quadruped robot's posture and linear acceleration information, reflecting its current orientation and acceleration changes and providing crucial reference for its current state. Furthermore, joint encoders are used to acquire the angle and velocity information of each joint, comprehensively describing the motion state of each joint and helping the quadruped robot understand its movement patterns and posture changes. Foot pressure sensors are used to detect whether the quadruped robot's feet are in contact with the ground. By monitoring the contact status of each foot, the quadruped robot's gait and posture can be adjusted in real time to ensure proper foot contact with the ground and maintain stable movement and balance.

[0089] Finally, by using the previous action as part of the state space, the quadruped robot can make decisions based on historical experience, thereby improving its intelligence and adaptability.

[0090] The biomimetic motion generation mechanism in step S1 above is as follows: Figure 4 As shown, this is to achieve the PD position target of the 12 joints of the quadruped robot. By setting the desired position target, and then adjusting the joint position of the quadruped robot by the controller according to the error between the current position and the desired position, as well as the rate of change of the error, the quadruped robot can be precisely controlled.

[0091] The biomimetic motion generation mechanism consists of the following steps:

[0092] Step 1: Set target position: Determine the target position of each joint based on the specific control task and motion requirements; this target position is a trajectory point calculated by an algorithm and is a specific position set by the user.

[0093] Step 2: Calculate the error: Calculate the positional error of each joint by comparing the current joint position with the target position;

[0094] Step 3: Calculate the rate of change of error: In addition to the position error, it is also necessary to calculate the rate of change of position error, that is, the first derivative of the error; this is obtained by differentiating the position error. In the motion control task of the quadruped robot, this invention uses the velocity information provided by the sensor to calculate it.

[0095] Step 4: PD controller adjustment: Using position error and error change rate, the PD controller calculates the control command for the joint;

[0096] Step 5: Execute control commands: Based on the control commands calculated by the PD controller, adjust the joints of the quadruped robot to gradually approach the target position.

[0097] Furthermore, in this invention, the PD controller in step 4 uses a given target position θ target Current position θ current and speed Then output u, whose mathematical expression is as follows:

[0098]

[0099] Where K p It is the proportional gain, used to adjust the effect of position error on control commands; K d It is the differential gain, used to adjust the effect of the rate of change of position error on the control command. (θ) target -θ current ) indicates position error. This represents the rate of change of position error; by adjusting K... p and K d The value of is used to achieve precise control over the joint position of the quadruped robot, allowing it to gradually approach the target position.

[0100] The limitation of the motion space directly affects the range of motion that a quadruped robot can perform, determining its behavioral possibilities in a specific environment. By limiting the motion space, it can be ensured that the quadruped robot does not exceed its capabilities when performing actions, thereby improving the stability and reliability of control.

[0101] The above Figure 1 S2 involved in this process is to establish a simulation training environment for deep reinforcement learning of the quadruped robot. The simulation training environment includes the model information of the quadruped robot and the simulated environment information.

[0102] like Figure 4As shown, the structural model of the quadruped robot was designed using the 3D mechanical modeling software SolidWorks, including a central body and four symmetrically distributed mechanical legs. Each leg is composed of a hip link, a thigh link, and a calf link connected in series, and connected to the body through three rotary joints (hip joint, thigh joint, and knee joint) to form a three-degree-of-freedom kinematic chain. The joints are connected by single-axis rotation, and mounting positions for motors and sensors are reserved to facilitate subsequent simulation modeling and physical realization.

[0103] After the design is completed, the model is exported as a unified robot description URDF format file. This format can be used to describe information such as robot structural hierarchy, kinematic relationships between connected bodies, mass and inertial parameters, collision bodies and visualization mesh.

[0104] Since MuJoCo natively supports the MJCF format, the exported URDF file needs to be converted to MJCF format using an intermediate tool. This process requires precise mapping of joint types (such as rotation and sliding), differentiation between visual models (such as textures and materials) and collision models (simplified geometry), completion of dynamic properties such as the center of mass and inertia matrix, and adjustment of the coordinate system orientation and initial joint angles according to MuJoCo's specifications.

[0105] Next, the converted MJCF file is imported into the MuJoCo simulation engine to build a complete quadruped robot simulation environment.

[0106] This simulation environment supports real-time physical simulation (including ground contact, friction, joint actuation, etc.), custom control interfaces for connecting subsequent control algorithms, setting up complex environments with different terrains, disturbances, obstacles, etc., and acquiring robot state information such as joint position, velocity, contact force, body posture, etc., for feedback control and reinforcement learning.

[0107] The robot itself mainly includes modules such as data acquisition module, perception and processing module, control strategy module, motion execution module, and feedback correction module.

[0108] The data acquisition module collects information on various aspects of the quadruped robot, including its posture, speed, and joint angles. The gyroscope outputs attitude data (yaw, pitch, and roll) via a microelectromechanical system (MEMS), and uses adaptive Kalman filtering to eliminate zero-bias noise, ensuring accurate data acquisition. A miniature IMU is integrated into the foot, and a contact detection algorithm enables zero-velocity correction during ground contact, suppressing integral drift errors. Hardware timestamps are used to synchronize data from multiple sensors, and SE coordinate transformation unifies the data to the robot's center-of-mass coordinate system.

[0109] The perception processing module uses preprocessing algorithms to perform data noise reduction and feature extraction, and then inputs the preprocessed state data into the locally deployed network model for decision-making. The control strategy module integrates the core algorithm framework, obtains the output control strategy information from the input 48-dimensional state information and 12-dimensional joint information, and then controls the 12 motors of the quadruped robot to reach the corresponding angle through the motion execution module; if there is a deviation, the feedback correction module will correct the deviated joints to achieve a stable control effect.

[0110] The complex external environment is constructed using the simulation platform MuJoCo. MuJoCo represents real-world objects using basic geometric shapes (such as planes, cubes, cylinders, and spheres). Each geometric shape can be configured, for example, a box to represent a wall, a plane to represent the ground, and a cylinder to represent a pillar. Position and orientation determine the object's spatial location within the environment. Color, reflections, and textures can also be set to enhance visual effects, including physical properties such as collision resistance, coefficient of friction, and density to control physical behavior.

[0111] The above Figure 1 S3, which involves training the constructed policy network model in the simulation training environment based on the robot information and the simulation environment information, is used to test whether the constructed policy network model is feasible.

[0112] The specific training objective is to enable the quadruped robot to achieve stable walking and autonomous navigation in complex environments such as irregular terrain, obstacle areas, and sloping surfaces, while also possessing good anti-interference capabilities and posture recovery abilities. Within the MuJoCo platform, this is specifically achieved through... Figure 6 As shown, an unstructured complex scene containing elements such as irregular ground, grass, slopes, and mazes is constructed based on MJCF files. Different friction coefficients and slopes are set to simulate differences in the real environment. At the same time, a quadruped robot body model is imported, and 12 joint motor control interfaces and sensor data channels are defined to realize a closed-loop structure of perception-control-feedback.

[0113] Based on the completion of state space design, composite reward function construction, and integration of biomimetic action generation mechanism, a policy network model based on deep reinforcement learning was constructed and deployed in a simulation training environment for multiple rounds of training. Policy optimization was carried out under different initial poses, terrain undulations, and perturbation conditions until the model converged. The model was then deployed to a simulation environment for multiple rounds of testing to verify the performance of the policy under different initial poses, terrain changes, and sudden disturbances. Evaluation metrics included: average stable steps and distance traveled, pose angle fluctuation range, travel speed, and efficiency. Among these, the average stable steps and distance traveled reflect the robot's ability to continuously walk while maintaining a stable gait; the pose angle fluctuation range measures the smoothness of the robot's posture control during movement, with small fluctuations being desirable; travel speed and efficiency indicate stable progress within the desired speed range, with better speed and continuity as close to the target speed as possible. Through comprehensive evaluation of the above metrics, this invention demonstrates excellent performance in multiple dimensions, proving that the constructed policy network possesses superior stability and generalization ability, and is more valuable for practical applications compared to traditional control strategies.

[0114] The above Figure 1 S4 involved in this process is to deploy the network model into a quadruped robot to verify the effectiveness of the method of the present invention in motion control in a quadruped robot.

[0115] Motion control effect in real-world environments, such as Figure 7 As shown, the specific implementation process is as follows:

[0116] The policy network model trained and converged in the simulation environment is exported and subjected to format conversion and lightweight processing to ensure it can run on the actual robot embedded computing platform. The policy output action information is interfaced with the robot's low-level drive interface (motor controller) through ROS or a middleware framework to form a control closed loop. Data collected by real robot sensors (IMU, joint encoders, foot contacts, etc.) is filtered, synchronized, and its state is extracted to ensure consistency with the state input space used in the simulation training.

[0117] Construct test scenarios that are consistent with or equivalent to the simulation environment, including: a flat ground walking test area, test sections with unstructured terrain such as grass and slopes, and an external interference test area such as thrust and foot interference. Use sensors such as IMUs to systematically perform auxiliary measurements and collect robot motion trajectory and attitude data.

[0118] Start the robot system, initialize the posture, and ensure that the model is loaded successfully; then continuously run the strategy control module to collect state input in real time and generate action output; then sequentially perform stable standing and starting tests, uniform linear walking, non-flat ground movement, posture recovery under sudden disturbances, and crossing simple obstacles.

Claims

1. A motion control method for a quadruped robot based on adaptive deep reinforcement learning, characterized in that, Includes the following steps; S1: Determine the network model, composite reward function, state space, and biomimetic action generation mechanism for the simulation training environment; The state space provides environmental information input, the network model processes the input information and generates decisions, the biomimetic action mechanism executes specific decision-making behaviors, and the composite reward function evaluates the effects of the behaviors and optimizes the decision direction. S2: Construct a simulation training environment for the quadruped robot, wherein the simulation environment includes quadruped robot model information and simulation environment information; S3: Based on the robot model information and the simulation environment information, the network model is trained using a deep reinforcement learning algorithm to obtain the trained motion control strategy; S4: Verify the feasibility of using the trained motion control strategy by controlling the movement of the quadruped robot in a real environment.

2. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 1, characterized in that, In S1, the network model uses an improved SAC algorithm to construct a model-free controller. The improved SAC algorithm includes a network model that integrates the state value network and the policy network. Among them, the state value network is used to estimate the expected long-term reward in the current state, providing a stable value benchmark for policy optimization; the policy network, with the goal of maximizing the function, outputs continuous motion distribution parameters to guide the quadruped robot in motion control in uncertain environments. The Huber loss function is introduced to evaluate the state value network; the state value network is used to predict the current state s. t Approximate value V(s) t ): Where Q(s) t ,a t ) is the state value function, representing the state value in state s. t Take action a t The expected return, logπ(a) t |s t ) indicates that in state s t The entropy of the strategy to take action; For the current state s t The state value function V(s) is calculated using the Bellman equation. t Target value: Where, r t In state s t Take action a t The immediate reward obtained afterward, γ is the discount factor, s t+1 Is taking action a t The next state obtained is then used to update the state value network so that it approximates the state value function.

3. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 2, characterized in that, The method of introducing the Huber loss function provides a clear optimization direction and quantitative evaluation standard for the state value network by improving the loss function, and outputs the action probability distribution to respond to the gradient signal of the loss function; the error penalty method is dynamically adjusted by the threshold parameter δ. Where δ is the switching point used to control between squared loss and absolute loss, and Er represents the error between the predicted value and the target value; When the absolute value of the prediction error is less than δ, squared loss is used to maintain gradient smoothness and convergence efficiency; when the error exceeds δ, linear loss is switched to suppress the excessive influence of outliers.

4. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 3, characterized in that, The composite reward function indirectly guides the training direction and behavioral preferences of the network model by influencing policy evaluation and policy optimization, including both direct and indirect rewards; the specific details are as follows: The direct reward is a speed reward R. v When the linear velocity v of the quadruped robot x Within the expected range [v t 1.5V t When the linear velocity v is within a certain range, a high reward is given; when the linear velocity v is within a certain range, a high reward is given. x Within the range (1.5V) t 3v t Within a certain range, give it a gradually decreasing reward; when the linear velocity v x No reward is given if the value falls within the other ranges; the specific formula is as follows: Indirect rewards are introduced by using a counter called `stablesteps` to record the number of time steps it takes for the pose to remain stable. An additional reward is given when `stablesteps` reaches a preset time window T. This incentivizes sustained stable movement; conversely, if a posture deviation occurs, stablesteps are reset, and the current posture is evaluated in real time to determine if it is within the normal range. The specific formula for indirect rewards is as follows:

5. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 4, characterized in that, The state space includes the position, posture, and movement speed of the quadruped robot; the angles and velocities of each joint, provided by the joint encoder; the plantar contact information, provided by the plantar pressure sensor; and the previous action, used to assist in decision-making.

6. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 5, characterized in that, The aforementioned biomimetic motion generation mechanism sets a desired position target, and then adjusts the joint positions of the quadruped robot through a PD controller based on the error between the current position and the desired position, as well as the rate of change of the error, to achieve precise control of the quadruped robot. The biomimetic motion generation mechanism consists of the following steps: Step (1): Set target position: Determine the target position of each joint according to the specific control task and motion requirements; the target position is the trajectory point calculated by the algorithm, which is a specific position set by the user. Step (2): Calculate the error: Calculate the position error of each joint by comparing the current joint position with the target position; Step (3): Calculate the error change rate: This is obtained by differentiating the position error. In the quadruped robot motion control task, this invention uses the speed information provided by the sensor to calculate it. Step (4): PD controller adjustment: Using position error and error change rate, the control command of the joint is calculated by the PD controller; Step (5): Execute control commands: Adjust the joints of the quadruped robot according to the control commands calculated by the PD controller so that it gradually approaches the target position.

7. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 6, characterized in that, In step (4), the PD controller uses the given target position θ target Current position θ current and speed Then output u, which is expressed as follows: Where K p It is the proportional gain, used to adjust the effect of position error on control commands; K d It is the differential gain, used to adjust the effect of the rate of change of position error on the control command; (θ) target -θ current ) indicates position error. This represents the rate of change of position error; by adjusting K... p and K d The value of is used to achieve precise control over the joint position of the quadruped robot, allowing it to gradually approach the target position.

8. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 7, characterized in that, In step S2, robot model information is constructed. SolidWorks is used to create a structural model of the quadruped robot and complete the joint types, inertia matrix, and material properties. Then, the model is imported into the MuJoCo simulation platform to construct simulation environment information that supports contact, friction, and joint actuation. The structural model of the quadruped robot includes a central body and four symmetrically distributed mechanical legs; each leg is composed of a hip link, a thigh link and a calf link connected in series, and connected to the body through the hip joint, thigh joint and knee joint to form a three-degree-of-freedom kinematic chain; The joints are connected by a single-axis rotation, and mounting positions for motors and sensors are reserved to facilitate subsequent simulation modeling and physical implementation; The mechanism described above utilizes this model to generate control commands to adjust the joint position by setting the target position, calculating the error between the current position and the target, and the rate of change of the error, and then executing them in the MuJoCo simulation environment.

9. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 8, characterized in that, In S3, the specific simulation training process is to enable the quadruped robot to walk stably in complex environments with irregular terrain, obstacle areas, and sloping ground. In the MuJoCo platform, an unstructured complex scene containing irregular ground, grass, slopes, and mazes is constructed based on MJCF files, and different friction coefficients and slopes are set to simulate the differences in the real environment. Import the quadruped robot model information, including the motor control interfaces and sensor data channels of 12 joints (hip, thigh, and calf joints) on each leg of the quadruped robot, to realize a closed-loop structure of perception-control-feedback; deploy the model to the simulation environment for multiple rounds of testing to verify the performance of the strategy under different initial postures, terrain changes, and sudden disturbances; Evaluation metrics include: average stable steps and distance traveled, range of posture angle fluctuations, speed and efficiency of movement, energy consumption control, and number of joint overloads.

10. The method for motion control of a quadruped robot based on adaptive deep reinforcement learning according to claim 9, characterized in that, In step S4, the model of the policy network trained and converged in the simulation environment is exported and subjected to format conversion and lightweight processing to ensure that it can run on the actual robot embedded computing platform; the policy output action information is connected to the robot's low-level drive interface through ROS or middleware framework to form a control closed loop; the data collected by the real robot's sensors is filtered, synchronized and state extracted to ensure that it is consistent with the state input space used in the simulation training. Construct test scenarios that are consistent with or equivalent to the simulation environment, including: a flat ground walking test area, unstructured terrain such as grass and sloping test sections, and an external interference test area such as thrust and foot interference; use IMU and other sensor systems to systematically perform auxiliary measurements and collect robot motion trajectory and posture data; Start the robot system, initialize the posture, and ensure that the model is loaded successfully; then continuously run the strategy control module to collect the status input in real time and generate the action output; then execute the stable standing and starting test, uniform linear walking, non-flat ground movement, posture recovery under sudden disturbance, and crossing simple obstacles in sequence.

Images