Self-adaptive gait generation method and system for hierarchical reinforcement learning of small robot
By employing a hierarchical reinforcement learning approach, combined with environmental perception and multibody dynamics models, an adaptive gait control strategy for small robots is generated. This solves the problem of gait adaptation in complex terrains and enhances the robot's motion intelligence and terrain adaptability.
Patent Information
- Application Number
- CN202511705325.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2045-11-20
AI Technical Summary
Gait adaptation control for small robots in complex terrain presents challenges. Traditional methods lack environmental perception and response capabilities, leading to low mobility or loss of control. Existing reinforcement learning strategies are difficult to train in multi-task environments and have high decision-making dimensions, making it difficult to guarantee accuracy and real-time performance.
A hierarchical reinforcement learning approach is adopted. The perception module collects environmental and robot state data, the high-level planning module generates gait planning instructions, and the multibody dynamics model is combined to generate joint control instructions, thereby realizing adaptive gait generation.
It enhances the robot's mobility in complex environments and its ability to quickly adapt to unknown terrain, ensuring the precision and real-time nature of control.
Smart Images

Figure CN121157055A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of robots, and particularly relates to a small robot hierarchical reinforcement learning gait adaptive generation method and system. BACKGROUND
[0002] Small robots face great challenges in gait adaptive control on complex terrains. Traditional gait control methods mostly adopt pre-programmed fixed gaits, lack the ability to perceive and respond to dynamic changes in the environment, and thus result in low mobility or even falling down when encountering unknown or uneven ground. Although some studies attempt to apply reinforcement learning to robot control, a single reinforcement learning strategy often has problems such as training difficulty, high decision dimension, slow convergence speed, etc. when facing multi-task and high dynamic environment, and it is difficult to ensure accuracy and real-time performance in joint control at the bottom level. In addition, the motion control of a robot is a typical hierarchical problem, and high-level decision-making and low-level execution are tightly coupled. Existing methods usually fail to effectively divide the decision levels and integrate the multi-body dynamics model into the learning framework, resulting in physically unfeasible or energy-inefficient gaits, which limits the autonomy and adaptability of the robot. SUMMARY
[0003] The purpose of the present application is to provide a small robot hierarchical reinforcement learning gait adaptive generation method and system to solve the problems in the prior art and to solve the adaptive generation of gaits in complex environments, improve the motion intelligence, control accuracy and rapid adaptation ability to unknown terrains of the robot.
[0004] One embodiment of the present application provides a small robot hierarchical reinforcement learning gait adaptive generation method, which comprises: collecting environment perception data and robot body state data through a perception module, and extracting environment features and robot state information; generating gait planning instructions through a high-level planning module based on the environment features and robot state information, wherein the gait planning instructions at least include gait modes or gait parameters; generating joint control instructions by combining a multi-body dynamics model through a low-level control module based on the gait planning instructions and robot state information, wherein the multi-body dynamics model includes an inertia matrix, a gravity term and a Coriolis force term; executing the joint control instructions to control the joint motion of the robot to achieve adaptive gait generation.
[0005] Optionally, the collecting environment perception data and robot body state data through a perception module, and extracting environment features and robot state information comprises: The environment original data set is generated by collecting environment original data through a visual sensor, and acquiring an image sequence or point cloud data of a front terrain through a camera or a laser radar; The body state original data set is generated by collecting robot body state original data through an inertial measurement unit, and acquiring three-axis acceleration, three-axis angular velocity and three-axis attitude angle data of the robot through an IMU; The environment feature vector is generated by performing feature extraction processing on the environment original data set, and applying a convolutional neural network to extract a slope level, a unevenness index and an obstacle distribution feature of a terrain height map; The robot state information vector is generated by performing state fusion processing on the body state original data set, and calculating joint angle, joint angular velocity, foot contact state and body tilt angle through a sensor fusion algorithm.
[0006] Optionally, the gait planning instruction is generated through a high-level planning module based on the environment feature and the robot state information, wherein the gait planning instruction at least includes a gait mode or a gait parameter, including: The environment feature vector and the robot state information vector are input into a high-level reinforcement learning strategy network, the network learns a mapping relationship from terrain features to gait modes through pre-training, and generates a gait mode probability distribution; The current optimal gait mode is selected from a pre-defined gait mode library by using a greedy strategy or random sampling based on the gait mode probability distribution, the mode includes a flat ground walking mode, an uphill mode, a downhill mode and an obstacle crossing mode, and a gait mode identifier is generated; The parameter generator is called according to the gait mode identifier, the motion parameters in the robot state information vector are combined, the step length, step height, gait cycle and leg lifting height parameters are dynamically adjusted through a reinforcement learning reward function, and a gait parameter set is generated; The gait mode identifier and the gait parameter set are integrated to construct an instruction protocol, which is encapsulated as a target foot position sequence, an expected body attitude angle and a gait timing parameter, and a standardized gait planning instruction is generated.
[0007] Optionally, the joint control instruction is generated through a low-level control module combined with a multi-body dynamics model based on the gait planning instruction and the robot state information, wherein the multi-body dynamics model includes an inertia matrix, a gravity term and a Coriolis force term, including: The current motion state is analyzed from the robot state information vector, the real-time angle, angular velocity and body space attitude of each joint are extracted, and a current joint state matrix is generated; The multi-body dynamics parameters are calculated based on the current joint state matrix, the Newton-Euler recursive algorithm is used to solve the robot dynamics equation, the inertia matrix, the gravity vector and the Coriolis force vector are obtained, and a dynamics parameter set is generated; Input the standardized gait planning instruction, the current joint state matrix and the set of dynamic parameters into a low-level reinforcement learning strategy network, which is trained by deep reinforcement learning and fuses the kinetic energy balance reward and the inertia compensation reward to generate a preliminary joint control quantity; Optimize the preliminary joint control quantity by dynamics constraint, apply the reinforcement learning strategy gradient method to correct the joint target rotation angle and torque value, and output the final joint control instruction.
[0008] Optionally, the executing the joint control instruction to control the robot joint movement to realize adaptive gait generation comprises: Convert the final joint control instruction into motor driving signals, convert the target joint angle and torque into motor PWM modulation signals through inverse kinematics calculation to generate a motor driving instruction set; Execute the motor driving instruction set through the robot joint servo system, transmit the PWM signals to the servo motors of the hip joint, knee joint and ankle joint, drive each joint to move according to the instruction requirements, and generate the actual joint movement trajectory; Real-time acquisition of joint movement feedback data, reading of the actual rotation angle of each joint through the encoder and detection of the foot contact force through the force sensor, combination of the IMU to update the body attitude data, and generation of the movement feedback data set; Input the movement feedback data set into the perception module to update the robot state information vector, form a reinforcement learning closed-loop control loop, and iteratively execute the gait generation process based on the updated state information to realize continuous adaptive gait adjustment.
[0009] Yet another embodiment of the present application provides a small robot layered reinforcement learning gait adaptive generation system, which comprises: A collection module for collecting environmental perception data and robot body state data through a perception module and extracting environmental features and robot state information; A first generation module for generating a gait planning instruction based on the environmental features and robot state information through a high-level planning module, wherein the gait planning instruction at least comprises a gait mode or gait parameters; A second generation module for generating a joint control instruction based on the gait planning instruction and robot state information through a low-level control module in combination with a multi-body dynamics model, wherein the multi-body dynamics model comprises an inertia matrix, a gravity term and a Coriolis force term; A control module for executing the joint control instruction to control the robot joint movement to realize adaptive gait generation.
[0010] Yet another embodiment of the present application provides a storage medium having a computer program stored therein, wherein the computer program is set to execute the method described in any of the above embodiments when running.
[0011] Yet another embodiment of the present application provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the method described in any of the above.
[0012] Compared with the prior art, the small robot hierarchical reinforcement learning gait adaptive generation method provided by the present application can collect environment perception data and robot body state data through a perception module, and extract environment features and robot state information; based on the environment features and robot state information, a high-level planning module generates gait planning instructions; based on the gait planning instructions and robot state information, a low-level control module generates joint control instructions in combination with a multi-body dynamics model; the joint control instructions are executed to control the joint movement of the robot to realize adaptive gait generation, thereby being able to solve the adaptive generation problem of gait in a complex environment, and improving the motion intelligence, control accuracy and rapid adaptation capability to unknown terrain of the robot. BRIEF DESCRIPTION OF DRAWINGS
[0013] Figure 1 A hardware structure block diagram of a computer terminal of the small robot hierarchical reinforcement learning gait adaptive generation method provided by the embodiment of the present application is shown in the figure. Figure 2 A flowchart of the small robot hierarchical reinforcement learning gait adaptive generation method provided by the embodiment of the present application is shown in the figure. Figure 3 A structure diagram of the small robot hierarchical reinforcement learning gait adaptive generation system provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0014] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be explained as a limitation of the present application.
[0015] The embodiment of the present application first provides a small robot hierarchical reinforcement learning gait adaptive generation method, which can be applied to an electronic device, such as a computer terminal, specifically, a general computer, etc.
[0016] The small robot hierarchical reinforcement learning gait adaptive generation method provided by the embodiment of the present application is described in detail below by taking a computer terminal as an example. Figure 1 A hardware structure block diagram of a computer terminal of the small robot hierarchical reinforcement learning gait adaptive generation method provided by the embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, the computer device comprises a processor, a memory and a network interface connected through a system bus, wherein the memory can comprise a non-volatile storage medium and an internal memory.
[0017] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions which, when executed, can cause the processor to perform any small robot hierarchical reinforcement learning gait adaptive generation method.
[0018] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.
[0019] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium, which, when executed by the processor, can cause the processor to perform any small robot hierarchical reinforcement learning gait adaptive generation method.
[0020] The network interface is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that, Figure 1 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0021] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor.
[0022] Referring to Figure 2 The embodiment of the present application provides a small robot hierarchical reinforcement learning gait adaptive generation method, which can include the following steps: S201, collecting environment perception data and robot body state data by a perception module, and extracting environment features and robot state information; Specifically, the environment raw data can be collected by a visual sensor, and the image sequence or point cloud data of the front terrain can be obtained by a camera or a laser radar to generate an environment raw data set; This step is the basic link of environment perception. The external terrain information is captured by a high-precision visual sensor to provide raw materials for subsequent feature extraction. The specific implementation manner is as follows: The selection of visual sensors needs to adapt to the size of the small biped robot and the walking scene, and the combination of "camera + lidar" is preferred, which takes into account image details and three-dimensional spatial information. The camera selects an industrial-grade RGB-D depth camera, with a resolution of 1920x1080 pixels, a frame rate of 30Hz, a lens focal length of 8mm, a horizontal field of view of 60°, and a vertical field of view of 45°. It can output color images and depth images simultaneously, with a range of 0.5-10 meters, meeting the robot's front terrain detection needs. The lidar uses a small solid-state lidar, with a range of 0.1-20 meters, a point cloud density of 100,000 points per second, a horizontal field of view of 120°, a vertical field of view of 15°, and a sampling frequency of 10Hz. It can penetrate light barriers (such as grass) and obtain three-dimensional point cloud data of the terrain. The sensor is installed on the robot's head gimbal, with a height of 0.3 meters from the ground. The gimbal can rotate horizontally by ±30° and vertically by -10° to 20°, ensuring coverage of the main walking area within 10 meters in front.
[0023] The data acquisition process needs to adjust parameters adaptively for different terrain scenes: on flat terrain (such as indoor floor), the camera automatically reduces the exposure time to 1 / 500 seconds to avoid glare interference, and the lidar maintains the default point cloud density; on uphill terrain (such as 5-15° gentle slope), the gimbal pitch angle is adjusted to 15° to expand the detection range of the uphill area, and the lidar increases the sampling frequency to 15Hz to enhance the capture of terrain height changes; on obstacle terrain (such as a 0.8-meter-high stone in front), the camera frame rate is increased to 60Hz, and 10 frames of image sequences are continuously collected, the lidar focuses on the obstacle area, and the point cloud density is temporarily increased to 150,000 points per second to ensure the integrity of the obstacle profile.
[0024] The generated environmental raw data set is structured in a "timestamp-sensor type-data content" structure, with each piece of data attached to a timestamp accurate to the millisecond (e.g. "20251030_152030.123"), distinguishing between image sequences and point cloud data types. For example, a data set segment under flat terrain is: "timestamp 20251030_152030.123, sensor type RGB-D camera, data content: color image (1920x1080, RGB three channels), depth image (1920x1080, single channel depth value, unit mm); timestamp 20251030_152030.223, sensor type lidar, data content: point cloud data (100,000 three-dimensional coordinate points, x / y / z unit m, x axis along the robot forward direction, y axis horizontal perpendicular to the forward direction, z axis perpendicular to the ground upward)". The data set also records the sensor state at the time of collection (such as camera exposure value, radar working mode), ensuring data traceability and providing complete and high-quality raw input for subsequent feature extraction.
[0025] The robot body state raw data is collected by an inertial measurement unit, and the IMU is used to obtain the three-axis acceleration, three-axis angular velocity and three-axis attitude angle data of the robot, to generate a body state raw data set; This step focuses on capturing the motion state of the robot itself, and the high-precision IMU is used to obtain the core kinetic parameters to provide basic data for body state fusion. The specific implementation is as follows: The inertial measurement unit (IMU) uses a small MEMS IMU module, with a sampling frequency of 100 Hz, meeting the capture requirements of rapid attitude changes when the robot is walking. The measurement range is acceleration ±16g (g=9.8m / s 2 ), angular velocity ±2000° / s, attitude angle ±180°, and the measurement accuracy is acceleration ±0.01g, angular velocity ±0.1° / s, attitude angle ±0.5°. It has temperature compensation function (working temperature -10℃ to 60℃), which can offset the influence of environmental temperature change on data. The IMU is installed at the center of the robot's torso (near the center of mass), rigidly connected to the torso, reducing vibration interference caused by joint motion. The data is transmitted in real time to the robot's main control unit through the SPI interface.
[0026] The data collection process is synchronized in real time with the motion state of the robot. When the robot is in different synchronous states, the IMU continuously captures key states: when walking on flat ground, the three-axis acceleration mainly fluctuates along the z axis (perpendicular to the ground), with a value of 8-12m / s 2 , and the x axis (forward direction) acceleration is -1 to 1m / s 2Between (acceleration and deceleration), the y-axis (lateral) acceleration is close to 0; when walking uphill, the average of the x-axis acceleration becomes 0.5-1.5m / s 2 (overcome gravity component), body tilt causes slight decrease in z-axis acceleration; when turning, y-axis angular velocity shows obvious peak value (e.g. when turning left, y-axis angular velocity is positive, value 50-100° / s).
[0027] The generated ontology state raw data set is organized in the format of "timestamp - three-axis acceleration - three-axis angular velocity - three-axis attitude angle", and each piece of data is accurately aligned with the timestamp of the environment raw data set (error ≤1ms), which is convenient for subsequent multi-source data fusion. For example, the data set segment when the robot walks uphill is: "timestamp 20251030_152030.123, three-axis acceleration (x=1.2m / s 2 , y=0.1m / s 2 , z=9.5m / s 2 ), three-axis angular velocity (x=0.3rad / s, y=0.05rad / s, z=0.1rad / s), three-axis attitude angle (roll angle=0.5°, pitch angle=8.2°, yaw angle=3.1°); timestamp 20251030_152030.133, three-axis acceleration (x=1.1m / s 2 , y=0.08m / s 2 , z=9.6m / s 2 ), three-axis angular velocity (x=0.28rad / s, y=0.04rad / s, z=0.09rad / s), three-axis attitude angle (roll angle=0.4°, pitch angle=8.1°, yaw angle=3.2°)". The data set also records the working state of the IMU (such as whether it is in temperature compensation mode), ensuring the reliability of the raw data.
[0028] Feature extraction is performed on the environment raw data set, and convolutional neural network is applied to extract the slope level, unevenness index and obstacle distribution features of the terrain height map, generating an environment feature vector; This step uses deep learning algorithms to extract effective information from raw environment data, converting unstructured data into structured features, providing decision-making basis for high-level planning. The specific implementation is as follows: First, the original environmental data is preprocessed: the depth image output by the RGB-D camera needs to be denoised, and Gaussian filtering (kernel size 5x5, standard deviation σ=1.0) is used to eliminate depth noise, and then the missing pixels (such as invalid depth values caused by too far distance) are interpolated to generate a complete terrain height map (resolution reduced to 640x480 pixels, unit m); the point cloud data output by the laser radar needs to be ground segmented, and the RANSAC algorithm is used to fit the ground plane, and after removing the ground points, the non-ground points (potential obstacles) are retained, and then the point cloud is projected onto a two-dimensional grid (grid size 0.1m x 0.1m), generating an aligned height map and obstacle binary map with the depth image.
[0029] Feature extraction uses a lightweight convolutional neural network (CNN), with a reference architecture of MobileNetV3-small, adapted to the computational power limitations of the robot's main control unit, with a network input of 640x480x2 dual-channel images (height map + obstacle binary map), extracting multi-scale features through depth separable convolutional layers: shallow convolutional layers (convolution kernel 3x3, step 1) extract terrain detail features (such as small bumps), medium convolutional layers (convolution kernel 5x5, step 2) extract medium-scale features (such as slope changes), and deep convolutional layers (convolution kernel 7x7, step 2) extract global features (such as large-scale obstacle distribution), and finally output a 32-dimensional environmental feature vector through global average pooling and fully connected layers.
[0030] The specific feature extraction logic is as follows: Slope level: calculate the terrain slope through the height map, divide the image into 10x10 local regions, and in each region, use linear regression to fit the ground slope, convert it to a slope angle (unit °), and then divide it into levels according to the slope angle (0-5° for 0-level flat ground, 5-15° for 1-level gentle slope, 15-30° for 2-level steep slope, >30° for 3-level impassable), the network learns the texture features of different slope levels through feature mapping, and finally outputs a 1-dimensional slope level feature (value range 0-3); Unevenness index: calculate the height variance (unit m 2 ) of each local region in the height map, and then take the mean of the variances of all regions to get the global unevenness index (reflecting the terrain flatness, the larger the value, the more rugged the terrain), the network captures the height fluctuation pattern through the medium-level features, and outputs a 1-dimensional unevenness index feature (value range 0-0.5); Obstacle distribution features: The position (relative to the x / y coordinates of the robot, unit: m), size (length x width, unit: m), and density (number of obstacles per unit area) of the obstacles are extracted from the obstacle binary image. The network integrates the global distribution of obstacles through deep features and outputs a 30-dimensional obstacle distribution feature (including the position and size information of 10 potential obstacles, and 0 is supplemented if insufficient).
[0031] The generated environment feature vector is a 32-dimensional real number vector, each dimension is normalized (mapped to the 0-1 interval), which is convenient for subsequent high-level strategy network processing. For example, the front of the robot is a gentle slope (slope 8°), slightly uneven (unevenness 0.03m 2 ), and there is a 0.8m x 0.5m obstacle 5 meters away. The environment feature vector example is: "[0.25, 0.06, 0.5, 0.0, 0.8, 0.5, 0.0, 0.0,..., 0.0]" (the first 3 dimensions are slope level 1 mapping value 0.25, unevenness 0.03 mapping value 0.06, obstacle distance 5m mapping value 0.5, and the subsequent dimensions are obstacle size and other potential obstacle information). The feature vector is accompanied by a confidence index (0-1), which reflects the reliability of feature extraction (such as when the occlusion is severe, the confidence is less than 0.5).
[0032] The state fusion processing is performed on the original data set of the body state, and the sensor fusion algorithm is used to calculate the joint angle, joint angular velocity, foot contact state and body inclination angle to generate the robot state information vector.
[0033] This step eliminates the error of a single sensor through multi-source data fusion, accurately obtains the motion state of the robot body, and provides accurate input for low-level control. The specific implementation is as follows: The original body state data needs to be fused with the robot joint encoder data (the encoder is installed on the hip joint, knee joint and ankle joint, the sampling frequency is 100Hz, and the measurement accuracy is ±0.1°), and the extended Kalman filter (EKF) is used as the sensor fusion algorithm to construct the state equation and observation equation: The state equation contains the joint angles of the robot (2 hip joints, 2 knee joints, 2 ankle joints, a total of 6), the joint angular velocity (6), and the body attitude angle (3), a total of 15 state quantities. The state transition matrix is constructed based on the kinematic model of the robot, considering the continuity of joint motion. The observation values of the observation equation are the three-axis acceleration, three-axis angular velocity, and three-axis attitude angle (9) output by the IMU and the 6 joint angles (a total of 15) output by the joint encoder. The observation matrix is a unit matrix, and the measurement noise matrix is set according to the sensor accuracy (IMU attitude angle noise variance 0.0025, encoder joint angle noise variance 0.0001).
[0034] The specific fusion process is as follows: Joint angle calculation: In the EKF prediction stage, the current joint angle prediction value is derived based on the joint angular velocity at the last time, and in the update stage, the prediction value is corrected by combining the measured joint angle of the encoder to obtain the optimal joint angle (unit rad), for example, the hip joint flexion angle prediction value is 1.0 rad, the encoder measured value is 1.02 rad, and the corrected output is 1.015 rad. Joint angular velocity calculation: Based on the fused joint angle sequence, the central difference method is used to derive the joint angular velocity (unit rad / s), the formula is ω=(θ(t+1)-θ(t-1)) / (2Δt), Δt=0.01s (sampling period), for example, the joint angle at t-1 time is 1.0 rad, the joint angle at t+1 time is 1.02 rad, and the angular velocity is =(1.02-1.0) / (2×0.01)=1.0 rad / s. Foot contact state judgment: Combined with the foot pressure sensor data (installed on the foot, sampling frequency 100Hz, threshold 0.5N) and the fused body tilt angle, when the pressure value ≥ threshold and the body tilt angle ≤ 15°, it is determined as "contact" (marked 1), otherwise "non-contact" (marked 0), for example, the left foot pressure is 0.8N, the body tilt angle is 3°, and it is determined as contact. Body tilt angle calculation: In the EKF update stage, the body attitude (based on the forward kinematic model of the robot) derived from the fusion of the IMU attitude angle and the joint angle is corrected to correct the cumulative error of the IMU, and the accurate body roll angle and pitch angle (unit °) are obtained. The yaw angle is directly output by the IMU (the yaw angle changes little when the robot is walking), for example, the IMU output pitch angle is 8.2°, the kinematic derivation is 8.0°, and the fusion output is 8.1°.
[0035] The generated robot state information vector is a 20-dimensional real number vector, including 6 joint angles, 6 joint angular velocities, 2 foot contact states (1 for each left and right foot), 3 body posture angles, 2 body tilt angles (roll, pitch), and 1 motion mode identifier (walking / static). For example, the state information vector of the robot when the right foot is supporting and the left foot is swinging is: "[1.015 rad, 0.32 rad,..., 1.0 rad / s, 0.5 rad / s,..., 1, 0,..., 0.4°, 8.1°, 3.2°,..., 0.4°, 8.1°, 1]" (1 indicates walking mode). All physical quantities in the vector are normalized (mapped to the interval -1 to 1) to ensure the consistency of the input of the low-level control strategy network, while also reflecting the credibility of the state information (such as whether the EKF converges).
[0036] In S202, based on the environment features and robot state information, a gait planning instruction is generated by a high-level planning module, wherein the gait planning instruction at least includes a gait mode or gait parameters; Specifically, the environment feature vector and the robot state information vector can be input into a high-level reinforcement learning strategy network. The network establishes a mapping relationship between terrain features and gait modes through pre-training learning, and generates a gait mode probability distribution. This step is the core decision-making link of high-level planning. Through the pre-trained reinforcement learning network, the association between terrain and gait is established to ensure that the decision conforms to the walking characteristics of the robot. The specific implementation is as follows: The high-level reinforcement learning strategy network adopts a lightweight architecture of "full connection + residual connection" to adapt to the computational power limitation of small robots. The input dimension of the network is the sum of the dimensions of the environment feature vector and the robot state information vector — for example, the environment feature vector is 32-dimensional (including slope level, unevenness, obstacle distribution), and the robot state information vector is 20-dimensional (including joint angles and foot contact states), and the number of input layer neurons is set to 52; the hidden layer is set to 3 fully connected layers, with 64, 32, and 16 neurons respectively, and the activation function uses ReLU to avoid gradient disappearance, while residual connection is added to span the first two layers to improve the efficiency of deep feature transmission; the output layer is set to 4 neurons, corresponding to 4 predefined gait modes (walking on flat ground, uphill, downhill, and obstacle crossing), and the output values are normalized by the Softmax function to obtain the probability distribution of each mode (the sum of the probabilities is 1).
[0037] Network pre-training needs to be done in a simulation environment (such as PyBullet), and a dataset containing 100,000 diverse training samples needs to be constructed: the sample input is the simulated environment features and robot state (such as "slope 5° + unevenness 0.03m 2 + no obstacles" corresponds to a flat ground scene, and "slope 12° + body inclination 3°" corresponds to an uphill scene), and the sample label is the manually labeled optimal gait pattern (such as "flat ground walking pattern" for flat ground scenes). The training process uses the Adam optimizer (learning rate 0.001), the loss function is cross-entropy loss (measures the difference between predicted probability and label), iteratively trains for 2000 rounds, randomly shuffles the sample order each round, and prevents overfitting through the early stopping mechanism (stops if the validation set loss does not decrease for 10 consecutive rounds). After training, the network can accurately learn the mapping relationship, for example, the input "slope 0° + unevenness 0.01m 2 + joint angular velocity 0.8 rad / s" environment and state, the output probability distribution is "flat ground walking pattern 0.75, uphill mode 0.15, downhill mode 0.05, obstacle mode 0.05", clearly reflecting the relationship between terrain and gait.
[0038] Based on the gait pattern probability distribution, a greedy strategy or random sampling is used to select the current optimal gait pattern from the predefined gait pattern library, which includes flat ground walking mode, uphill mode, downhill mode, and obstacle mode, to generate a gait pattern identifier; This step balances "exploitation (exploitation of known optimal)" and "exploration (exploration of potential better)" through decision-making strategies to ensure that the gait pattern adapts to the current terrain. The specific implementation is as follows: The predefined gait pattern library needs to clearly define the core features and applicable scenarios of each mode. The definitions of the 4 modes in the library are as follows: Flat ground walking mode: suitable for terrain with slope 0-5° and no obvious obstacles, core features are "moderate step length (0.3-0.35m), low step height (0.05-0.08m), and body posture close to horizontal (pitch angle -1° to 1°)", walking rhythm is stable, and energy consumption is low; Uphill mode: suitable for terrain with slope 5-15°, core features are "shortened step length (0.2-0.25m), increased step height (0.1-0.12m), and body inclination (pitch angle 3-5°)", which reduces uphill resistance by reducing step length and keeps the center of gravity stable by inclining the body; Downhill mode: suitable for terrain with slope -15° to -5° (negative slope represents downhill), the core features are "further shorten the step length (0.18-0.22m), maintain the step height 0.08-0.1m, slightly lean back the body (pitch angle -3° to -1°)", to prevent the center of gravity from moving forward and causing imbalance when going downhill; Obstacle mode: suitable for terrain with obstacle height 0.1-0.2m, the core features are "increase the step length (0.35-0.4m), significantly increase the step height (0.15-0.2m), prolong the gait cycle (0.9-1.1s)", to reserve sufficient time to cross the obstacle.
[0039] The decision-making strategy adopts the ε-greedy strategy (ε is the exploration probability, set to 0.1): When the randomly generated decimal number > ε (90% probability), select the mode with the largest value in the probability distribution, for example, "flat walking mode 0.75" output in the previous step is the maximum probability, directly select this mode; When the randomly generated decimal number ≤ ε (10% probability), randomly select one of the four modes (not dependent on probability), for exploration of new scenarios (such as occasionally trying flat mode on uphill terrain to verify whether there is a better choice), to avoid the strategy falling into local optimum.
[0040] The generated gait mode identifier uses integer encoding (0-3), for example, "0" represents flat walking mode, "1" represents uphill mode, "2" represents downhill mode, and "3" represents obstacle mode, the identifier needs to be accompanied by decision-making basis (such as "selected flat walking mode, probability 0.75, strategy type: ε-greedy (ε=0.1)"), to facilitate subsequent parameter generator to identify mode type.
[0041] According to the gait mode identifier, call the parameter generator, combine the motion parameters in the robot state information vector, and dynamically adjust the step length, step height, gait cycle, and leg lifting height parameters through the reinforcement learning reward function to generate the gait parameter set; This step converts the macro gait mode into specific executable parameters through the parameter generator and reward function, ensuring that the parameters adapt to the current motion state of the robot, and the specific implementation is as follows: The parameter generator has a built-in "mode-basic parameter" mapping table, which calls the corresponding basic parameters (initial values based on a large number of experimental data) according to the gait mode identifier, for example: When calling flat walking mode (identifier 0), the basic parameters are "step length 0.32m, step height 0.06m, gait cycle 0.8s, leg lifting height 0.07m"; When calling uphill mode (identity 1), the basic parameters are "step length 0.23m, step height 0.11m, gait cycle 0.85s, leg lifting height 0.1m".
[0042] The combined robot motion parameters mainly include "average joint angular velocity (such as hip joint angular velocity 0.9 rad / s), body inclination angle (such as 0.5°), and foot contact stability (contact time ratio 0.95)", which reflect the current motion ability and stability of the robot - for example, if the joint angular velocity is too high (>1.2 rad / s), the step length should be reduced to reduce the joint load; if the body inclination angle is too large (>3°), the step height or body posture angle should be adjusted to restore balance.
[0043] The reinforcement learning reward function is designed in the form of "multi-objective weighted summation", and the weights are dynamically adjusted according to the gait mode. The formula is: Reward=α×EfficiencyReward+β×StabilityReward+γ×SmoothnessReward, where: EfficiencyReward (Efficiency Reward): positively related to step length and negatively related to gait cycle, for example, when the step length is within the range of the basic value ±0.03m, the reward is 10×(step length / basic step length), and it decreases linearly when it exceeds (reward 8 when step length is 0.35m, reward 9 when step length is 0.29m), encouraging to improve walking efficiency under the premise of stability; StabilityReward (Stability Reward): negatively related to the absolute value of body inclination angle and positively related to the contact time ratio of foot, for example, when the inclination angle is 0°, the reward is 15, when it is 3°, the reward is 5, when the contact time ratio is 0.95, the reward is 10, and when it is 0.8, the reward is 2, punishing unstable motion; SmoothnessReward (Smoothness Reward): negatively related to the variance of joint angular velocity fluctuation, when the variance is <0.1 rad 2 / s 2 , the reward is 8, and when it is >0.3, the reward is 0, encouraging smooth joint motion; Weights α, β, γ: α=0.4, β=0.4, γ=0.2 (balance efficiency and stability) in flat mode, β=0.5, α=0.3, γ=0.2 (priority stability) in uphill mode, and α=0.3, β=0.5, γ=0.2 (priority stability crossing) in obstacle mode.
[0044] Dynamic adjustment process example: call the flat walking mode basic parameters (step length 0.32m), combined with the robot state "joint angular velocity 0.8rad / s (smooth), body tilt angle 0.5° (stable)", the reward function calculates EfficiencyReward=10×(0.32 / 0.32)=10, StabilityReward=14, SmoothnessReward=7, total Reward=0.4×10+0.4×14+0.2×7=4+5.6+1.4=11; if the step length is adjusted to 0.31m, EfficiencyReward=9.7, StabilityReward=14.5, total Reward=0.4×9.7+0.4×14.5+0.2×7=3.88+5.8+1.4=11.08 (higher), the final adjusted parameters are "step length 0.31m, step height 0.06m, gait cycle 0.8s, leg lifting height 0.07m", generate gait parameter set, parameters are kept to 3 decimal places to ensure accuracy.
[0045] Integrate gait mode identification and gait parameter set construction instructions, encapsulate as target foot position sequence, expected body posture angle and gait timing parameters, generate standardized gait planning instructions.
[0046] This step converts discrete modes and parameters into structured instructions that can be parsed by low-level modules through a unified instruction protocol, ensuring unambiguous information transmission. The specific implementation is as follows: The instruction protocol adopts a three-segment structure of "header identification + core data + check code", the header identification occupies 2 bytes (1 byte is the protocol version number "V1.0", 1 byte is the gait mode identification "0"), the core data contains three types of key information, and the check code is CRC16 check (to ensure data transmission without error): Target foot position sequence: based on step length and step height in gait parameters, combined with the current foot center position of the robot (extracted from the state information vector, such as the current position of the left foot (0,0,0) m), calculate the target position of the foot center in the next 2 gait cycles, for example, in flat walking mode, the next target position of the left foot is (0.31, 0.15, 0.06) m (x axis advances 0.31m, y axis laterally 0.15m, z axis lifts 0.06m), the next target position of the right foot is (0.31, -0.15, 0.06) m, the sequence is sorted by timestamp (such as "t=0.0s: left foot (0,0,0)→t=0.4s: left foot (0.31,0.15,0.06)→t=0.8s: left foot(0.62,0,0)"); Expected body posture angle: determine the target posture of the body according to the gait pattern, which is "pitch angle 0°, roll angle 0°, yaw angle 0°" in flat mode, "pitch angle 4°, roll angle 0°, yaw angle 0°" in uphill mode, and "pitch angle -3°, roll angle 0°, yaw angle 0°" in downhill mode, the unit of the posture angle is degree, and the precision is retained to 1 decimal place; Gait timing parameters: including gait cycle (0.8s), support phase duration (0.5s, accounting for 62.5% of the cycle), swing phase duration (0.3s, accounting for 37.5% of the cycle), leg lifting delay time (0.1s, starting leg lifting 0.1s after the start of the support phase), timing parameters are accurate to the millisecond level, ensuring time synchronization of low-level control.
[0047] During packaging, all data is stored in big-endian byte order, position coordinates are in meters (retaining 3 decimal places), posture angles are in degrees (retaining 1 decimal place), and timing parameters are in seconds (retaining 3 decimal places). For example, the standardized gait plan instruction example for flat walking mode is: "header identification: V1.0+0; core data: target foot position sequence [(0.31, 0.15, 0.06), (0.31, -0.15, 0.06)], expected body posture angle (0.0°, 0.0°, 0.0°), gait timing parameters (0.800s, 0.500s, 0.300s, 0.100s); check code: 0x1A3B". After the instruction is generated, it is sent to the low-level control module through industrial Ethernet (transmission delay <10ms), and at the same time, it is stored in the local cache (retaining the last 10 instructions), which is convenient for subsequent fault backtracking.
[0048] S203, based on the gait plan instruction and the robot state information, generate joint control instructions through the low-level control module combined with the multi-body dynamics model, wherein the multi-body dynamics model includes inertia matrix, gravity term and Coriolis force term; Specifically, the current motion state can be parsed from the robot state information vector, the real-time angles, angular velocities of each joint, and the body space posture are extracted, and the current joint state matrix is generated; This step is the basis of the state input of the low-level control, and the core motion parameters need to be accurately split from the structured robot state information vector to ensure the accuracy of subsequent dynamics calculation and control instruction generation, and the specific implementation is as follows: The robot state information vector is a 20-dimensional real number vector (normalized, mapped to the interval of -1 to 1), which needs to be restored to the physical quantity unit by inverse normalization first, and then the motion state is analyzed according to the preset dimension definition. The vector dimension definition is clear: the first 6 dimensions are joint angles (unit: rad), corresponding to left hip flexion / extension, right hip flexion / extension, left knee flexion / extension, right knee flexion / extension, left ankle dorsiflexion / plantarflexion, and right ankle dorsiflexion / plantarflexion; the 7th-12th dimensions are joint angular velocities (unit: rad / s, flexion is positive and extension is negative); the 13th-15th dimensions are body space attitude angles (unit: °, pitch angle, roll angle, and yaw angle, forward tilt / right roll / right deviation is positive); and the remaining dimensions are auxiliary information such as foot contact state (not involved in joint state matrix construction).
[0049] The inverse normalization process needs to combine the actual range of each physical quantity: the joint angle range is 0-1.5 rad (hip), 0-1.0 rad (knee), and 0-0.5 rad (ankle), and the normalization mapping formula is "physical value = (normalized value + 1) x (max value - min value) / 2 + min value". For example, the left hip angle normalization value in the vector is 0.0, and the restored physical value is (0 + 1) x (1.5 - 0) / 2 + 0 = 0.75 rad; the right hip angular velocity normalization value is -0.2, and the angular velocity range is -1.0 to 1.0 rad / s, and the restored physical value is (-0.2 + 1) x (1.0 - (-1.0)) / 2 + (-1.0) = 0.8 x 1 - 1.0 = -0.2 rad / s (negative sign indicates extension direction).
[0050] The inverse normalization range of the body space attitude angle is -15° to 15°, for example, the pitch angle normalization value is 0.1, and the restored physical value is (0.1 + 1) x (15 - (-15)) / 2 + (-15) = 1.1 x 15 - 15 = 1.5° (forward tilt 1.5°); the roll angle normalization value is -0.05, and the restored value is (-0.05 + 1) x 30 / 2 - 15 = 0.95 x 15 - 15 = -0.75° (left roll 0.75°).
[0051] The generated current joint state matrix is a column vector of 15 rows and 1 column (6 joint angles + 6 joint angular velocities + 3 body attitude angles), for example, at a certain time, the matrix is: [0.75 rad (left hip), 0.6 rad (right hip), 0.4 rad (left knee), 0.5 rad (right knee), 0.2 rad (left ankle), 0.3 rad (right ankle), 0.5 rad / s (left hip angular velocity), -0.2 rad / s (right hip angular velocity), -0.3 rad / s (left knee angular velocity), -0.4 rad / s (right knee angular velocity), 0.1 rad / s (left ankle angular velocity), -0.1 rad / s (right ankle angular velocity), 1.5° (pitch), -0.75° (roll), 0.3° (yaw)]^T. The matrix needs to mark the physical meaning and unit of each element to ensure the uniformity of parameter dimension in subsequent dynamics calculation.
[0052] Based on the current joint state matrix, the multi-body dynamics parameters are calculated, the Newton-Euler recursive algorithm is used to solve the robot dynamics equation, and the inertia matrix, gravity vector and Coriolis force vector are obtained, and a set of dynamics parameters is generated; This step is the core of low-level control fusion dynamics model, which extracts the inertia, gravity and centrifugal / Coriolis force influence of robot motion through classical dynamics algorithm, and provides physical priori for control strategy, the specific implementation is as follows: Firstly, the basic parameters of robot multi-body dynamics model are determined (based on the design of small biped robot): trunk mass 2.0 kg, left / right thigh mass 0.5 kg, left / right leg mass 0.3 kg, left / right foot mass 0.1 kg; the distance between the mass center of each link and the joint: the distance between the mass center of thigh and hip is 0.3 m, the distance between the mass center of leg and knee is 0.25 m, and the distance between the mass center of foot and ankle is 0.1 m; the acceleration of gravity g=9.8 m / s 2 .
[0053] Newton-Euler recursive algorithm is executed according to the process of "forward recursive kinematics + backward recursive torque": Forward recursion (from trunk to foot): based on the joint angles and angular velocities in the current joint state matrix, the positions, velocities and accelerations of each link are calculated. For example, the position of left thigh link = trunk position + hip joint angle corresponding rotation vector x thigh length, velocity = hip joint angular velocity x thigh length, acceleration = hip joint angular acceleration (calculated by angular velocity difference, such as 0.2 rad / s 2 ) x thigh length + centrifugal acceleration (ω 2 x length); Backward recursion (from feet to torso): Calculate inertia forces and moments of each link, then add gravity and Coriolis forces. Inertia force = link mass x link acceleration, inertia moment = link inertia tensor x angular acceleration + angular velocity x (link inertia tensor x angular velocity); gravity = link mass x g (direction vertically downward); Coriolis force = 2 x link mass x (angular velocity x linear velocity) (only exists in multi-joint coupled motion).
[0054] Based on the results of recursion, solve the robot dynamics equation "M (q) x +C (q, ) x +G (q) = τ" (τ is joint torque), get three types of core parameters: Inertia matrix M (q): 6 x 6 symmetric matrix (corresponding to 6 joints), element M_ij represents the inertia coupling effect of joint j motion on joint i. For example, M (1,1) = the main inertia moment of the left hip joint = left thigh inertia (0.5 x 0.3 2 = 0.045 kg m 2 ) + left leg inertia (0.3 x (0.3 + 0.25) 2 = 0.09075 kg m 2 ) + left foot inertia (0.1 x (0.3 + 0.25 + 0.1) 2 = 0.04225 kg m 2 ) = 0.178 kg m 2 ; M (1,2) = the coupled inertia moment of left hip and right hip = 0.01 kg m 2 (the coupling is weak because the distance between left and right hips is far); Gravity vector G (q): 6-dimensional vector, element G_i represents the static torque required for joint i to overcome gravity. For example, G (1) = left hip gravity moment = left thigh gravity x center of mass horizontal distance + left leg gravity x (thigh length + leg center of mass horizontal distance = 0.5 x 9.8 x 0.3 x sin (0.75 rad) + 0.3 x 9.8 x (0.3 + 0.25) x sin (0.75 rad + 0.4 rad) ≈ 2.4 N m; Coriolis force vector C (q, ): 6-dimensional vector, element C_i represents the dynamic coupling torque generated by joint motion. For example, C (1) = left hip Coriolis torque = M (1,2) x right hip angular velocity + 2 x M (1,3) x left knee angular velocity = 0.01 x (-0.2) + 2 x 0.005 x (-0.3) = -0.002 - 0.003 = -0.005 N m (negative sign indicates that the torque direction is opposite to the motion direction).
[0055] The generated set of dynamic parameters includes a 6x6 inertia matrix, a 6-dimensional gravity vector, a 6-dimensional Coriolis force vector, all with units (inertia: kg·m 2
[0056] The standardized gait planning instructions, the current joint state matrix, and the set of dynamic parameters are input into a low-level reinforcement learning strategy network, which is trained through deep reinforcement learning and combines kinetic energy balance rewards and inertia compensation rewards to generate preliminary joint control quantities. This step realizes the mapping of "instructions - state - dynamics" to control quantities through a low-level reinforcement learning network, and the core is the design of a reward function that incorporates physical priors. The specific implementation is as follows: The low-level reinforcement learning strategy network uses a "convolution + fully connected" hybrid architecture to adapt to multiple input sources: The input layer: The standardized gait planning instructions (such as the target left hip angle of 1.0 rad and the target gait cycle of 0.8 s, totaling 8 dimensions), the current joint state matrix (15 dimensions), and the set of dynamic parameters (6x6+6+6=48 dimensions) are concatenated, with a total input dimension of 61. The hidden layer: 2 layers of convolutional layers (kernel size 3x1, extracting temporal features) + 3 layers of fully connected layers (neuron count 128, 64, 32), with LeakyReLU activation functions (to avoid negative interval gradient disappearance) and BatchNorm layers added to accelerate training convergence. The output layer: 6-dimensional neurons corresponding to the target angular increments (units: rad) and target torque increments (units: N·m) of the 6 joints, with the output values mapped to the range [-0.2rad, 0.2rad] and [-0.5N·m, 0.5N·m] through the Tanh function.
[0057] Network training is completed in a simulation environment (PyBullet), with 100,000 sets of "input - optimal control quantity" data included in the training samples. The reward function is designed as a weighted sum of "basic reward + kinetic energy balance reward + inertia compensation reward" (weights α=0.3, β=0.4, γ=0.3): Basic reward: Negatively related to gait tracking error, error = | target joint angle - current joint angle |, for example, left hip tracking error 0.05 rad, basic reward = 10 - 50x0.05 = 7.5. Kinetic energy balance reward: Encourages balanced kinetic energy distribution between the left and right legs, kinetic energy = 0.5xM_legxv_leg 2 , left leg kinetic energy 1.2J, right leg kinetic energy 1.3J, balance degree = 1 - |1.2-1.3| / (1.2+1.3) = 0.96, kinetic energy balance reward = 10x0.96 = 9.6; Inertia compensation reward: penalize the deviation of control quantity from the demand of dynamics, deviation = |policy output torque - (Mx _des + Cx +G) |, for example, left hip deviation 0.1 N·m, inertia compensation reward = 8 - 20x0.1 = 6.
[0058] Total reward = 0.3x7.5 + 0.4x9.6 + 0.3x6 = 7.89, the network minimizes the reward loss by policy gradient algorithm (PPO), and converges after 3000 iterations of training.
[0059] When generating the initial joint control quantity, the network calculates the output according to the input: for example, the current left hip angle is 0.75 rad, the target angle is 1.0 rad, the network outputs an angle increment of 0.2 rad (the corrected left hip target angle is 0.95 rad), and the torque increment is 0.3 N·m (the corrected left hip target torque is 2.7 N·m); the right hip angle is 0.6 rad, the target angle is 0.8 rad, the output angle increment is 0.15 rad (the target is 0.75 rad), and the torque increment is 0.2 N·m (the target is 2.1 N·m), finally forming a set of initial control quantities for 6 joints (angle increment + torque increment).
[0060] The initial joint control quantity is optimized by dynamics constraint, and the reinforcement learning policy gradient method is used to correct the joint target angle and torque value to output the final joint control instruction.
[0061] This step ensures the feasibility of the control quantity by physical constraints to avoid joint overtravel or torque overload, and the specific implementation is as follows: First, the dynamics constraint boundary of the robot is determined (based on hardware performance and safety requirements): Joint angle constraint: left / right hip flexion ≤ 1.5 rad, ≥ 0 rad; left / right knee flexion ≤ 1.0 rad, ≥ 0 rad; left / right ankle dorsiflexion ≤ 0.5 rad, ≥ 0 rad; Joint torque constraint: left / right hip ≤ 3.0 N·m, ≥ -1.0 N·m (extension is negative); left / right knee ≤ 2.0 N·m, ≥ -1.5 N·m; left / right ankle ≤ 1.0 N·m, ≥ -0.5 N·m; Joint angular velocity constraint: ≤ 1.5 rad / s, ≥ -1.5 rad / s.
[0062] The constraint optimization adopts the "policy gradient correction method", and the core is to add a constraint penalty term in the policy loss function: Identify non-compliant control values: For example, in the initial control values, the target angle of the left knee is 1.05 rad (exceeding the upper limit by 1.0 rad), and the target torque of the left hip is 3.2 N·m (exceeding the upper limit by 3.0 N·m). Calculate the penalty coefficient: The larger the violation, the higher the penalty coefficient. The formula is "penalty coefficient = 1 + 5 × (violation amount / constraint range)". For example, the violation amount of the left knee is 0.05 rad, so the penalty coefficient = 1 + 5 × (0.05 / 1.0) = 1.25; the violation amount of the left hip is 0.2 N·m, so the penalty coefficient = 1 + 5 × (0.2 / 3.0) ≈ 1.33. Correcting control values: Through policy gradient backpropagation, the illegal control values are adjusted to be within the constraints. The target angle of the left knee is corrected to 1.0 rad (upper limit), and the target torque of the left hip is corrected to 3.0 N·m (upper limit). At the same time, the related joints are fine-tuned (such as the right knee angle being corrected from 0.5 rad to 0.45 rad) to ensure overall gait coordination. Verify kinetic consistency: After correction, the kinetic parameters need to be recalculated to ensure "M× +C× +G ≈ τ”, for example, the corrected torque of the left hip is 3.0 N·m, the dynamic requirement is 2.9 N·m, the deviation is 0.1 N·m (≤0.2 N·m within the allowable range), and it is judged to be qualified.
[0063] The final output joint control commands are structured protocols, containing "Joint ID - Target Angle (rad) - Target Torque (N·m) - Execution Time (ms)", for example: "Left hip (ID1): 0.95rad, 3.0N·m, 200ms; Right hip (ID2): 0.75rad, 2.1N·m, 200ms; Left knee (ID3): 1.0rad, 1.8N·m, 200ms; Right knee (ID4): 0.6rad, 1.9N·m, 200ms; Left ankle (ID5): 0.25rad, 0.8N·m, 200ms; Right ankle (ID6): 0.3rad, 0.9N·m, 200ms". The commands must be checked by CRC (e.g., 0x2B7D) to ensure error-free transmission, and constraint optimization descriptions must be marked (e.g., "Left knee angle correction 0.05rad, left hip torque correction 0.2N·m").
[0064] S204, execute the joint control command to control the robot joint movement to achieve adaptive gait generation.
[0065] Specifically, the final joint control commands can be converted into motor drive signals, and the target joint angles and torques can be converted into motor PWM modulation signals through inverse kinematics calculations to generate a motor drive command set; This step is the key bridge connecting control instructions and hardware execution, and the core is to convert abstract joint control requirements into motor recognizable drive signals through inverse kinematics calculation and PWM signal conversion. The specific implementation is as follows: First, the mapping relationship between the robot joint and the motor is determined: the hip joint, knee joint and ankle joint of the small biped robot are each configured with one DC servo motor, a total of 6 motors. Each motor is rigidly connected to the joint through a reduction mechanism (reduction ratio i=1:10, i.e. 10 turns of the motor correspond to 1 turn of the joint), and the motor model is adapted to the joint load (hip joint uses MG996R servo motor, rated torque 10kg·cm, knee joint / ankle joint uses SG90 servo motor, rated torque 2.5kg·cm).
[0066] Inverse kinematics calculation focuses on the conversion of "target joint parameters → motor motion parameters": Target joint angle conversion: given the left hip target angle θ_joint=0.95 rad (about 54.4°) in the final joint control instruction, considering the reduction ratio i=10, the motor needs to rotate the mechanical angle θ_motor=θ_joint×i×(180 / π) (convert radians to degrees) =0.95×10×57.3≈544.35°, since the maximum rotation angle of the motor is 360° per cycle, it needs to be split into 1 cycle (360°) +184.35°, which corresponds to 1.512 cycles of motor rotation; Target joint torque conversion: the left hip target torque T_joint=3.0N·m, the motor output torque T_motor=T_joint / i=3.0 / 10=0.3N·m (about 3.06kg·cm), which needs to match the motor torque characteristics — MG996R motor torque is positively correlated with PWM duty cycle, when the duty cycle is 1.5ms, the torque is 0.5kg·cm, and when the duty cycle is 1.8ms, the torque is 1kg·cm, therefore 0.3N·m (3.06kg·cm) needs to correspond to a duty cycle of 1.8ms (since the motor rated torque is 10kg·cm, 1.8ms can meet the demand).
[0067] PWM modulation signal parameters need to adapt to the standard of servo motor: adopt 50Hz PWM frequency (period T=20ms), duty cycle range 1ms-2ms, 1ms corresponds to 0° of motor (joint 0rad), 2ms corresponds to 180° of motor (joint 0.314rad, because 180°=πrad, π / 10≈0.314rad), duty cycle and motor angle are linearly related, the formula is "duty cycle D=1ms + (θ_motor / 180°)×1ms". For example, the target angle of left hip motor is 184.35° (after disassembly, the angle in a single circle), which is substituted into D=1 + (184.35 / 180)×1≈2.024ms, taking the approximate value 2.02ms, to ensure that the motor rotation angle accurately matches the joint requirements.
[0068] The generated motor driving instruction set is structured as "motor ID-joint type-PWM parameter-execution time limit", for example: "motor 1 (left hip): PWM frequency 50Hz, duty cycle 2.02ms, execution time limit 200ms; motor 2 (right hip): PWM frequency 50Hz, duty cycle 1.85ms, execution time limit 200ms; motor 3 (left knee): PWM frequency 50Hz, duty cycle 1.90ms, execution time limit 200ms; motor 4 (right knee): PWM frequency 50Hz, duty cycle 1.75ms, execution time limit 200ms; motor 5 (left ankle): PWM frequency 50Hz, duty cycle 1.60ms, execution time limit 200ms; motor 6 (right ankle): PWM frequency 50Hz, duty cycle 1.65ms, execution time limit 200ms", the instruction set is accompanied by CRC check code (such as 0x3C8D) to avoid transmission errors.
[0069] The motor driving instruction set is executed by the robot joint servo system, and the PWM signal is transmitted to the servo motor of the hip joint, knee joint and ankle joint to drive each joint to move according to the instruction requirements, and the actual motion trajectory of the joint is generated. This step is executed by the hardware of the servo system, which converts the PWM signal into physical motion of the joint. The core is to ensure the real-time performance of signal transmission and the smoothness of motion trajectory. The specific implementation is as follows: The robot joint servo system is composed of "PWM signal generator-signal transmission bus-servo driver-servo motor-joint transmission mechanism": PWM signal generator: integrated in the robot's main control unit (such as STM32H743 single-chip microcomputer), generate 6 independent PWM signals according to the parameters of the driving instruction set, signal amplitude 3.3V (adapt to the input voltage of the servo driver), rising / falling time <100ns, avoid signal distortion; Signal transmission bus: use RS485 industrial bus (transmission rate 115200bps, transmission distance <10m, adapt to the size of the robot body), each PWM signal is transmitted through differential transmission to reduce electromagnetic interference, for example, the PWM signal of the left hip motor is transmitted from the main control unit to the hip joint driver with a delay <5ms, ensuring that the 6 motors start synchronously; Servo driver: each motor is equipped with a dedicated driver (such as MG996R), which receives the PWM signal and analyzes the duty cycle, outputs the corresponding voltage (left hip driver outputs 12V voltage, matches the motor rated voltage) and current (outputs 0.5A current according to torque demand), drives the motor to rotate; Joint transmission mechanism: uses a harmonic reducer (transmission efficiency 90%, backlash <1 arcminute), accurately transmits the motor rotation to the joint, for example, when the left hip motor rotates at a 2.02ms duty cycle, the reducer converts the motor's 1.512 rotations into the joint's 0.95rad rotation without significant idle error.
[0070] The actual motion trajectory of the joint needs to follow the "uniform acceleration - uniform speed - uniform deceleration" three-stage motion law to avoid impact caused by sudden stop and start: Left hip joint motion example: 0-50ms is the uniform acceleration phase, the angle increases from 0.75rad (initial value) to 0.85rad, and the angular velocity increases linearly from 0.5rad / s to 0.8rad / s; 50-150ms is the uniform speed phase, the angle increases from 0.85rad to 0.90rad, and the angular velocity is stable at 0.8rad / s; 150-200ms is the uniform deceleration phase, the angle increases from 0.90rad to 0.95rad, and the angular velocity decreases from 0.8rad / s to 0.5rad / s; Trajectory recording: record the joint angle every 10ms through the built-in encoder of the motor to form a time-angle sequence, for example, "10ms:0.77rad, 20ms:0.79rad, …, 200ms:0.95rad", which is the actual motion trajectory of the left hip joint, the trajectory smoothness error is <0.01rad (meets the requirements of smooth gait).
[0071] The execution logic of other joints (right hip, knee, ankle) is consistent with the left hip, only the PWM parameters and motion trajectory parameters are different, for example, the target angle of the right knee joint is 0.6 rad, and the motion trajectory is “0-50ms: 0.5 rad→0.55 rad, 50-150ms: 0.55 rad→0.58 rad, 150-200ms: 0.58 rad→0.6 rad”, which ensures the coordinated movement of the 6 joints and forms a coherent gait.
[0072] Real-time acquisition of joint motion feedback data, actual rotation angle of each joint is read through encoder and foot contact force is detected through force sensor, combined with IMU to update body attitude data, to generate motion feedback data set; This step collects the actual state of motion through multiple sensors to provide real feedback for subsequent closed-loop control, the key is to ensure the accuracy and real-time performance of the feedback data, the specific implementation is as follows: Joint actual rotation angle acquisition: each servo motor is equipped with an incremental encoder (resolution 1024 lines, 4096 pulses per revolution), the encoder is rigidly connected with the motor shaft, and the motor outputs pulse signals when it rotates, which are recorded by the counter module (integrated in the main control unit). For example, if the left hip motor rotates 1.512 turns, the encoder outputs 6193 pulses, which corresponds to a motor rotation angle of 544.5°, which is divided by the reduction ratio of 10 to get the actual joint rotation angle of 0.95 rad (consistent with the target value, error <0.01 rad). The rotation angle data is collected every 10ms to form a sequence of joint rotation angles.
[0073] Foot contact force detection: one thin film pressure sensor (model FSR402, range 0-50N, resolution 0.1N, response time <10ms) is installed on the left / right foot, which is evenly distributed on the forefoot and heel (4 per foot). When calculating the contact force, the average value of the 4 sensors is taken. For example, when the left foot contacts the ground, the forefoot sensor detects a force of 12.3N and the heel sensor detects a force of 15.6N, the average value is 13.95N. When the average value is greater than 5N, it is determined that the foot is in contact with the ground, and when it is less than 5N, it is determined that the foot is in the air. The contact force data is collected every 5ms to record the contact state changes.
[0074] Robot body posture data update: The robot torso center is equipped with an inertial measurement unit (IMU, model MPU6050, sampling frequency 100 Hz, acceleration measurement range ±16g, angular velocity measurement range ±2000° / s), which collects three-axis acceleration (ax, ay, az) and three-axis angular velocity (gx, gy, gz). The acceleration and angular velocity data are fused by a complementary filtering algorithm to calculate the robot body posture angle: Pitch angle θ: calculated from ax and az, θ = arctan2(ax, az) x (180 / π), for example, ax = 0.5 m / s 2 , az = 9.8 m / s 2 , θ = arctan2(0.5, 9.8) x 57.3 ≈ 2.9° (forward tilt); Roll angle φ: calculated from ay and az, φ = arctan2(ay, az) x (180 / π), for example, ay = -0.3 m / s 2 , az = 9.8 m / s 2 , φ = arctan2(-0.3, 9.8) x 57.3 ≈ -1.7° (left roll); Yaw angle ψ: calculated by integrating gy, ψ = ψ_prev + gy x Δt (Δt = 0.01 s), for example, gy = 0.1° / s, ψ_prev = 0.3°, ψ = 0.3 + 0.1 x 0.01 ≈ 0.301°, update the posture angle every 10 ms.
[0075] The generated motion feedback dataset is structured as "timestamp - joint angle - contact force - posture angle", with a timestamp accurate to milliseconds (e.g. "20251030_152030.200"), for example, a certain time dataset segment is: "timestamp 20251030_152030.200, left hip angle 0.95 rad, right hip angle 0.75 rad, left knee angle 1.0 rad, right knee angle 0.6 rad, left ankle angle 0.25 rad, right ankle angle 0.3 rad; left foot contact force 13.95 N, right foot contact force 12.8 N; pitch angle 2.9°, roll angle -1.7°, yaw angle 0.301°", the dataset is also labeled with sensor working status (e.g. "encoder normal, force sensor normal, IMU normal"), ensuring data reliability.
[0076] The motion feedback dataset is input into the perception module to update the robot state information vector, forming a reinforcement learning closed-loop control loop, and based on the updated state information, the gait generation process is iteratively executed to realize continuous adaptive gait adjustment.
[0077] This step converts feedback data into input for the next round of optimization through closed-loop control. The core is to achieve continuous iteration of "perception - decision - execution - feedback", ensuring adaptive adjustment of gait. The specific implementation is as follows: Robot state information vector update: After the perception module receives the motion feedback data set, replace the old data according to the vector dimension definition (20 dimensions): Joint angle dimension (first 6 dimensions): Replace the original initial value (such as left hip 0.75 rad) with left hip 0.95 rad, right hip 0.75 rad, left knee 1.0 rad, right knee 0.6 rad, left ankle 0.25 rad, and right ankle 0.3 rad; Joint angular velocity dimension (7th-12th dimension): Calculate the angular velocity based on the rotation angle sequence, left hip angular velocity = (0.95-0.94) / 0.01=1.0 rad / s (0.01s is the sampling interval), replace the original 0.5 rad / s; Body posture angle dimension (13th-15th dimension): Replace the original values with pitch angle 2.9°, roll angle -1.7°, and yaw angle 0.301°; Foot contact state dimension (16th-17th dimension): Replace the original state with left foot contact force 13.95N>5N labeled as 1 (contact) and right foot contact force 12.8N>5N labeled as 1 (contact); The updated state information vector is "[0.95, 0.75, 1.0, 0.6, 0.25, 0.3, 1.0, -0.2, -0.3, -0.4, 0.1, -0.1, 2.9, -1.7, 0.301, 1, 1, 0, 0, 0]^T" (the last 3 dimensions are auxiliary states, which are not changed), the update frequency is consistent with the feedback sampling frequency (100Hz).
[0078] Reinforcement learning closed-loop control loop construction: The updated state information vector is used as new input and flows back into the gait generation process: Feedback to the perception module: Supplement the environment feature vector (such as the terrain slope is still 5°, no change), form a new "environment - state" input; Feedback to the high-level planning module: If the state vector shows that the left foot contact force is stable, the high-level does not need to adjust the gait mode (still walking on flat ground), only fine-tune the step length to 0.32m (original 0.31m); Feedback to the low-level control module: Based on the new state vector and the fine-tuned gait plan instruction, recalculate the dynamics parameters (such as the left hip inertia matrix M (1,1) increases slightly to 0.18 kg·m 2), a new joint control instruction (left hip target angle 0.96 rad, torque 3.1 N·m) is generated; The loop completes one iteration every 200 ms (synchronized with the gait cycle) to realize the cycle of "execution - feedback - optimization".
[0079] Continuous adaptive gait adjustment: when the environment or state changes, the closed-loop automatically triggers adjustment: Terrain change example: if the robot enters a gentle slope (slope 8°), the feedback data shows that the body pitch angle increases to 5°, the foot contact force is 15 N on the left foot and 10 N on the right foot (uneven force), after the state vector is updated, the high-level planning module switches to "uphill mode", the step length is shortened to 0.23 m, and the low-level control module adjusts the left hip torque to 2.8 N·m to balance the uphill resistance; Dynamics change example: if the robot carries a 0.5 kg load, the feedback data shows that the joint angular velocity decreases to 0.8 rad / s (original 1.0 rad / s), after the state vector is updated, the low-level control module adjusts the motor PWM duty cycle to 2.05 ms through the dynamics parameter set (inertia matrix increases to 0.19 kg·m 2 ), increases the motor torque to ensure that the joint moves as planned.
[0080] Through continuous iteration, the robot gait always adapts to the current environment and its own state, for example, in the gait transition from flat ground to uphill, the step length is smoothly reduced from 0.32 m to 0.23 m, and the body pitch angle increases from 2.9° to 4°, without obvious lag or imbalance, realizing the core goal of adaptive gait generation.
[0081] As can be seen, the environment perception data and the robot body state data are collected by the perception module, and the environment features and the robot state information are extracted; based on the environment features and the robot state information, the high-level planning module generates a gait planning instruction; based on the gait planning instruction and the robot state information, the low-level control module generates a joint control instruction in combination with a multi-body dynamics model; the joint control instruction is executed to control the robot joint motion to realize adaptive gait generation, thereby solving the problem of adaptive gait generation in complex environments, improving the motion intelligence, control accuracy and rapid adaptation ability to unknown terrain of the robot.
[0082] Another embodiment of the present application provides a small robot layered reinforcement learning gait adaptive generation system, which is shown in Figure 3 The system can include: The acquisition module 301 is configured to collect environment perception data and robot body state data through a perception module, and extract environment features and robot state information; The first generation module 302 is configured to generate gait planning instructions based on the environment features and the robot state information by a high-level planning module, wherein the gait planning instructions at least include a gait mode or gait parameters. The second generation module 303 is configured to generate joint control instructions based on the gait planning instructions and the robot state information by a low-level control module in combination with a multi-body dynamics model, wherein the multi-body dynamics model includes an inertia matrix, a gravity term and a Coriolis force term. The control module 304 is configured to execute the joint control instructions to control the robot joint movement to realize adaptive gait generation.
[0083] The embodiment of the present application further provides a storage medium, and the storage medium stores a computer program, wherein the computer program is set to execute the steps in any one of the method embodiments.
[0084] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is set to execute the computer program to execute the steps in any one of the method embodiments.
[0085] Specifically, the electronic device can further comprise a transmission device and an input-output device, wherein the transmission device is connected with the processor, and the input-output device is connected with the processor.
[0086] The above embodiment according to the drawings illustrates the structure, features and effects of the present application, and the above description is only the preferred embodiment of the present application, but the present application is not limited to the drawings, any change or modification within the concept of the present application, or the equivalent embodiment within the scope of the specification and drawings, should be within the protection scope of the present application.
Claims
1. A hierarchical reinforcement learning gait adaptive generation method for small robots, characterized in that, The method includes: The sensing module collects environmental perception data and robot body state data, and extracts environmental features and robot state information. Based on the environmental features and robot state information, a gait plan instruction is generated through a high-level planning module, wherein the gait plan instruction includes at least a gait pattern or gait parameters; Based on the gait planning instructions and robot state information, joint control instructions are generated by combining the low-level control module with the multibody dynamics model, wherein the multibody dynamics model includes an inertia matrix, a gravity term, and a Coriolis force term. The joint control commands are executed to control the movement of the robot joints in order to achieve adaptive gait generation.
2. The method according to claim 1, characterized in that, The process of collecting environmental perception data and robot body state data through the perception module, and extracting environmental features and robot state information, includes: Raw environmental data is collected by visual sensors, and image sequences or point cloud data of the terrain ahead are obtained by cameras or lidar to generate raw environmental datasets. Raw data of the robot's body state are collected by an inertial measurement unit (IMU), and the robot's three-axis acceleration, three-axis angular velocity, and three-axis attitude angle data are obtained by the IMU to generate a raw dataset of the body state. Feature extraction processing is performed on the original environmental dataset. Convolutional neural networks are applied to extract the slope level, unevenness index and obstacle distribution features of the terrain elevation map, and generate environmental feature vectors. The original dataset of the robot's state is processed by state fusion. A sensor fusion algorithm is used to calculate joint angles, joint angular velocities, foot contact states, and body tilt angles to generate robot state information vectors.
3. The method according to claim 2, characterized in that, Based on the environmental features and robot state information, a gait planning instruction is generated through a high-level planning module. This gait planning instruction includes at least a gait pattern or gait parameters, including: The environmental feature vector and the robot state information vector are input into a high-level reinforcement learning policy network. This network learns the mapping relationship between terrain features and gait patterns through pre-training and generates a gait pattern probability distribution. The decision-making process is based on the probability distribution of gait patterns. The optimal gait pattern is selected from a predefined gait pattern library using a greedy strategy or random sampling. The patterns include flat walking, uphill walking, downhill walking, and obstacle crossing. Gait pattern identifiers are generated. The parameter generator is invoked based on the gait pattern identifier. Combined with the motion parameters in the robot's state information vector, the step length, step height, gait cycle and leg lift height parameters are dynamically adjusted through the reinforcement learning reward function to generate a gait parameter set. Integrate gait pattern identifiers and gait parameter sets to construct an instruction protocol, encapsulate it as a target foot position sequence, desired body posture angle, and gait timing parameters, and generate standardized gait planning instructions.
4. The method according to claim 3, characterized in that, Based on the gait planning instructions and robot state information, joint control instructions are generated through a low-level control module combined with a multibody dynamics model. The multibody dynamics model includes an inertia matrix, gravity terms, and Coriolis force terms, including: The current motion state is parsed from the robot's state information vector, and the real-time angles, angular velocities, and body spatial attitude of each joint are extracted to generate the current joint state matrix. Based on the current joint state matrix, the multibody dynamic parameters are calculated, and the robot dynamic equations are solved using the Newton-Euler recursive algorithm to obtain the inertia matrix, gravity vector and Coriolis force vector, and generate a set of dynamic parameters. The standardized gait planning instructions, the current joint state matrix, and the set of dynamic parameters are input into a low-level reinforcement learning policy network. This network is trained through deep reinforcement learning and integrates kinetic energy balance rewards and inertia compensation rewards to generate preliminary joint control quantities. The initial joint control parameters are optimized by dynamic constraints. The reinforcement learning strategy gradient method is applied to correct the target joint rotation angle and torque value, and the final joint control command is output.
5. The method according to claim 4, characterized in that, The execution of the joint control commands to control the robot joint movement to achieve adaptive gait generation includes: The final joint control commands are converted into motor drive signals. The target joint angles and torques are converted into motor PWM modulation signals through inverse kinematics calculations to generate a motor drive command set. The robot's joint servo system executes the motor drive instruction set, transmits PWM signals to the servo motors of the hip, knee and ankle joints, drives each joint to move according to the instructions, and generates the actual motion trajectory of the joints. Real-time acquisition of joint motion feedback data; reading the actual rotation angle of each joint through an encoder and detecting the contact force of the foot through a force sensor; combining with the IMU to update the body posture data and generate a motion feedback dataset. The motion feedback dataset is input into the perception module to update the robot's state information vector, forming a reinforcement learning closed-loop control loop. Based on the updated state information, the gait generation process is iteratively executed to achieve continuous adaptive gait adjustment.
6. A hierarchical reinforcement learning gait adaptive generation system for a small robot, characterized in that, The system includes: The acquisition module is used to collect environmental perception data and robot body state data through the perception module, and extract environmental features and robot state information. The first generation module is used to generate gait planning instructions through the high-level planning module based on the environmental features and robot state information, wherein the gait planning instructions include at least a gait pattern or gait parameters. The second generation module is used to generate joint control commands based on the gait planning commands and robot state information, through a low-level control module combined with a multibody dynamics model. The multibody dynamics model includes an inertia matrix, a gravity term, and a Coriolis force term. The control module is used to execute the joint control commands and control the robot joint movement to achieve adaptive gait generation.
7. The system according to claim 6, characterized in that, The acquisition module is specifically used for: Raw environmental data is collected by visual sensors, and image sequences or point cloud data of the terrain ahead are obtained by cameras or lidar to generate raw environmental datasets. Raw data of the robot's body state are collected by an inertial measurement unit (IMU), and the robot's three-axis acceleration, three-axis angular velocity, and three-axis attitude angle data are obtained by the IMU to generate a raw dataset of the body state. Feature extraction processing is performed on the original environmental dataset. Convolutional neural networks are applied to extract the slope level, unevenness index and obstacle distribution features of the terrain elevation map, and generate environmental feature vectors. The original dataset of the robot's state is processed by state fusion. A sensor fusion algorithm is used to calculate joint angles, joint angular velocities, foot contact states, and body tilt angles to generate robot state information vectors.
8. The system according to claim 7, characterized in that, The first generation module is specifically used for: The environmental feature vector and the robot state information vector are input into a high-level reinforcement learning policy network. This network learns the mapping relationship between terrain features and gait patterns through pre-training and generates a gait pattern probability distribution. The decision-making process is based on the probability distribution of gait patterns. The optimal gait pattern is selected from a predefined gait pattern library using a greedy strategy or random sampling. The patterns include flat walking, uphill walking, downhill walking, and obstacle crossing. Gait pattern identifiers are generated. The parameter generator is invoked based on the gait pattern identifier. Combined with the motion parameters in the robot's state information vector, the step length, step height, gait cycle and leg lift height parameters are dynamically adjusted through the reinforcement learning reward function to generate a gait parameter set. Integrate gait pattern identifiers and gait parameter sets to construct an instruction protocol, encapsulate it as a target foot position sequence, desired body posture angle, and gait timing parameters, and generate standardized gait planning instructions.
9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1-5 when it is run.
10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1-5.
Citation Information
Patent Citations
Quadruped robot motion planning method based on hierarchical reinforcement learning
CN112936290A
Robot motion parameter adaptive control method and system based on deep reinforcement learning
CN113478486A
Multi-legged robot gait and trajectory planning method based on hierarchical cross-coordination strategy
CN115877861A
Motion control method for quadruped robot with damaged legs
CN118915802A
Hierarchical AI navigation obstacle avoidance method for humanoid robot based on depth information
CN119356318A
Cited By
Quadruped robot joint zero point progressive self-calibration method based on incremental encoder
CN121625177A
