Robot walking control methods, storage media, electronic devices and software products
By using closed-loop verification of historical data from multimodal perception and foot contact observation information, the problem of inaccurate walking control of the robot in complex terrain was solved, and a more stable walking control effect was achieved.
Patent Information
- Application Number
- CN202610912097.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-06-24
AI Technical Summary
In existing technologies, when robots are walking and controlling in complex terrain, the single-modal prediction of visual sensors is easily affected by noise, occlusion and changes in lighting, resulting in inaccurate terrain perception and thus causing walking instability.
By combining multimodal sensing historical data with foot contact observation information, a closed-loop terrain perception mechanism is constructed. Through terrain state prediction and contact observation correction, a more accurate terrain state representation is generated, thereby determining walking control commands.
This improves the robot's walking stability in complex terrain. Through closed-loop verification of multimodal perception and contact observation, it ensures that the walking control commands adapt to the real terrain contact conditions, thereby enhancing the robot's stability and robustness in complex terrain.
Smart Images

Figure CN122469842B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot control and intelligent sensing technology, specifically to a robot walking control method, storage medium, electronic device and program product. Background Technology
[0002] In the field of autonomous walking control for robots, dealing with complex terrains such as steps, slopes, and irregular surfaces is a core and challenging technical requirement. To achieve stable walking, the control system needs to accurately perceive the terrain features beneath the feet and generate or adjust the gait accordingly, avoiding imbalance and falls caused by sudden changes in terrain or perception errors.
[0003] In related technologies, visual or depth sensors (such as depth cameras and LiDAR) are often used to acquire 3D point clouds of the environment to construct terrain maps, and then gait planning is performed. However, such methods can only provide terrain estimation at the visual prediction level, are susceptible to noise, occlusion, and changes in lighting, and cannot directly reflect the mechanical interaction between the foot and the ground at the moment of contact. This leads to a deviation between the perception results and the actual contact constraints, limiting the accuracy and reliability of control strategies in complex terrain. Summary of the Invention
[0004] In view of this, embodiments of this application provide a robot walking control method, a storage medium, an electronic device, and a program product.
[0005] In a first aspect, one embodiment of this application provides a robot walking control method, comprising: acquiring multimodal perception historical data of the robot; acquiring contact observation information of the robot's feet; generating a terrain state prediction representation of the robot's feet based on the multimodal perception historical data; adjusting the terrain state prediction representation of the robot's feet using the contact observation information of the robot's feet to obtain a corrected terrain state representation; and determining walking control commands for the robot based on the corrected terrain state representation.
[0006] In conjunction with the first aspect, in some implementations of the first aspect, the robot's walking control command is determined based on the modified terrain state representation, including: generating a motion correction amount for adjusting the basic walking motion based on the modified terrain state representation; and superimposing the motion correction amount with the basic walking motion to obtain the robot's walking control command.
[0007] In conjunction with the first aspect, in some implementations of the first aspect, a motion correction amount for adjusting the basic walking motion is generated based on the modified terrain state representation, including: acquiring the state information of the robot body, the state information of the robot body including at least one of joint angle information, joint velocity information and posture information; generating the motion correction amount based on the state information of the robot body and the modified terrain state representation; wherein, the basic walking motion is generated based on the state information of the robot body.
[0008] In conjunction with the first aspect, in some implementations of the first aspect, the modified terrain state representation includes at least one of the following physical quantities: terrain height, used to guide the robot's foot lift-up height and foot landing time; terrain slope, including a first slope along the robot's forward direction and a second slope along the robot's lateral direction, used to adjust the robot's center of gravity position and gait stability; terrain roughness, used to adjust the robot's gait stability margin and stride length; and contact stability index, used to determine the softness of the ground and the risk of slippage.
[0009] In conjunction with the first aspect, in some implementations of the first aspect, a terrain state prediction representation of the robot's foot is generated based on multimodal perception historical data, including: mapping the multimodal perception historical data to a latent space using a terrain state prediction network to obtain a latent representation of the terrain state; decoding the latent representation of the terrain state using a decoder to obtain a terrain state prediction representation of the robot's foot; wherein the loss function for training the terrain state prediction network is composed of a weighted combination of at least two of the following: terrain height prediction error, terrain slope prediction error, terrain roughness prediction error, and contact stability index prediction error; wherein the multimodal perception historical data includes at least two of the following: depth image historical sequence, foot contact force historical sequence, and joint angle historical sequence.
[0010] In conjunction with the first aspect, in some implementations of the first aspect, the terrain state prediction representation of the robot's foot is adjusted using contact observation information of the robot's foot to obtain a corrected terrain state representation, including: acquiring the foot contact state in the contact observation information; calculating the terrain state correction amount based on the difference between the foot contact state and the terrain state prediction representation of the robot's foot; and updating the terrain state prediction representation of the robot's foot based on the terrain state correction amount to obtain the corrected terrain state representation.
[0011] In conjunction with the first aspect, in some implementations of the first aspect, obtaining contact observation information of the robot's foot includes: obtaining the contact force of the robot's foot sole; determining the contact observation information of the robot's foot based on a comparison between the contact force of the robot's foot sole and a preset contact force threshold; wherein the contact observation information includes the foot contact state and the magnitude of the contact force.
[0012] Secondly, one embodiment of this application provides a robot walking control device, comprising: a first acquisition module for acquiring multimodal perception historical data of the robot; a second acquisition module for acquiring contact observation information of the robot's feet; a generation module for generating a terrain state prediction representation of the robot's feet based on the multimodal perception historical data; a correction module for adjusting the terrain state prediction representation of the robot's feet using the contact observation information of the robot's feet to obtain a corrected terrain state representation; and a determination module for determining walking control commands for the robot based on the corrected terrain state representation.
[0013] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the method mentioned in the first aspect.
[0014] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the method mentioned in the first aspect.
[0015] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the method mentioned in the first aspect.
[0016] The above scheme uses multimodal perception historical data to predict terrain and introduces foot contact observation information to correct the prediction results, thus constructing a closed-loop terrain perception mechanism from prediction to verification. This overcomes the problem of inaccurate walking control caused by relying solely on visual prediction or unverified terrain information, making the final walking commands more adaptable to real terrain contact conditions and effectively improving the walking stability of the robot in complex terrain. Attached Figure Description
[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0018] Figure 1 The diagram shown is a schematic representation of the architecture of a robot walking control system provided in an embodiment of this application.
[0019] Figure 2 The diagram shown is a flowchart of a robot walking control method provided in an embodiment of this application.
[0020] Figure 3 The diagram shown is a structural schematic of a robot walking control device provided in an embodiment of this application.
[0021] Figure 4 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] In the field of robot walking control, to achieve stable walking in complex terrain, the common approach is to use visual sensors, such as depth cameras, to collect environmental point clouds or depth maps, construct terrain height maps from these, and then input this terrain information into a walking planner or control strategy to generate corresponding walking actions. The basic working principle is to use visually perceived terrain geometry as prior knowledge to guide the robot's foot placement and gait parameter settings. This approach is widely used because it provides the robot with a non-contact, somewhat forward-looking terrain perception capability, allowing it to make predictions before approaching the terrain.
[0024] However, the above-mentioned solutions do not perform ideally when applied to scenarios with blurred terrain edges, varied ground materials, or visual blind spots. The main problem lies in the fact that these solutions, in order to optimize the predictability of robot perception, primarily rely on non-contact visual sensors. However, the inherent physical limitations of visual sensors, such as sensitivity to light, occlusion, and inability to perceive contact mechanics, inevitably lead to inaccurate terrain perception information, resulting in walking instability. Specifically, when walking on irregular steps or ground containing loose stones, local noise or occlusion in the depth map can cause centimeter-level errors in the constructed terrain height. This error causes the actual terrain to differ from the predicted terrain when the robot lands, potentially leading to unexpected and severe impacts between the robot's feet and the ground, or excessive tilting due to deviations in the center of gravity calculation.
[0025] Analysis revealed that the root cause of the aforementioned contradiction lies in the fact that the acquisition of terrain conditions in related technologies remains at the single-modal prediction level. This prediction relies solely on historical visual information, forming an open-loop perception process. The generated terrain information lacks verification at the moment of contact with the real physical world. When there is a deviation between the predicted and actual values, this deviation is directly transmitted to the downstream control strategy, which lacks the ability to identify and correct such unverified erroneous terrain information online. Therefore, predictions relying solely on vision cannot discern whether the ground will collapse or slide under stress, and there is an information gap between the static geometric representation it provides and the dynamic mechanical feedback actually felt by the robot's feet.
[0026] In view of this, this application proposes a robot walking control method. This method uses the contact observation information of the feet as a verification signal for the terrain prediction representation, constructs a closed-loop perception mechanism from prediction to correction, and improves the generation of control commands. Thus, without significantly sacrificing the foresight provided by visual perception, it effectively improves the realism of terrain perception and avoids walking instability caused by perception bias. This solves the problem of inaccurate walking control caused by the single terrain perception mode and lack of verification in related technologies, and realizes the stability of robot walking under complex terrain contact conditions.
[0027] Figure 1 The diagram shown is an architectural schematic of a robot walking control system provided in one embodiment of this application. Figure 1 As shown, the system includes a sensor assembly 110 mounted on the robot, which includes a depth camera 101, a plantar force sensor 102, and a joint encoder 103. The depth camera 101 is configured to acquire a sequence of environmental depth images along the robot's direction of travel. The plantar force sensor 102 is positioned on the robot's foot to acquire contact force signals along the normal direction of the robot's foot. The joint encoder 103 is configured to detect the rotation angles of the robot's various driven joints.
[0028] The depth camera 101, the plantar force sensor 102, and the joint encoder 103 are each connected to a computing processing unit 120 via a data interface. The computing processing unit 120 receives depth image sequences from the depth camera 101, contact force signals from the plantar force sensor 102, and joint angle readings from the joint encoder 103 as multimodal sensing historical data.
[0029] The computational processing unit 120 includes a terrain state prediction subunit 121, which internally maintains a trained neural network model for processing received multimodal perception historical data. This neural network model receives historical data organized by time windows and outputs a set of predicted physical quantities describing the terrain features of the area beneath the robot's feet.
[0030] The computing processing unit 120 further includes a contact state determination subunit 122, which receives the contact force signal from the plantar force sensor 102. For example, the contact state determination subunit 122 internally has a predefined force threshold, which compares the contact force signals in real time to output a discrete state indication representing whether the foot is in contact with the ground, as well as a quantized value representing the magnitude of the contact force.
[0031] The outputs of both the terrain state prediction subunit 121 and the contact state determination subunit 122 are connected to the terrain correction subunit 123. The terrain correction subunit 123 performs calculations on the predicted physical quantities provided by the terrain state prediction subunit 121 and the contact observation information provided by the contact state determination subunit 122, and outputs a set of corrected terrain physical quantities.
[0032] The system also includes a basic motion generation subunit 130. This basic motion generation subunit 130 is used to generate basic joint target positions or torque commands that maintain the robot's basic walking gait, a process that does not depend on the modified terrain physics.
[0033] The computational processing unit 120 also includes a motion correction subunit 124. The first input port of the motion correction subunit 124 receives the corrected terrain physical quantities output by the terrain correction subunit 123, and its second input port receives state information representing the robot body. For example, the motion correction subunit 124 maintains a trained policy network and outputs a set of joint motion correction quantities.
[0034] The computational processing unit 120 also includes an instruction overlay node 125. The input terminals of the instruction overlay node 125 are respectively connected to the output terminals of the basic motion generation subunit 130 and the motion correction subunit 124, and its output terminals are connected to the underlying drive controllers of each joint of the robot. The underlying drive controllers calculate the drive current of each joint motor based on the overlay instructions to drive the robot limbs to perform walking movements.
[0035] It should be noted that the aforementioned computing and processing unit 120 can be implemented as an embedded processing board mounted on the robot body, or some of its functions can be distributed on remote computing nodes, with data exchanged via wireless links.
[0036] Figure 2 The diagram shown is a flowchart illustrating a robot walking control method according to an embodiment of this application. Exemplarily, this control method can be executed by a computing processing unit mounted on the robot body or a remote computing node. Figure 2 As shown, the method includes the following steps.
[0037] Step S210: Obtain the robot's multimodal perception historical data.
[0038] Specifically, multimodal sensing historical data refers to a sequence of observation data spanning a certain time span, collected from various types of sensors. See also Figure 1 The system architecture shown depicts a robot equipped with a sensor assembly, including a depth camera, a foot force sensor, and a joint encoder. Data acquired by these sensors is cached within a time window to form multimodal perception historical data.
[0039] As one implementation method, the multimodal perception historical data includes at least two of the following: depth image history sequences, foot contact force history sequences, and joint angle history sequences. Specifically, the depth image history sequences originate from environmental depth maps continuously acquired at a set frequency by a binocular or depth camera mounted on the robot's head. Each depth image frame records the distance value from the object surface in the scene to the camera in pixel coordinates. After denoising, filtering, and coordinate transformation, a three-dimensional point cloud or height map sequence containing the terrain contours of the area in front of the robot and under its feet is formed, which can capture the spatiotemporal evolution of geometric shapes such as ground protrusions, depressions, slopes, and steps. The foot contact force history sequences originate from multi-point array force sensors arranged on the left and right soles of the robot. They sample the changes in the normal and tangential forces of the feet during interaction with the ground at high frequency, especially recording the dynamic force values during stages such as ground impact, push-off rebound, and center of gravity transfer. From this, implicit mechanical characteristics such as ground hardness, friction coefficient, and local tilt can be extracted. The joint angle history sequence comes from the incremental or absolute encoders built into each drive joint (such as the hip joint, knee joint, and ankle joint), which records the positional changes of each joint of the robot over time. Through forward kinematics calculation, it can be used to infer the foot trajectory, body posture, and gait phase, thereby reflecting the impact of terrain changes on the robot's motion state.
[0040] The aforementioned data may exhibit different sampling rates and timestamp deviations at the acquisition end. Therefore, it is necessary to uniformly map them onto a common time axis with a set control period (such as 100Hz or 200Hz) using linear interpolation, spline interpolation, or timestamp-based alignment algorithms to form a synchronized multimodal observation tensor. This tensor is expanded along the time dimension to constitute a multimodal observation set that evolves over time.
[0041] It is understandable that in this application, the use of multimodal perception historical data instead of snapshot data at a single moment can provide rich temporal contextual information for subsequent terrain state prediction. This allows terrain inference to not only rely on the spatial geometric features of the current frame, but also to make full use of the ground support stiffness abrupt change, foot slippage trend, and dynamic response mode of body joint movement to terrain implied in the force interaction process. This improves the robustness and adaptability of walking control on unstructured, dynamically changing terrain (such as sand, snow, and scree slopes).
[0042] Step S220: Obtain contact observation information of the robot's feet.
[0043] In some embodiments, the process of acquiring contact observation information of the robot's foot further includes: acquiring tactile sensor signals from the bottom of the robot's foot, determining the contact area and contact intensity between the foot and the ground based on the spatial distribution and amplitude variation of the tactile sensor signals, thereby generating contact observation information.
[0044] Specifically, the system uses an array of tactile sensors (such as capacitive or resistive pressure sensors) deployed on the sole of the foot to acquire contact force data of each sensing unit in real time. Then, based on the number and location of the activated units and the continuity of the output values, it identifies whether the foot is in actual contact with the ground and calculates the distribution characteristics of the contact force.
[0045] Optionally, contact observation information may include foot contact state and contact force distribution characteristics. Foot contact state is a Boolean quantity derived from a multi-sensor voting mechanism or clustering algorithm, used to characterize whether the foot is in full contact with the ground at the current moment. Contact force distribution characteristics are a set of continuous physical quantities reflecting the contact pressure center of gravity, contact area, and point of maximum pressure.
[0046] Step S203: Based on multimodal perception historical data, generate a predicted representation of the terrain state of the robot's foot.
[0047] For example, such as Figure 1 The computational processing unit includes a terrain state prediction subunit. This subunit, based on multimodal sensing historical data, employs a hybrid estimation method combining a fused physical model and extended Kalman filtering to output a set of predicted physical quantities describing the terrain features of the area beneath the robot's feet. Specifically, firstly, a dynamic equation for the terrain state is established based on the robot's kinematic model and terrain geometric constraints. Then, multimodal sensing data (such as foot force, joint angles, and inertial measurement unit output) are used as observation inputs, and extended Kalman filtering is used to recursively estimate and predict state quantities such as terrain height, slope, and roughness in one step. Furthermore, to improve the robustness of the prediction, an adaptive noise covariance adjustment mechanism can be introduced during the filtering update process to adjust the observation noise matrix in real time according to the current foot contact stability. Using the above method, time-aligned terrain state prediction values can be directly output.
[0048] Step S204: Adjust the predicted terrain state representation of the robot's foot using the contact observation information of the robot's foot to obtain the corrected terrain state representation.
[0049] It is understandable that the predicted representation obtained in step S203 is based on inference from historical perception sequences and may contain biases due to factors such as visual occlusion, changes in illumination, or model generalization errors. Therefore, this step introduces a self-calibration mechanism to optimize the prediction results online.
[0050] In some embodiments, the process of adjusting the predicted representation using contact observation information further includes: the terrain correction subunit receiving the foot contact force vector (including the magnitude, direction, and pressure center position) and the torque information measured by the six-dimensional force sensor on the foot from the contact observation information; based on the robot kinematic model and the current joint angle, converting the contact force vector into the ground reaction force in the global coordinate system; according to the geometric relationship between the ground reaction force and the theoretical contact point of the robot's foot, solving the current ground height and normal dip angle corresponding to the foot contact area as the observed terrain state sample; inputting the observed sample and the terrain state prediction representation output in step S203 into the extended Kalman filter, and through prediction-update iteration, using the observed value calculated by the ground reaction force as the measurement and the terrain state prediction representation as the state prior, fusing to obtain the corrected terrain state representation.
[0051] For example, when the foot contact force vector shows that the foot has actually landed and the center of pressure is located at the front of the foot, but the predicted terrain height deviates significantly from the theoretical height of the foot at the moment of contact, the extended Kalman filter will automatically adjust the state estimate based on the ratio of the observation noise covariance to the process noise covariance, so that the corrected terrain representation quickly converges to the true ground position indicated by the mechanical measurement at the contact point.
[0052] This correction mechanism, which combines mechanical observation with Kalman filtering, integrates visual and proprioceptive predictions with continuous measurement of foot contact force. This enables precise online calibration of terrain conditions, providing reliable terrain information for walking control.
[0053] Step S205: Determine the robot's walking control commands based on the corrected terrain state representation.
[0054] Understandably, the corrected terrain state representation integrates the results of multimodal prediction and contact verification, and can accurately reflect the geometric and mechanical properties of the current terrain beneath the feet. The computational processing unit uses this representation to determine the joint control commands used to drive the robot's movement.
[0055] For example, such as Figure 1The corrected terrain state representation is input into the action correction subunit as a basis for guiding action adjustments, so that the final generated walking control commands can make targeted adjustments to the robot's gait according to the actual terrain contact conditions. For example, adjusting the foot lift height (e.g., from 0.1m to 0.15m to cross a protrusion), adjusting the foot landing timing (delaying or advancing by 50ms), changing the center of gravity distribution (e.g., shifting it to the supporting leg side by 0.02m), or increasing the stability margin (e.g., reducing the stride length from 0.4m to 0.3m), so that the robot can maintain walking stability in complex terrain.
[0056] The above scheme uses multimodal perception historical data to predict terrain and introduces foot contact observation information to correct the prediction results, thus constructing a closed-loop terrain perception mechanism from prediction to verification. This overcomes the problem of inaccurate walking control caused by relying solely on visual prediction or unverified terrain information, making the final walking commands more adaptable to real terrain contact conditions and effectively improving the walking stability of the robot in complex terrain.
[0057] To further optimize the modularity and scalability of the walking control command generation method in the above embodiments, this application also provides a solution. Specifically, in one embodiment, determining the robot's walking control command based on the corrected terrain state representation further includes: generating a motion correction amount for adjusting the basic walking motion based on the corrected terrain state representation; and superimposing the motion correction amount with the basic walking motion to obtain the robot's walking control command.
[0058] In this scheme, the generation of walking control commands is decomposed into two relatively independent components. The basic walking motion is a sequence of motion commands used to maintain the robot's basic gait on flat or simple terrain, and these motion commands do not depend on the modified terrain state representation. For example, as... Figure 1 As shown, the basic motion generation subunit receives state information from the robot body, such as joint angles and angular velocities from the joint encoder, and generates basic joint target position or torque commands accordingly. More specifically, the basic motion generation subunit can run a reinforcement learning-based policy network, which takes joint angles, angular velocities, and the previous action as input, and outputs basic joint position commands (12-dimensional, one position target per joint).
[0059] The motion correction parameter is a motion adjustment term specifically generated for terrain features, calculated by the motion correction subunit based on the corrected terrain state representation. The final motion executed by the robot is obtained by superimposing both parameters, i.e., the output of the execution command superposition node. ,in Based on basic walking movements (such as 12-dimensional joint position increments). This refers to motion correction amounts (within the same dimension). In some implementations, the superimposed values need to be protected by joint constraints (e.g., the hip joint). knee joint Ankle This design ensures safety. It allows terrain adaptation adjustments to be developed and trained independently of basic walking functionality. With the basic walking strategy remaining stable, only the residual portion needs to be adjusted (…). Adjustments can be made to adapt to different terrains, reducing the coupling and complexity of the overall control strategy.
[0060] The above scheme decomposes the generation of control commands into two parts: basic walking actions and terrain-driven action corrections, and then superimposes them. This provides a specialized action adjustment mechanism for dealing with complex terrain, allowing terrain adaptability to be developed and trained independently of basic walking ability. This reduces the complexity of the overall control strategy while maintaining the stability of basic walking function.
[0061] To further address the issue of how to comprehensively coordinate terrain information and the robot's real-time motion state when generating motion correction values, this application also provides a solution. Specifically, in one embodiment, the process of generating motion correction values is further defined as follows: acquiring the robot's state information, which includes at least one of joint angle information, joint velocity information, and posture information; and generating motion correction values based on the robot's state information and the corrected terrain state representation.
[0062] For example, such as Figure 1 The motion correction subunit has two input ports. The first input port receives the corrected terrain state representation (4D) output by the terrain correction subunit. The second input port receives the state information representing the robot body, which originates at least from the joint encoder (30D: 12 joint angles + 12 angular velocities + 6 virtual spring damping states) and the inertial measurement unit (6-axis: three-axis acceleration and three-axis angular velocity, providing torso posture quaternions, angular velocity, and acceleration) mounted on the robot's torso. Then, the policy network maintained within the motion correction subunit simultaneously processes the terrain information and the body state information, outputting the joint motion correction amount. The network input has 34 dimensions, and the output is the action correction value after passing through a fully connected layer.
[0063] The basic motion generation subunit also receives the body's state information (30-dimensional) such as joint angles and joint angular velocities, and outputs the basic motion. The action outputs of both are merged at the instruction overlay node to jointly constitute the final control instruction.
[0064] The aforementioned scheme stipulates that the motion correction amount must be based on both terrain information and the robot's real-time state information to ensure that the corrected motion matches the robot's current motion posture and dynamic state. For example, when the robot is in a single-leg support phase (determined by contact state), the strategy requirements for center of gravity adjustment due to terrain slope differ from those in a two-leg support phase. For instance, in single-leg support, the terrain slope requires a larger amount of trunk tilt compensation (e.g., tilt angle = slope angle × 0.8), while in two-leg support, the tilt angle will be smaller (e.g., slope angle × 0.3). When the robot's joint velocity is high (e.g., knee joint angular velocity > 5 rad / s), the margin for handling the same terrain roughness will be smaller, requiring a reduction in the correction amount to avoid oscillation.
[0065] As can be seen, this solution avoids command conflicts that may arise from modifications based solely on terrain, which could contradict the robot's current state. It achieves coordination between terrain adaptation and the robot's own kinematics, thereby improving the safety, coordination, and robustness of motion output.
[0066] In the process of terrain perception and walking control, how to comprehensively and structurally describe key terrain features so as to effectively support the generation of control commands is a technical problem that needs to be solved. In the foregoing embodiments of this application, the modified terrain state representation may include at least one physical quantity selected from terrain height, terrain slope, terrain roughness, and contact stability index.
[0067] Specifically, terrain height represents the ground height of the area in contact with the robot's feet, i.e., the vertical position of the ground described in the robot's foot coordinate system. This parameter can be used to guide the robot's foot lift height and landing timing. When the predicted or corrected terrain height is higher than expected (e.g., predicted value is 0.08m, expected value is 0.05m), the control strategy can increase the foot lift amplitude in advance to avoid tripping, for example, adjusting the foot lift height from 0.10m to 0.15m; when the terrain height is lower than expected (e.g., ... When the landing speed is 0.03m, you can prepare a ground contact buffer in advance, such as reducing the landing speed.
[0068] Terrain slope represents the degree of inclination of a local terrain feature, and can be defined as the first slope (pitch angle) along the robot's forward direction and laterally. ) and second slope (roll angle, roll angle), for example, the range is This is used to adjust the robot's center of gravity and gait stability. For example, it guides the body to lean forward when going uphill (preset offset of torso pitch angle). Adjust the stride width on the lateral slope (lateral offset of the supporting leg). ).
[0069] Topographic roughness indicates the degree of irregularity of a terrain surface, and can be measured by local areas (such as the area covered by a foot). The height variance of the rectangular region is quantified. This parameter can be used to adjust the robot's gait stability margin and stride length. For example, in areas with high roughness (such as r>0.5), a robust gait with small strides (such as a 20% reduction in stride length) and a low center of gravity (such as a 5% reduction in hip height) can be adopted.
[0070] The contact stability index represents the stability of the robot's feet in contact with the ground, and can be calculated based on the changes in the contact force of the foot over the most recent 0.1 seconds. This index can be used to determine the softness of the ground and the risk of slippage. For example, when the ground is soft and the contact force fluctuates greatly (e.g., greater than 100 N²), the control strategy can adjust the gait accordingly to reduce the risk of slipping or sinking, such as by increasing the contact area of the foot or reducing the thrust.
[0071] The above scheme defines a structured representation system consisting of multiple physical quantities such as terrain height, slope, roughness, and contact stability. This system quantifies the key terrain geometric and mechanical properties that affect walking control, enabling downstream control strategies to make precise gait adjustments based directly on these objective physical quantities. This improves the precision and adaptability of control when facing unstructured terrain.
[0072] It should be noted that the above four physical quantities can be selectively used depending on the specific application scenario. This solution does not require all four quantities to be included simultaneously, but the included combination should be able to support the walking control scenario in which it is applied. For example, when walking on a flat indoor surface, only height and contact stability can be used, while when walking off-road outdoors, slope, roughness, and height are required.
[0073] To further optimize the accuracy and robustness of the terrain state prediction representation generation, this application also provides a preferred implementation of step S203. In this preferred embodiment, a terrain state prediction network is used to map multimodal perception historical data to a latent space to obtain a latent representation of the terrain state; then, a decoder is used to decode the latent representation of the terrain state to obtain a predicted representation of the terrain state of the robot's foot.
[0074] As mentioned earlier, the multimodal perception historical data includes at least two categories: depth image historical sequences, such as a size sequence of 10 frames × 640 × 480, which is downsampled to 64 × 48 to reduce computational load; foot contact force historical sequences, such as 100 time steps × 3 axes, totaling 300 dimensions; and joint angle historical sequences, such as 100 time steps × 12 joints, totaling 1200 dimensions.
[0075] Specifically, terrain state prediction networks (such as variational autoencoders) compress high-dimensional multimodal historical data into a low-dimensional latent space through an encoder, using the latent variable z_terrain (dimension 16) as the latent representation of terrain state, forcing the network to learn the most valuable essential features for terrain state inference from the input data. Optionally, the encoder structure is as follows: multimodal perceived historical data are processed through their respective sub-networks, such as depth image sequences being processed through 3 layers of 3D convolution, outputting 256-dimensional features; force sequences being processed through 2 layers of LSTM (128 hidden units), outputting 128-dimensional features; joint sequences being processed through 2 layers of LSTM (128 hidden units), outputting 128-dimensional features, then concatenated into 512-dimensional features, and then processed through 2 fully connected layers.
[0076] The decoder starts with the latent variable z, goes through 3 fully connected layers (e.g., 128, 256, 4), and outputs the predicted values of four terrain physical quantities, as well as a reconstructed depth image sequence.
[0077] During training, the true values of multiple terrain physical quantities are used as supervision signals, and the total loss is formed by combining the prediction errors of each quantity according to their weights. Specifically, the loss function for training the terrain state prediction network is composed of a weighted combination of at least two of the following: terrain height prediction error, terrain slope prediction error, terrain roughness prediction error, and contact stability index prediction error.
[0078] For example, the loss function expression can be: .
[0079] in, , , , Specifically, the actual terrain state parameters obtained from the simulation environment, Represents the actual terrain elevation. Represents the actual terrain slope. Represents the actual terrain roughness. This represents the true contact stability index. Indicates the predicted terrain elevation. Indicates the predicted terrain slope. This represents the predicted terrain roughness. This represents the predicted contact stability index. KL_loss is the KL divergence between the latent distribution and the standard normal distribution in the variational autoencoder. The optimal combination of weights can be determined through grid search; for example, in simulation experiments, the optimal weights are... =1.0, =0.7, =0.4, =0.2.
[0080] The advantage of the encoder-decoder structure described above lies in its ability to extract deep, complementary feature representations from heterogeneous multimodal data. Compared to direct end-to-end regression mapping, it is more robust to noise and locally missing data. Furthermore, the design of the multi-task weighted loss function allows the network to balance the learning progress of various physical quantities during training, capturing the intrinsic relationships between different terrain features. For example, there is a derivative relationship between height and slope, and roughness is related to contact stability, thus outputting a more physically consistent comprehensive terrain state prediction.
[0081] It is understood that those skilled in the art can adjust the weighting coefficients according to the actual terrain distribution and task emphasis. , , , The value of is selected to optimize prediction performance in specific application scenarios.
[0082] To specifically implement online closed-loop correction of terrain state based on contact feedback, this application also provides a preferred embodiment of step S204. In this preferred embodiment, the process of adjusting the predicted representation using contact observation information includes: acquiring the foot contact state in the contact observation information; calculating the terrain state correction amount based on the difference between the foot contact state and the predicted terrain state representation of the robot's foot; and updating the predicted terrain state representation of the robot's foot based on the terrain state correction amount to obtain the corrected terrain state representation.
[0083] For example, such as Figure 1 The terrain correction subunit receives terrain state prediction representations from the terrain state prediction subunit and contact observation information from the contact state determination subunit. Then, it compares the degree of inconsistency between the actual contact information implied by the foot contact state and the terrain state prediction representation. For example, when the foot contact state indicates that contact has occurred, but the predicted terrain height indicates that the sole of the foot has not yet reached the ground (e.g., a predicted height of 0.02m), while the actual foot height calculated according to forward kinematics is... 0.005m indicates that the actual ground level corresponds to a lower absolute height, at which point a height difference term arises between the two. When the predicted terrain slope does not match the direction of the contact force vector, for example, if the predicted slope is 0° but the horizontal component of the contact force is 0.2, corresponding to a slope of approximately 11.3, a slope discrepancy term is generated. .
[0084] Furthermore, based on these differences, a terrain state correction is calculated. The calculation method for this correction is flexible and varied. The first method is direct proportional calculation; for example, the height difference is directly added as a height correction, and the slope difference is added as a slope correction, proportionally added to the predicted terrain slope to correct the perceived terrain tilt. The second method involves introducing filtering algorithms to improve smoothness; for example, a smoothing coefficient can be used for terrain slope. Make corrections, the corrected slope Here It is based on the predicted slope and the slope difference term. The actual observed slope is obtained by reverse calculation. The third method uses a learning network to dynamically infer the correction amount. For example, this network is a two-layer fully connected neural network. The input layer of this two-layer fully connected network simultaneously receives height difference... Slope differences The network incorporates various possible discrepancies, including those from other sources, along with the current predicted terrain state representation. Through internal computation, it fuses all these discrepancies and directly outputs a comprehensive, multi-dimensional terrain state correction.
[0085] After obtaining this comprehensive correction, the original terrain state prediction representation is updated. This update also corrects all relevant dimensions of the terrain state prediction representation, such as simultaneously adjusting the height and slope angle of the area in the internal map, so that the corrected terrain state representation can fully reflect the height and slope of the real ground.
[0086] Understandably, this process is not triggered once, but rather executed cyclically. That is, with each step the robot takes, new perceived information immediately generates new discrepancies, and the system repeatedly performs correction calculations and updates to the terrain state prediction representation, continuously correcting the terrain state online.
[0087] The above scheme uses real physical contact as a monitoring signal to perform online real-time calibration of the terrain state prediction representation that may have visual noise or inference errors. This ensures that the accuracy of the core terrain representation can quickly converge to the true value, thus providing a reliable decision basis for subsequent control.
[0088] To reliably acquire contact observation information of the robot's foot, this application also provides an implementation method based on force threshold judgment. Specifically, in this solution, the process of acquiring contact observation information of the robot's foot includes: acquiring the contact force of the robot's foot sole; determining the contact observation information of the robot's foot based on a comparison between the contact force of the robot's foot sole and a preset contact force threshold; wherein, the contact observation information includes the foot contact state and the magnitude of the contact force.
[0089] For example, such as Figure 1 The foot force sensors are arranged on the robot's feet (one six-dimensional force sensor on each foot, or four single-axis force sensors arranged at the four corners) to sense the contact force along the foot normal in real time. The contact state determination subunit receives the contact force signal and compares it with an internally predefined force threshold. Comparison is performed when contact force is detected. When the foot is in contact with the ground, the system determines that the foot is in contact and outputs a contact status as "contact"; otherwise, it outputs "no contact". Simultaneously, the magnitude of the contact force is quantified into a numerical value, which, along with the contact status, constitutes contact observation information.
[0090] To enhance anti-interference capabilities, a hysteresis comparison mechanism can also be introduced, for example, when When it is determined to be contact, It is determined that there is no contact when the time is right, so as to avoid frequent switching near the threshold. Can be set to 10% (e.g.) ).
[0091] By employing a threshold comparison method, continuously changing analog force signals (foot force) can be transformed into clear contact / non-contact states, effectively avoiding state ambiguity and false triggering that may be caused by sensor zero drift, vibration noise, and other factors. Furthermore, the preset threshold can be calibrated and selected based on the robot's weight, foot structure compliance, and sensitivity requirements. In some embodiments, the threshold can be a few percent to a dozen percent of the robot's weight; for example, for a 60kg robot, the threshold is 5%, or 30N.
[0092] In this way, a clear, precise, and delay-controlled contact state input is provided to the terrain correction module, ensuring the determinism and reliability of the closed-loop correction mechanism. Furthermore, the magnitude of the contact force can be used to calculate the contact center location, which can then be used to correct the terrain slope and stability margin.
[0093] The robot walking control method embodiments of this application have been described in detail above. The following section will combine... Figure 3 This application provides a detailed description of embodiments of the robot walking control device. It should be understood that the descriptions of the robot walking control method embodiments correspond to the descriptions of the robot walking control device embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.
[0094] Figure 3 The diagram shown is a structural schematic of a robot walking control device according to an embodiment of this application. Figure 3 As shown, the robot walking control device provided in this application embodiment includes: The first acquisition module 310 is used to acquire the robot's multimodal perception historical data; The second acquisition module 320 is used to acquire contact observation information of the robot's foot. The generation module 330 is used to generate a predicted representation of the terrain state of the robot's foot based on multimodal perception historical data. The correction module 340 is used to adjust the predicted terrain state representation of the robot's foot using contact observation information of the robot's foot, so as to obtain the corrected terrain state representation. The determination module 350 is used to determine the robot's walking control commands based on the corrected terrain state representation.
[0095] In some embodiments, the determining module 350 is further configured to generate a motion correction amount for adjusting the basic walking motion based on the corrected terrain state representation; and to superimpose the motion correction amount with the basic walking motion to obtain the robot's walking control command.
[0096] In some embodiments, the determining module 350 is further configured to: acquire state information of the robot body, the state information of the robot body including at least one of joint angle information, joint velocity information and posture information; and generate motion correction amount based on the state information of the robot body and the corrected terrain state representation; wherein the basic walking motion is generated based on the state information of the robot body.
[0097] In some embodiments, the modified terrain state characterization includes at least one of the following physical quantities: terrain height, used to guide the robot's foot lift height and landing time; terrain slope, including a first slope along the robot's forward direction and a second slope along the robot's lateral direction, used to adjust the robot's center of gravity position and gait stability; terrain roughness, used to adjust the robot's gait stability margin and stride length; and contact stability index, used to determine the softness of the ground and the risk of slippage.
[0098] In some embodiments, the generation module 330 is further configured to: map multimodal perception historical data to a latent space using a terrain state prediction network to obtain a latent representation of terrain state; and decode the latent representation of terrain state using a decoder to obtain a predicted representation of the terrain state of the robot's foot; wherein the loss function for training the terrain state prediction network is composed of a weighted combination of at least two of the following: terrain height prediction error, terrain slope prediction error, terrain roughness prediction error, and contact stability index prediction error; wherein the multimodal perception historical data includes at least two of the following: depth image historical sequence, foot contact force historical sequence, and joint angle historical sequence.
[0099] In some embodiments, the correction module 340 is further configured to: acquire the foot contact state in the contact observation information; calculate the terrain state correction amount based on the difference between the foot contact state and the terrain state prediction representation of the robot's foot; and update the terrain state prediction representation of the robot's foot based on the terrain state correction amount to obtain the corrected terrain state representation.
[0100] In some embodiments, the second acquisition module 320 is further configured to acquire the contact force of the robot's foot; and determine the contact observation information of the robot's foot based on a comparison between the contact force of the robot's foot and a preset contact force threshold; wherein the contact observation information includes the foot contact state and the magnitude of the contact force.
[0101] Below, for reference Figure 4 This describes an electronic device according to embodiments of the present application. Figure 4 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.
[0102] like Figure 4 As shown, the electronic device 40 includes one or more processors 401 and memory 402.
[0103] The processor 401 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 40 to perform desired functions.
[0104] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may execute the program instructions to implement the robot walking control methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various contents such as multimodal perception historical data, contact observation information, terrain state prediction representations, and corrected terrain state representations.
[0105] In one example, the electronic device 40 may also include an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0106] The input device 403 may include, for example, a keyboard, a mouse, etc.
[0107] The output device 404 can output various types of information to the outside, including multimodal sensing historical data, contact observation information, terrain state prediction representation, and corrected terrain state representation. The output device 404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0108] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 40 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 40 may include any other suitable components depending on the specific application.
[0109] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the robot walking control methods according to various embodiments of this application as described above.
[0110] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0111] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the robot walking control methods according to various embodiments of this application described above.
[0112] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0113] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.
[0114] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0115] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.
[0116] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0117] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A robot walking control method, characterized in that, include: Acquire multimodal perception history data of the robot, wherein the multimodal perception history data includes at least two of the following: depth image history sequence, foot contact force history sequence, and joint angle history sequence; Acquire contact observation information of the robot's feet; Based on the multimodal perception historical data, a terrain state prediction representation of the robot's foot is generated, including: mapping the multimodal perception historical data to a latent space using a terrain state prediction network to obtain a latent representation of the terrain state; decoding the latent representation of the terrain state using a decoder to obtain a terrain state prediction representation of the robot's foot; wherein, the loss function for training the terrain state prediction network is composed of a weighted combination of at least two of the following: terrain height prediction error, terrain slope prediction error, terrain roughness prediction error, and contact stability index prediction error; The terrain state prediction representation of the robot's foot is adjusted using the contact observation information of the robot's foot to obtain a corrected terrain state representation; Based on the corrected terrain state representation, the robot's walking control command is determined, including: generating a motion correction amount for adjusting the basic walking motion based on the corrected terrain state representation; superimposing the motion correction amount with the basic walking motion to obtain the robot's walking control command, wherein the basic walking motion is a sequence of motion commands for maintaining the robot's basic gait on flat or simple terrain, and the motion commands do not depend on the corrected terrain state representation.
2. The method according to claim 1, characterized in that, The generation of motion correction amounts for adjusting basic walking movements based on the corrected terrain state representation includes: Obtain the state information of the robot body, wherein the state information of the robot body includes at least one of joint angle information, joint velocity information, and posture information; The motion correction amount is generated based on the state information of the robot body and the corrected terrain state representation; The basic walking motion is generated based on the state information of the robot body.
3. The method according to claim 1 or 2, characterized in that, The corrected terrain state characterization includes at least one of the following physical quantities: Terrain height is used to guide the robot's foot lift height and landing time; Terrain slope, including a first slope along the robot's forward direction and a second slope along the robot's lateral direction, is used to adjust the robot's center of gravity position and gait stability; Terrain roughness is used to adjust the robot's gait stability margin and stride length; Contact stability indexes are used to assess the softness of the ground and the risk of slippage.
4. The method according to claim 1 or 2, characterized in that, The step of adjusting the predicted terrain state representation of the robot's foot using contact observation information to obtain a corrected terrain state representation includes: Obtain the foot contact state from the contact observation information; Based on the difference between the foot contact state and the predicted terrain state of the robot's foot, a terrain state correction amount is calculated; Based on the terrain state correction amount, the terrain state prediction representation of the robot's foot is updated to obtain the corrected terrain state representation.
5. The method according to claim 1 or 2, characterized in that, The acquisition of contact observation information of the robot's foot includes: To obtain the contact force of the robot's foot; Based on the comparison between the contact force of the robot's foot and a preset contact force threshold, the contact observation information of the robot's foot is determined; The contact observation information includes the foot contact state and the magnitude of the contact force.
6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the robot walking control method according to any one of claims 1 to 5.
7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is used to execute the robot walking control method according to any one of claims 1 to 5.
8. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the robot walking control method according to any one of claims 1 to 5.
