Robot walking control method, system, storage medium and robot

Abstract

The application discloses a walking control method and system of a robot, a storage medium and the robot, and belongs to the field of robot control. The walking control method comprises the following steps: acquiring current observation information of the robot at a current time and historical observation information of the robot at a plurality of continuous historical times before the current time; determining motion state information of the robot at the current time according to the historical observation information of the plurality of continuous historical times and the current observation information; acquiring terrain elevation information of a surrounding area of the robot, and generating a control instruction for controlling the robot to walk according to the current observation information, the motion state information at the current time and the terrain elevation information. The technical scheme of the application can improve the control stability and safety of the robot in the walking process on complex terrains such as steps and slopes.

Description

Technical Field

[0001] This application relates to the field of robot control technology, specifically to a robot walking control method, a robot walking control system, a storage medium, a computer program product, and a robot. Background Technology

[0002] In scenarios such as disaster relief, field inspection, industrial inspection, and exploration of unknown environments, robots often need to perform walking tasks in complex terrain environments, including steps, slopes, gravel surfaces, and uneven roads. Due to the varied terrain and irregular footholds in these environments, the difficulty of walking control is significantly increased. Therefore, how to achieve stable robot walking control in complex terrain environments has become an important research direction in the field of robotics. Summary of the Invention

[0003] This application provides a robot walking control method, a robot walking control system, a storage medium, a computer program product, and a robot, which improves the control stability of the robot during walking in complex terrain.

[0004] According to a first aspect of this application, a walking control method for a robot is provided, comprising: acquiring current observation information of the robot at the current moment and historical observation information of each of a series of consecutive historical moments prior to the current moment; determining motion state information of the robot at the current moment based on the historical observation information of each of the series of consecutive historical moments and the current observation information; and generating control commands for controlling the robot to walk based on the current observation information and the motion state information.

[0005] According to a second aspect of this application, a walking control system for a robot is provided, comprising: an acquisition module for acquiring current observation information of the robot at the current moment and historical observation information of each of a series of consecutive historical moments prior to the current moment; a determination module for determining motion state information of the robot at the current moment based on the historical observation information of each of the series of consecutive historical moments and the current observation information; and a generation module for generating control commands for controlling the robot to walk based on the current observation information and the motion state information.

[0006] According to a third aspect of this application, a robot is provided, including a processor and a memory, wherein a computer program is stored in the memory, and when executed by the processor, the computer program implements the walking control method of the robot as described in the first aspect.

[0007] According to a fourth aspect of this application, a computer-readable storage medium is provided that stores a plurality of instructions adapted for loading by a processor to perform the steps of the walking control method for a robot as described in the first aspect.

[0008] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the robot walking control method as described in the first aspect.

[0009] The robot walking control method, robot walking control system, storage medium, computer program product, and robot of this application embodiment determine motion state information by utilizing the robot's current observation information at the current moment and the historical observation information corresponding to each of multiple consecutive historical moments before the current moment. This allows reliable motion state information to be obtained in real-time during deployment without relying on additional sensors or privileged state information. Control commands for controlling the robot are then generated based on this motion state information and the current observation information, improving the robot's control stability during walking in complex terrain environments. Furthermore, the information upon which the control commands are generated integrates motion state information derived from the temporal observation sequence and the control commands from the previous moment, ensuring that the generated control commands are temporally continuous. This avoids abrupt gait changes and motion jitter, further enhancing the robot's walking stability.

[0010] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the robot walking control method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating a robot walking control method provided in another embodiment of this application; Figure 3A The diagram illustrates the distribution of attention weights assigned to each grid cell in a local elevation map of the robot's surrounding area acquired in a staircase terrain scene. Figure 3B This illustration shows the robot's ability to perform early warning and decision-making in dangerous terrain scenarios. Figure 4 This is a flowchart illustrating the training method of the walking control model provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the principle of the training method for the walking control model provided in the embodiments of this application; Figure 6 The diagram illustrates the training curves with and without sample motion state information estimation. Figure 7 This is a flowchart illustrating the training method of the walking control model provided in another embodiment of this application; Figure 8 This is a schematic diagram illustrating the principle of the training method for the walking control model provided in another embodiment of this application; Figure 9 This is a schematic diagram comparing the walking control effect of combining terrain reconstruction and motion state estimation provided in the embodiments of this application; Figure 10 This is a schematic diagram illustrating the principle of the training method for the walking control model provided in another embodiment of this application; Figure 11 Schematic illustration based on Figure 10 The training method example shown is a detailed comparison of the robot climbing stairs. Figure 12 This is a schematic diagram of the symmetry plane used for the mirror transformation provided in the embodiments of this application; Figure 13 This is a comparison diagram of the robot's trajectory when walking in a straight line, provided in the embodiments of this application; Figure 14 This is a comparison diagram of the landing positions of the robot in the first 5 steps when it walks along a straight line, as provided in the embodiments of this application; Figure 15 This is a block diagram of the walking control system of the robot provided in the embodiments of this application; Figure 16 This is a schematic diagram of the robot provided in the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0015] With the development of robotics technology, autonomous walking control of robots in complex terrain environments such as steps, slopes, and gravel surfaces has attracted increasing attention. In recent years, walking control methods based on deep reinforcement learning have gradually become mainstream. These methods typically involve extensive interactive training in a simulation environment, utilizing a policy network to directly output control commands based on the robot's current state information, thus achieving a certain degree of walking control for robots in complex terrain environments.

[0016] In deep reinforcement learning-based walking control methods, the stability of robot walking is often highly dependent on the real-time acquisition of the robot's state information. During simulation training, the motion state information contained in this state information (such as the robot's speed and foot position information) is privileged information, which can be directly obtained in the simulation environment. However, in real-world robot systems, this motion state information is often difficult to obtain directly and accurately. For example, in the actual deployment phase, it is usually necessary to add additional sensors or use optical motion capture devices to acquire the aforementioned motion state information. However, these methods are not only complex and costly to deploy, but also prone to introducing noise that affects the accuracy of the state variables. This leads to an information discrepancy between the simulation training environment and the actual deployment scenario, thus affecting the control stability of the strategy in real terrain environments.

[0017] In view of this, embodiments of this application provide a robot walking control method, a robot walking control system, a storage medium, a computer program product, and a robot. By acquiring the robot's current observation information at the current moment and the historical observation information of multiple consecutive historical moments prior to the current moment, and utilizing the motion trend information contained in the historical observation information of these multiple historical moments, the robot's motion state information at the current moment is estimated based on the historical observation information of each of the multiple historical moments and the current observation information. Then, based on the current observation information and the motion state information at the current moment, control commands for controlling the robot's walking are generated. Thus, without relying on motion state information that is difficult to obtain directly in the real environment, complete and reliable state information can be provided for walking control. This solves the problem in related technologies where the inconsistency between the state information during simulation training and the state information during actual deployment, caused by the difficulty in directly and accurately obtaining motion state information in the real environment, leads to insufficient stability in walking control.

[0018] The following is a detailed description in conjunction with the accompanying drawings. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments. Although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the drawings.

[0019] Figure 1 This is a flowchart illustrating the robot walking control method provided in the embodiments of this application.

[0020] Please see Figure 1 The robot's walking control method includes steps S101 to S103. This walking control method can be executed by the robot itself or by a walking control system integrated into the robot. This robot's walking control method is suitable for autonomous and stable walking control of the robot in complex terrain environments such as steps, slopes, gravel surfaces, and uneven surfaces. The steps of this method are described below.

[0021] Step S101: Obtain the robot's current observation information at the current moment and the historical observation information of each of the previous multiple historical moments.

[0022] Specifically, the aforementioned robot can be, for example, a humanoid robot, a quadruped robot, or other legged mobile robot. During the robot's movement, it collects observation information once per control cycle. This observation information serves as part of the input information used for the robot's walking control decisions and can be directly obtained from the real environment. For ease of description, each control cycle will be referred to as a moment below.

[0023] Current observation information refers to the observation information collected by the robot at the current moment, including at least one of the following: the robot's current body state information, the command information received by the robot at the current moment, and the control command from the previous moment. The body state information refers to a set of physical quantities that characterize the robot's own motion state, such as the robot's joint positions, joint velocities, angular velocities, and the projection component of the gravity direction in the robot's coordinate system—commonly measurable state quantities. Command information is used to characterize the robot's received walking intentions, such as walking direction and walking speed. Control commands are joint-level commands for the robot to execute walking actions, such as target angles, target angular velocities, or target torques for each joint.

[0024] The historical observation information for multiple consecutive historical moments refers to the observation information collected at each of the consecutive historical moments preceding the current moment. For example, if the current moment is t, and the number of historical moments is k-1, where k is an integer greater than or equal to 3, then the consecutive historical moments are t-k+1, t-k+2, ..., t-1. The historical observation information for each historical moment includes at least one of the following: the robot's body state information at that historical moment, the instruction information received by the robot at that historical moment, and the control instruction from the previous moment.

[0025] Historical observation information from multiple consecutive historical moments records the robot's motion changes over a recent period (e.g., from time t-k+1 to time t-1). This information contains clues about the evolution of the robot's current motion state (such as motion speed and foot position) over time, which can provide a basis for accurately estimating motion state information that is difficult to measure directly at the current moment.

[0026] Step S102: Based on the historical observation information and current observation information of the above-mentioned multiple consecutive historical moments, determine the robot's motion state information at the current moment.

[0027] Motion state information plays a crucial role in robot walking control but is difficult to obtain directly and accurately from conventional sensors. This motion state information includes, for example, at least one of the robot's motion velocity information and foot state information. The motion velocity information includes, for example, the linear velocity of the robot body. The foot state information includes, for example, at least one of foot contact state information and foot position information. The foot contact state information characterizes the contact state between the robot's foot and the ground, such as whether the foot is touching the ground or off the ground. The foot position information includes, for example, the position of the foot relative to the robot's body coordinate system.

[0028] In a simulation training environment, motion state information at each moment can be directly obtained, which is considered privileged state information. However, in a real environment, directly measuring this motion state information often requires additional linear velocity sensors, foot tactile sensors, or optical motion capture equipment. This is not only complex and costly to deploy, but also results in measurement noise and latency that reduce the accuracy of the state variables. Therefore, this step utilizes historical and current observation information that can be directly measured by the robot to estimate the motion state information that is difficult to obtain directly, thereby obtaining the state variables required for control decisions without introducing additional sensing equipment.

[0029] Specifically, the historical and current observation information from multiple consecutive historical moments can be constructed into a time-series observation sequence. This time-series observation sequence refers to the sequence data formed by splicing or stacking the historical and current observation information from multiple consecutive historical moments in chronological order. It can reflect the continuous change trend of the robot's motion state in the time dimension. For example, the time-series observation sequence can be represented as O t ={o t-k+1 ,…, o t-1 ,o t}, where o t This represents the robot's current observation information at the current time (i.e., time t), and correspondingly, o t-k+1 to o t-1This represents the historical observation information corresponding to the robot at each of the k-1 consecutive historical moments preceding the current moment. Then, the robot's motion state information at the current moment is determined based on this time-series observation sequence.

[0030] Step S103: Based on the current observation information and the motion state information at the current moment, generate control commands for controlling the robot's movement.

[0031] Understandably, current observation information represents the robot's instantaneous state variables that are directly obtainable at the current moment, while motion state information represents state variables that are difficult to measure directly but are crucial for walking stability. The current state information formed by combining the two constitutes a complete representation of the robot's current actual motion state. This ensures that the control commands generated based on this current state information match the robot's current actual motion state, thereby guaranteeing the stability of walking control. Furthermore, because the current state information incorporates motion state information derived from the time-series observation sequence and the control commands from the previous moment, the generated control commands are continuous in time. This avoids abrupt gait changes and jitter during robot walking, thus improving the stability of robot walking.

[0032] The robot walking control method of this application determines motion state information by utilizing the robot's current observation information at the current moment and the historical observation information corresponding to multiple consecutive historical moments before the current moment. This allows reliable motion state information to be obtained in real-time during deployment without relying on additional sensors or privileged state information. Control commands for controlling the robot are then generated based on this motion state information and the current observation information, improving the robot's control stability during walking in complex terrain environments. Furthermore, the information upon which the control commands are generated integrates motion state information derived from the time-series observation sequence and the control commands from the previous moment, ensuring that the generated control commands are temporally continuous. This avoids abrupt gait changes and motion jitter, further enhancing the robot's walking stability.

[0033] In some embodiments, to further enhance the robot's ability to walk in complex terrain environments with undulating terrain, such as steps and slopes, the terrain elevation information of the area surrounding the robot can be used as one of the bases for generating control commands, so that the generated control commands are adapted to the robot's actual motion state and the terrain environment at the current moment. See below for reference. Figure 2 This needs to be explained.

[0034] Figure 2 This is a flowchart illustrating a robot walking control method according to another embodiment of this application. Figure 2As shown, the robot's walking control method includes steps S201, S202, S204, and S2031. Steps S201 and S202 are implemented in the same manner as steps S101 and S102, respectively; the repeated parts will not be described in detail here.

[0035] Step S201: Obtain the robot's current observation information at the current moment and the historical observation information of each of the previous multiple historical moments.

[0036] Step S202: Determine the robot's motion state information at the current moment based on the historical observation information and current observation information of multiple consecutive historical moments.

[0037] Step S204: Obtain terrain elevation information of the area surrounding the robot.

[0038] Topographic elevation information is used to characterize the undulations of the ground surface in the area surrounding the robot. For example, a local elevation map established based on the robot's body coordinate system can be used to characterize this topographic elevation information.

[0039] Specifically, 3D terrain point clouds of the area surrounding the robot can be acquired using sensing devices such as depth cameras and LiDAR. These point clouds are then converted to the robot's coordinate system and rasterized into a regular grid elevation map covering a preset area around the robot (e.g., 1.6m along the forward direction and 1.0m laterally). In one implementation, the elevation map's length extends along the robot's forward direction (i.e., the x-axis of the robot's coordinate system), and its width extends along the robot's horizontal lateral direction (i.e., the y-axis of the robot's coordinate system). This ensures that the terrain information expressed by the local elevation map remains consistent with the robot's pose and direction of movement; that is, the local elevation map is always a local terrain representation "with the robot itself as a reference." The size of the local elevation map is designed to cover the core sensing areas in front of and to the sides of the robot. The resolution and specific size of the local elevation map can be preset based on the robot's sensing capabilities, control computing power, and the terrain environment. In one example, the resolution of the local elevation map is, for example, 0.1m. Each grid in the local elevation map records the height value of the terrain surface relative to the reference plane at the corresponding grid location. The spatial location of each grid can be represented by its coordinates in the robot's coordinate system. Understandably, compared to directly processing disordered, massive 3D point cloud data, regular rasterized local elevation map data is smaller in scale, has a regular structure, and has low computational overhead, which can meet the real-time processing needs of terrain information during the robot's dynamic walking process.

[0040] Step S2031: Generate control commands based on current observation information, current motion status information, and terrain elevation information.

[0041] It is understandable that in complex terrain environments such as steps and slopes, if control commands are generated solely based on the robot's own observation and motion state information, the robot will be unable to perceive terrain undulations. For example, if the robot cannot perceive the height of the steps when climbing them, insufficient leg lift will lead to a collision with the steps, and it will be difficult to pre-adjust its gait when descending slopes. In view of this, embodiments of this application use the acquired terrain elevation information as one of the bases for generating control commands, so that the generated control commands are adapted to the robot's actual motion state and the terrain environment at the current moment.

[0042] Specifically, the robot's current state can be determined first based on current observation information and its motion state at the current moment. Then, control commands can be generated based on the current state information and terrain elevation information.

[0043] As mentioned earlier, the current state information, obtained by combining the current observation information and the current motion state information, constitutes a complete representation of the robot's current actual motion state. Meanwhile, the terrain elevation information characterizes the undulations and landing conditions of the surrounding terrain. In other words, the current state information and terrain elevation information respectively depict the "body-side" and "environment-side" information required for walking control decisions. By combining the two, the generation of control commands is no longer limited to the robot's actual motion state. Instead, based on an estimate of its own motion state, the robot can further perceive and utilize surrounding terrain information, adjusting its gait according to terrain undulations and landing conditions. This generates control commands that match the robot's current actual motion state and its surrounding terrain environment, adapting to complex terrain changes and maintaining stable walking. For example, adjusting leg lift height and landing position when approaching terrain with sudden height changes such as steps, and adjusting torso posture on sloping terrain.

[0044] Although the steps of the method have been described above in a specific order, the embodiments of this application are not limited thereto, and the steps may be performed in other orders as needed. For example, in some embodiments, step S204 may be performed before step S201, or simultaneously with step S201, and this application does not impose any restrictions on this.

[0045] In the above embodiments, by using terrain elevation information and the aforementioned current state information together as the basis for generating control commands, the generation of control commands is no longer limited to the robot's own motion state. Instead, based on a complete perception of its own motion state, the robot can further understand the undulations and landing conditions of the surrounding terrain. By adjusting the gait and landing points in advance according to terrain changes, the robot's walking control effect in complex terrain environments with undulations, such as steps and slopes, is improved. This effectively avoids problems such as improper landing, insufficient leg lift, or walking instability caused by a lack of terrain perception, and enhances the robot's adaptability to complex terrain environments and walking safety.

[0046] In some embodiments, the step S2031 described above, which generates control commands based on current observation information, current motion state information, and terrain elevation information, may include the following steps.

[0047] First, based on the current observation information and the motion state information at the current moment, the current state information of the robot is determined.

[0048] In one example, the current observation information and the current motion state information can be concatenated to obtain the current state information. The current state information constitutes a complete representation of the robot's current actual motion state. It includes both the instantaneous state quantities that the robot can directly obtain at the current moment (i.e., the current observation information) and the state quantities that are difficult to measure directly but are crucial to walking stability (i.e., the motion state information at the current moment). This current state information can be used as the basis for querying the robot's own motion state in subsequent attention encoding.

[0049] Next, terrain features are extracted based on terrain elevation information.

[0050] Specifically, terrain features can be obtained by extracting features from terrain elevation information. These terrain features can characterize the changes in terrain height and spatial distribution around the robot. The terrain features can then be used to determine the key and value vectors in subsequent attention encoding. These vectors are then used to perform attention calculations with the query vector obtained by mapping the current state information, thereby achieving adaptive filtering and aggregation of terrain information.

[0051] In one implementation, extracting terrain features based on terrain elevation information includes: performing convolutional feature extraction on the terrain elevation information to obtain terrain texture features; performing feature extraction on the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features; and obtaining terrain features based on the terrain texture features and spatial features.

[0052] The following example, using a local elevation map as an example, illustrates the process of extracting terrain features.

[0053] Suppose the above local elevation map is H(x,y), denoted as H(x,y)=z(x,y), where (x,y) is the two-dimensional coordinate of each grid in the local elevation map in the robot coordinate system, and z(x,y) is the height value of the terrain surface relative to the reference plane at the position corresponding to the two-dimensional coordinate.

[0054] First, convolutional feature extraction is performed on the aforementioned local elevation map H(x,y) to obtain terrain texture features.

[0055] Specifically, by performing a convolution operation on the local elevation map, the undulation texture patterns of the local terrain surface are extracted, such as height abrupt changes at the edges of steps, slope direction, and obstacle outlines.

[0056] Next, the spatial coordinate information corresponding to the terrain elevation information is used to extract features to obtain spatial features.

[0057] Specifically, spatial features are obtained by performing feature mapping on the spatial coordinate information of each grid in the local elevation map (e.g., the three-dimensional spatial coordinates of each grid in the robot coordinate system). These spatial features are used to characterize the spatial positional relationships of each terrain grid, such as the orientation and distance information of each terrain grid relative to the robot.

[0058] Finally, the terrain features are obtained based on the terrain texture features and spatial features.

[0059] Specifically, topographic texture features and spatial features are spliced ​​together along the channel dimension to obtain topographic features. These topographic features contain both texture information representing the topographic undulation structure and orientation information representing the spatial relationship of the topography.

[0060] After obtaining the terrain features, the next step is to perform attention encoding on the current state information and the terrain features to obtain attention features.

[0061] Attention encoding is a weighted aggregation operation based on an attention mechanism. Its core idea is to calculate attention weights based on the relevance between the query vector and the key vector, and then use these weights to perform a weighted summation of the value vectors, thereby highlighting information relevant to the query vector. In this embodiment, the current state information can be mapped to a query vector, and terrain features can be mapped to key and value vectors respectively. Then, attention weights are calculated based on the query vector and the key vector, and the value vectors are weighted and aggregated according to these attention weights to obtain attention features.

[0062] Specifically, the current state information can be mapped to a query vector using a linear mapping, and terrain features can be mapped to key and value vectors using independent linear mappings. Then, attention weights are calculated based on the query and key vectors. The magnitude of the attention weight reflects the importance of each terrain grid region to the current walking decision. For example, key terrain regions relevant to the robot's current movement state, such as feasible areas where the robot is about to land, the edges of steps with abrupt changes in height, and potential obstacle areas, will receive higher attention weights; while redundant or irrelevant terrain regions will receive relatively lower attention weights. Next, the value vectors are weighted and aggregated according to the attention weights to obtain the attention features.

[0063] In some embodiments, a multi-head attention mechanism can also be used to obtain the aforementioned attention features. Specifically, the query vector, key vector, and value vector can be generated into multiple sets of sub-vectors through multiple independent linear mappings. Each set of sub-vectors includes a sub-query vector, a sub-key vector, and a sub-value vector. Then, based on the attention mechanism, each set of sub-vectors is processed to obtain multiple sub-attention features. Next, the multiple sub-attention features are concatenated along the feature dimension, and the concatenated attention features are linearly mapped to obtain the aforementioned attention features. Obtaining attention features through a multi-head attention mechanism can capture the correlation information between terrain features and robot state from multiple perspectives, improving the richness and comprehensiveness of feature representation.

[0064] Through the aforementioned attention encoding, terrain features are no longer indiscriminately and directly incorporated into the decision-making process. Instead, they are adaptively filtered and weighted under the guidance of the robot's current motion state: key terrain areas relevant to the robot's current state (such as footholds, obstacle areas, and step edges) receive higher attention weights, while redundant or irrelevant terrain areas receive lower attention weights. Thus, the robot can dynamically filter and aggregate terrain information based on its own motion state, adaptively focusing on key terrain areas such as footholds and obstacle areas, enabling the prediction of dangerous terrain and avoiding control decision failures due to interference from redundant terrain information.

[0065] Figure 3A The diagram illustrates the distribution of attention weights assigned to each grid cell in a local elevation map of the robot's surrounding area acquired in a staircase terrain scene.

[0066] Please see Figure 3A , Figure 3A Figure 31 schematically illustrates a scenario where the robot is currently in a staircase terrain environment. The robot stands atop the stairs, equipped with environmental sensing devices (…). Figure 3A The image uses "radar" as an analogy (which could correspond to a depth camera, lidar, etc.) to scan the terrain in front of and around the robot, collecting terrain depth data or 3D point clouds. This data is then processed through rasterization and projection to obtain a local elevation map based on the robot's coordinate system. The staircase structure in the image illustrates the physical form of the collected terrain, specifically a staircase terrain with significant height changes. Figure 3AFigure 32 shows the visualization result of the local elevation map from a top-down view after attention encoding: the local elevation map is arranged in a multi-layered elliptical array, with each ellipse corresponding to a terrain grid. The depth of the fill color indicates the level of attention weight assigned to that grid position during the attention encoding process. The darker the grid color, the higher the attention weight assigned to the corresponding terrain area, meaning that the area is more important to the robot's current walking decision; conversely, the lighter the grid color, the lower the attention weight assigned to the corresponding terrain area.

[0067] Combination Figure 3A As shown in sections 31 and 32, in staircase terrain, areas assigned higher attention weights are concentrated on the next step to be taken; while flat areas below the stairs, where the relative height difference with the robot's position is large and difficult to reach safely, are assigned lower attention weights and identified as danger zones. Therefore, the attention encoding scheme of this embodiment can selectively concentrate attention weights on the next landing area most critical to the walking decision, based on the robot's current movement state, while actively reducing redundant attention to distant areas or areas with large height differences. This ensures accurate perception of key terrain while avoiding redundant interference introduced by treating all terrain grids with equal input. Next, control commands are generated based on the attention characteristics.

[0068] Specifically, by using attentional characteristics as the basis for generating control commands, the robot can perceive the undulations and landing conditions of the surrounding terrain in advance, so as to adjust the robot's gait and landing in a timely manner. This generates control commands that are adapted to the robot's current motion state and the terrain environment, improving the accuracy of the control commands in response to key terrain areas (such as landing points, obstacle areas, and step edges) and the adaptability to complex terrain environments, thereby ensuring the stability and safety of the robot's walking process.

[0069] To verify the effectiveness of the above attention encoding scheme, in a simulation environment, when the robot that has completed terrain attention encoding is placed on the edge of a high platform and given a speed command to continue moving forward, the robot is able to autonomously stay in place based on its prediction of dangerous terrain and refuse to execute the command to walk down the platform (e.g., Figure 3B (As shown). Robots that do not employ the aforementioned attention encoding will continue walking towards the edge of the platform, potentially falling off. Therefore, the walking control method in this embodiment enables the robot to anticipate and avoid dangerous terrain in a timely manner, significantly improving the safety of the robot's movement across complex terrain.

[0070] Based on the above embodiments, control commands can also be generated according to attention characteristics and current state information.

[0071] Specifically, attention features provide the walking control decision with the terrain areas (such as footholds, obstacle areas, and step edges) and their spatial orientations that should be prioritized in the current state, while the current state information provides the robot's own motion posture information. By utilizing attention features and current state information to generate control commands, the generated control commands can accurately adapt to the robot's current terrain environment and match the robot's actual motion state. This avoids problems such as improper footholds or unstable movements caused by a disconnect between the robot's motion state and terrain information, further improving the robot's walking stability and safety in complex terrain environments.

[0072] In some embodiments, a pre-built walking control model can also be used to generate control commands. This walking control model may include a state estimation network and a policy network. This process is described below.

[0073] In one example, historical observation information and current observation information from multiple consecutive historical moments can be constructed into a temporal observation sequence, as described in the embodiments above. Then, the temporal observation sequence is input into the state estimation network of a pre-built walking control model to obtain the robot's motion state information at the current moment. Subsequently, control commands for controlling the robot's walking are generated based on the current motion state information and the current observation information.

[0074] Specifically, the state estimation network can be, for example, a multilayer perceptron (MLP) network. The aforementioned temporal observation sequence can be input into the state estimation network, which outputs the robot's motion state information at the current moment, including at least one of the robot's current motion velocity and foot state information. Then, the aforementioned control commands are generated based on the current motion state information and the current observation information.

[0075] Based on the above embodiments, a time-series observation sequence is constructed by combining historical observation information from multiple consecutive historical moments with current observation information to reflect the continuous changing trend of the robot's motion state over time. This time-series observation sequence is then input into a state estimation network to estimate motion state information. This allows the motion trend information contained in the time-series observation sequence to be effectively mined and mapped into an estimate of the current motion state. Thus, motion state information can be obtained in real time without the need for additional sensors or reliance on simulation privileged information, providing reliable state input for the generation of subsequent control commands. This alleviates the problem of insufficient walking control stability caused by the difficulty in directly obtaining motion state information in real-world environments.

[0076] In another example, historical observation information from multiple consecutive historical moments and current observation information can be constructed into a temporal observation sequence. Then, the robot's motion state information at the current moment is determined based on the temporal observation sequence. Subsequently, the motion state information and current observation information are input into the policy network of the walking control model to generate control commands for controlling the robot's movement.

[0077] The policy network can, for example, employ a multilayer perceptron structure, comprising an input layer, multiple hidden layers (e.g., three hidden layers), and an output layer connected sequentially. The input layer receives the current motion state information and the current observation information, or the current state information determined by the current motion state information and the current observation information. Each hidden layer is, for example, a fully connected layer, and each hidden layer is followed by a ReLU nonlinear activation layer to perform a nonlinear mapping on the output of the previous layer. The output layer outputs control commands.

[0078] Specifically, a temporal observation sequence can be constructed as described in the above embodiments, and the robot's motion state information at the current moment can be determined based on the temporal observation sequence. Then, the robot's current state information is determined based on the motion state information at the current moment and the current observation information, and this current state information is input into the policy network of the walking control model, from which the policy network outputs control commands.

[0079] Based on the above embodiments, motion state information is determined based on time-series observation sequences, and the motion state information and current observation information are input into the policy network to generate control commands. This makes the decision input of the policy network include both the robot's real and measurable instantaneous state at the current moment and the motion state information derived from the time-series observation sequences. The state on which the decision is based is more complete, thereby generating control commands that match the robot's current actual motion state and improving the stability and reliability of walking control.

[0080] In another example, historical observation information from multiple consecutive historical moments and current observation information can be constructed into a temporal observation sequence. This temporal observation sequence is then input into the state estimation network of a pre-built walking control model to obtain the robot's motion state information at the current moment. Subsequently, the current motion state information and the current observation information are input into the policy network of the walking control model to generate control commands for controlling the robot's movement.

[0081] Based on the above embodiments, the motion state information at the current moment is estimated by inputting the time-series observation sequence into the state estimation network. Then, the motion state information and the current observation information are input into the policy network to generate control commands. The state estimation network and the policy network form a complete closed loop of "state estimation-decision control": the state estimation network is responsible for converting the motion trend contained in the time-series observation sequence into a motion state prior that is difficult to measure directly at the current moment, while the policy network makes decisions based on the complete state representation of "instantaneously measurable state + motion state prior". The two perform their respective functions and cooperate with each other, so that the control commands can accurately reflect the robot's current actual motion state and have the state support provided by historical motion trends, thereby effectively improving the stability and reliability of the robot's walking control in the real environment.

[0082] In other embodiments, the pre-built walking control model may include a state estimation network, a terrain feature extraction network, and a policy network. The state estimation network and policy network have the same structure and function as described in the above embodiments, and will not be repeated here. The process of generating control commands based on the walking control model in this embodiment is described below.

[0083] For example, the terrain feature extraction network described above may include a convolutional neural network (CNN) and a multilayer perceptron. The CNN, for example, includes an input layer, multiple convolutional feature extraction layers, and an output layer connected in sequence. Each convolutional feature extraction layer includes at least one convolutional layer and a ReLU nonlinear activation layer connected thereafter. The input layer of the CNN receives terrain elevation information, and the multiple convolutional feature extraction layers sequentially perform convolutional feature extraction on the terrain elevation information to obtain terrain texture features. The output layer outputs these terrain texture features. The multilayer perceptron extracts features from the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features. The obtained terrain texture features and spatial features are concatenated along the channel dimension to obtain the terrain features.

[0084] Specifically, historical observation information from multiple consecutive historical moments and current observation information can be constructed into a time-series observation sequence. This time-series observation sequence is then input into the state estimation network of a pre-built walking control model to obtain the robot's motion state information at the current moment. Next, the terrain elevation information of the robot's surrounding area is acquired, and convolutional features are extracted from this terrain elevation information using a convolutional neural network in the terrain feature extraction network to obtain terrain texture features. Simultaneously, a multilayer perceptron in the terrain feature extraction network is used to extract features from the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features. Then, the terrain texture features and spatial features are concatenated along the channel dimension to obtain the terrain features. Finally, the current observation information, the current motion state information, and the terrain features are input into the policy network of the walking control model, which outputs control commands based on the above information.

[0085] Based on the above embodiments, by inputting the motion state information estimated by the state estimation network and the terrain features extracted by the terrain feature extraction network, along with the current observation information, into the policy network to generate control commands, the decision-making basis of the policy network simultaneously encompasses the robot's own motion state and the surrounding terrain information. On the one hand, the motion state information provides decision-making with state quantities that are difficult to measure directly, such as motion speed information and foot position information, while the current observation information provides decision-making with state quantities that can be measured directly. On the other hand, the terrain features provide decision-making with the undulation structure and spatial orientation information of the surrounding terrain. Thus, the generation of control commands is no longer limited to the robot's own motion state, but can adjust the gait by combining the undulation changes of the surrounding terrain and the foot landing conditions, thereby improving the robot's walking adaptability in complex terrain environments such as steps and slopes.

[0086] In some other embodiments, the pre-built walking control model may include a state estimation network, a terrain feature extraction network, an attention encoding module, and a policy network. The state estimation network, terrain feature extraction network, and policy network have the same structure and function as those described in the previous embodiments, and will not be repeated here. The process of generating control commands based on the walking control model in this embodiment is described below.

[0087] Specifically, a temporal observation sequence can be constructed by combining historical observation information from multiple consecutive historical moments with current observation information. This temporal observation sequence is then input into the state estimation network of a pre-built walking control model to obtain the robot's motion state information at the current moment. Next, the terrain elevation information of the robot's surrounding area is acquired, and convolutional features are extracted from this terrain elevation information using a convolutional neural network in the terrain feature extraction network to obtain terrain texture features. Simultaneously, a multilayer perceptron in the terrain feature extraction network extracts features from the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features. Then, the terrain texture features and spatial features are concatenated along the channel dimension to obtain the terrain features. Next, based on the current observation information and motion state information, the robot's current state information is determined, and attention encoding is performed on the current state information and terrain features using an attention encoding module to obtain attention features. Finally, the attention features are input into the aforementioned policy network to generate control commands.

[0088] Building upon the aforementioned embodiments, some embodiments of the walking control model in this application further incorporate an attention encoding module. By utilizing this module to encode the current state information and terrain features, terrain information is no longer indiscriminately appended into the decision-making process. Instead, it is adaptively filtered and weighted under the guidance of the robot's current motion state. Key terrain areas relevant to the current state (such as footholds, obstacle areas, and step edges) receive higher attention weights, while redundant or irrelevant terrain areas receive lower attention weights. Thus, the robot can dynamically focus on terrain areas important to its current walking decision based on its own motion state, achieving accurate perception of key terrain and early prediction of dangerous terrain, further improving the effectiveness of control commands and the safety of walking in complex terrain environments.

[0089] Based on the above embodiments, attention features and current state information can also be input into the policy network, and the policy network can output control commands based on the attention features and current state information.

[0090] Understandably, attention features provide decision-making with "terrain information that should be prioritized in the current state," while current state information preserves "the robot's own motion state information" (such as position, posture, speed, and foot position). These two information sources are complementary and non-substitutable: if only attention features are input, the direct perception of the robot's real-time motion state may be weakened during the aggregation of terrain information; if only current state information is input, terrain information cannot participate in decision-making. Inputting both into the policy network allows the network to grasp the robot's real-time motion based on current state information and to acquire key terrain information related to the current state based on attention features. This integrates the robot's motion state with the external terrain environment for decision-making, generating control commands that accurately adapt to the robot's current motion state and the surrounding terrain. This avoids improper landing or instability caused by the separation of robot state and terrain information, further improving the stability and safety of robot walking control in complex terrain environments.

[0091] Based on the same inventive concept, embodiments of this application also provide a method for training a walking control model. The training process of this walking control model is described below.

[0092] In some embodiments, the training method for the walking control model includes: acquiring training sample data, which includes sample observation information of the robot at the sample time, sample historical observation information of each of the multiple consecutive sample historical times before the sample time, and motion state information labels of the robot at the sample time; and updating the parameters of the initial walking control model based on the sample observation information, the sample historical observation information of each of the multiple consecutive sample historical times, and the motion state information labels, to obtain the walking control model.

[0093] Specifically, training sample data can include the robot's sample observation information at the sample time, the sample history observation information of multiple consecutive sample history times prior to the sample time, and the robot's motion state information label at the sample time. In one example, training sample data can be collected by controlling the robot to walk in a simulation environment. Specifically, different types of complex terrain (such as steps, slopes, gravel surfaces, etc.) can be arranged in the simulation environment, the robot can be controlled to perform walking tasks in the complex terrain, and training sample data can be collected at each sample control cycle (or sample time).

[0094] The sample observation information includes at least one of the robot's body state information at the sample time, the instruction information received by the robot at the sample time, and the control instruction from the previous time. The sample history observation information for each historical sample time includes at least one of the robot's body state information at that historical sample time, the instruction information received by the robot at that historical sample time, and the control instruction from the previous time. It should be noted that the body state information, instruction information, and control instructions from the previous time contained in the sample observation information and the sample history observation information are similar to the body state information, instruction information, and control instructions from the previous time described in the above embodiments, and will not be repeated here.

[0095] The motion state information label is used to indicate the robot's true motion state at the sample time, such as including at least one of the robot's true motion velocity information and true foot state information acquired in the simulation environment at the sample time. The true foot state information includes, for example, at least one of true foot contact state information and true foot position information.

[0096] In this embodiment, because the training sample data incorporates sample observation information and historical observation information from multiple consecutive historical sample moments, the model can utilize the trend information of motion state evolution over time during training, providing a temporal information basis for estimating motion states that are difficult to measure directly at sample moments. Simultaneously, motion state information labels are used as supervision signals to update the initial walking control model parameters, enabling the model to learn the mapping ability to infer motion state information from observable information. The walking control model trained in this way can estimate motion state information in real time during deployment using only the current observation information and the historical observation information corresponding to multiple consecutive historical moments, without relying on privileged state information that is difficult to obtain in the real environment. This alleviates the problem of insufficient walking control stability caused by inconsistencies between training and deployment states, and improves the stability of robot walking control in complex terrain environments.

[0097] In some embodiments, the parameters of the initial walking control model are updated based on sample observation information, sample historical observation information for each of multiple consecutive sample historical moments, and motion state information labels to obtain the walking control model. This includes: inputting sample observation information and sample historical observation information for each of multiple consecutive sample historical moments into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment; and updating the parameters of the initial walking control model based on the sample observation information, sample motion state information, and motion state information labels to obtain the walking control model.

[0098] Specifically, the state estimation network processes a sequence of sample time-series observations, constructed from sample observation information and the historical observation information of multiple consecutive sample time points, to estimate the robot's sample motion state information at each sample time point. Its training objective is to make the estimated sample motion state information approximate the motion state information label (i.e., the true motion state information in the simulation environment). Through supervised training, the state estimation network learns the mapping relationship between the sample time-series observation sequence and the motion state information. Therefore, in the deployment phase after training, only measurable observation information needs to be input to output reliable motion state information, without relying on privileged state information. For example, the state estimation network can employ a multilayer perceptron.

[0099] Based on the above embodiments, during the training phase, sample observation information and the historical observation information of each of multiple consecutive sample historical moments are input into the state estimation network. Motion state information labels are used as supervision, enabling the state estimation network to learn the mapping ability to estimate motion velocity information, foot state information, and other difficult-to-measure state quantities from directly observable information (especially multi-frame observations containing temporal variation trends). Correspondingly, the trained walking control model only needs to input measurable observation information during the deployment phase to obtain the aforementioned motion state information in real time, without the need for additional sensors or reliance on privileged information. This eliminates the information difference between the training and deployment phases, thereby improving the stability of walking control and the ability to transfer and deploy strategies.

[0100] In some embodiments, the training method for the walking control model further includes acquiring sample terrain elevation information of the area surrounding the robot; the above-mentioned updating the parameters of the initial walking control model based on sample observation information, sample motion state information, and motion state information labels to obtain the walking control model includes: updating the parameters of the initial walking control model based on sample observation information, sample motion state information, sample terrain elevation information, and motion state information labels to obtain the walking control model.

[0101] Based on the above embodiments, by incorporating sample terrain elevation information into the parameter updates of the initial walking control model during the training phase, the walking control model learns to make decisions by combining its own movement state with surrounding terrain information during training. Consequently, the trained walking control model can perceive terrain undulations such as steps and slopes during deployment and adjust its leg lift height, foot placement, and gait in advance, thereby improving its walking adaptability and success rate in complex terrain environments.

[0102] Figure 4 This is a flowchart illustrating the training method for the walking control model provided in this application embodiment.

[0103] like Figure 4As shown, the training method for the walking control model includes steps S401 to S406.

[0104] Step S401: Obtain training sample data.

[0105] Specifically, the training sample data may include the robot's sample observation information at the sample time, the sample history observation information of each of the multiple consecutive sample history times preceding the sample time, and the robot's motion state information label at the sample time. The sample observation information, the sample history observation information, and the motion state information label are the same as those described in the above embodiments, and will not be repeated here.

[0106] The training sample data in this embodiment is used to train the initial walking control model to obtain the walking control model described in the above embodiment.

[0107] Please see Figure 5 The initial walking control model in this embodiment includes a state estimation network 510, a policy network 520, and a value network 530.

[0108] The state estimation network 510 is used to estimate the robot's sample motion state information at a sample time based on a sample time-series observation sequence constructed from sample observation information and the sample historical observation information of multiple consecutive sample historical time points. Its training objective is to make the estimated sample motion state information approximate the motion state information label (i.e., the true motion state information in the simulation environment), so that the trained state estimation network can output reliable motion state information without relying on privileged state information during actual deployment. The motion state information label provides a label basis for the supervised training of the state estimation network, enabling it to learn the mapping relationship between the time-series observation sequence and the motion state information, thus providing data support for obtaining the motion state without relying on privileged state information during actual deployment. The state estimation network 510 can be implemented, for example, using a multilayer perceptron.

[0109] The policy network 520 is used to generate sample control commands for controlling the robot at sample moments based on the input decision information. Its training objective is to maximize the cumulative reward obtained by the robot after executing the generated control commands, thereby enabling the policy network to output control commands that match the decision information and maintain stable robot movement. The policy network 520 may employ, for example, the multilayer perceptron structure described in the above embodiments.

[0110] Value network 530 is used to evaluate the value of sample states and output state value information representing the quality of sample states. Its training objective is to make the output state value information approximate the feedback results obtained after the robot executes control commands, thereby providing an accurate advantage estimate for updating policy network 520. Value network 530 can adopt network structures from related technologies, which will not be elaborated here.

[0111] Step S402: Input the sample observation information and the sample historical observation information of multiple consecutive sample historical moments into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment.

[0112] Specifically, in step S402, the sample observation information and the sample historical observation information of multiple consecutive sample historical moments can be constructed into a sample time-series observation sequence, and this sample time-series observation sequence is input into the state estimation network 510 of the initial walking control model, so as to use the state estimation network 510 to estimate the sample motion state information of the robot at the sample moment. The sample motion state information includes, for example, at least one of the robot's motion speed information, foot contact state information, and foot position information at the sample moment.

[0113] Step S403: Input the sample observation information and sample motion state information into the policy network of the initial walking control model to generate sample control commands for controlling the robot's walking.

[0114] Specifically, based on sample observation information and sample motion state information, the sample state information of the robot at the sample time is determined. Then, the sample state information is input into the policy network 520 to obtain sample control commands for controlling the robot's movement.

[0115] Step S404: Input the sample observation information and motion state information labels into the value network of the initial walking control model to obtain state value information.

[0116] Specifically, based on sample observation information and motion state information labels, the robot's true state information at the sample time is determined. This true state information corresponds to the sample state information estimated in the input of the policy network 520, and it characterizes the robot's true motion state at the sample time. Then, the true state information is input into the value network 530 to obtain state value information. It should be noted that the value network is only used during the training phase, and its value assessment using motion state information labels does not create privileged information dependency during the deployment phase.

[0117] Step S405: Determine the state estimation loss based on the difference between the sample motion state information and the motion state information label; determine the policy loss based on the feedback result obtained after the robot executes the sample control command; and determine the value loss based on the difference between the state value information and the feedback result obtained after the robot executes the sample control command. The feedback result used to determine the policy loss refers to the reward information provided by the simulation environment based on the robot's single-step behavior after executing the sample control command. This reward information is used to characterize the quality of the behavior at the sample moment, and can be set based on factors such as whether the robot remains stable, whether it tracks the command speed, its energy consumption, and whether it falls. The feedback result used to determine the value loss refers to the actual cumulative reward obtained by the robot after executing the sample control command. This actual cumulative reward is the long-term reward obtained by weighting and summing the reward information obtained by the robot at each moment in the subsequent interaction process from the sample moment, using a discount factor, and is used to characterize the long-term performance from the sample moment onwards.

[0118] The state estimation loss can be determined based on the differences between the sample motion state information and the various information items contained in the motion state information label. For example, taking a sample motion state information that includes the robot's motion speed, foot contact state, and foot position information at the sample time, and the motion state information label that includes the robot's actual motion speed, actual foot contact state, and actual foot position information at the sample time, the state estimation loss can be determined based on the differences between the robot's motion speed information at the sample time and the actual motion speed information, the differences between the robot's foot contact state information at the sample time and the actual foot contact state information, and the differences between the robot's foot position information at the sample time and the actual foot position information. The differences between the various information items can be determined using mean squared error or other suitable methods; this embodiment does not limit this approach.

[0119] The policy loss can be determined based on the feedback obtained after the robot executes sample control commands. The feedback result is a feedback signal given by the simulation environment based on the robot's behavior, such as walking speed, posture stability, and energy consumption. For example, the policy loss can be pruned using the Proximal Policy Optimization (PPO) algorithm to limit the policy update range and prevent the policy from collapsing due to excessively large single updates.

[0120] Value loss can be determined based on the difference between the state value information and the feedback results obtained after the robot executes sample control commands.

[0121] It should be noted that the aforementioned strategy loss and value loss can be determined according to the calculation methods in relevant technologies, which will not be elaborated here.

[0122] Step S406: Determine the sample loss based on the state estimation loss, policy loss, and value loss, and update the parameters of the initial walking control model based on the sample loss to obtain the walking control model.

[0123] Specifically, the sample loss is determined based on the state estimation loss, policy loss, and value loss. For example, the sample loss can be obtained by weighted summation of the various loss terms. The weight coefficients of each loss term can be preset or dynamically adjusted during training, depending on actual needs. Then, the parameters of the state estimation network 510, policy network 520, and value network 530 in the initial walking control model are updated based on the sample loss, enabling collaborative optimization among these networks. After the initial walking control model is trained, the aforementioned walking control model can be constructed based on the trained state estimation network 510 and policy network 520. It should be noted that the value network 530 in the initial walking control model is only used during the training phase to evaluate state values ​​to assist in the optimization of the policy network 520; in the actual deployment phase, the value network does not participate in the generation of control commands.

[0124] Through the training method described above, the state estimation network in the initial walking control model learns the ability to estimate motion state information using temporal observation sequences, the policy network learns the ability to generate control commands based on complete state information, and the value network learns the ability to accurately evaluate state value. Upon deployment after training, only the state estimation network and the policy network need to work together. Specifically, the state estimation network outputs the motion state information at the current moment in real time based on the current observation information and the historical observation information corresponding to multiple consecutive historical moments, while the policy network generates control commands based on the current motion state information and the current observation information. Therefore, the walking control model can obtain state support consistent with the training phase in real deployment without relying on difficult-to-obtain privileged state information. This effectively solves the problem of insufficient walking control stability caused by the difficulty in obtaining motion state information and the inconsistency between simulation training and actual deployment states in related technologies, improving the stability of robot walking control in complex terrain environments. Furthermore, because the information used for decision-making integrates sample motion state information derived from sample temporal observation sequences and control commands from the previous moment, the control commands output by the policy network are continuous in time, avoiding gait abrupt changes and jitter during robot walking, thereby improving the stability of robot walking.

[0125] Figure 6The figures show the training curves for those with and without sample motion state information estimation, respectively. Figure 6 It can be seen that, compared with the scheme that does not explicitly estimate motion state information, under the guidance of more specific state quantities (such as the robot's motion speed information, foot state information, etc.), the robot's ineffective trial and error during training is significantly reduced, it can complete the terrain difficulty level improvement faster, and the terrain difficulty level that can be achieved in the end is also higher.

[0126] exist Figure 5 Based on the initial walking control model shown, another embodiment of this application further includes a terrain feature extraction network. This terrain feature extraction network is used to extract features from sample terrain elevation information in the area surrounding the robot, obtaining sample terrain features. These sample terrain features can be used as one of the decision information for the policy network, and are input into the policy network along with sample observation information and sample motion state information, so that the policy network can learn to use terrain information to make walking decisions during the training phase. (See below for reference.) Figure 7 The training process of the initial walking control model according to another embodiment of this application is illustrated by way of example.

[0127] like Figure 7 As shown, the training method for the walking control model according to another embodiment of this application includes steps S701, S702, S707, S7031, S7032, S7041, S705, and S706. Steps S701, S702, and S705 are implemented in the same manner as steps S401, S402, and S405 in the above embodiments, and will not be repeated here.

[0128] Step S701: Obtain training sample data. The training sample data includes the robot's sample observation information at the sample time, the sample historical observation information of each of the multiple consecutive sample historical times before the sample time, and the robot's motion state information label at the sample time.

[0129] Step S702: Input the sample observation information and the sample historical observation information of multiple consecutive sample historical moments into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment.

[0130] Step S707: Obtain sample terrain elevation information of the area surrounding the robot.

[0131] Specifically, the sample terrain elevation information of the area surrounding the robot can be obtained according to the methods described in the above embodiments.

[0132] Step S7031: Use a terrain feature extraction network to extract features from the sample terrain elevation information to obtain the sample terrain features.

[0133] Specifically, a terrain feature extraction network can be used to perform convolutional feature extraction on the sample terrain elevation information to obtain sample terrain texture features. Feature extraction can then be performed on the spatial coordinate information corresponding to the sample terrain elevation information to obtain sample spatial features. Finally, based on the sample terrain texture features and sample spatial features, the sample terrain features are obtained. These sample terrain features contain both texture information representing the terrain undulation structure and orientation information representing the spatial relationship of the terrain, providing a structured description of the terrain surrounding the robot for the policy network.

[0134] Step S7032: Input the sample observation information, sample motion state information and sample terrain features into the policy network to generate sample control instructions.

[0135] Specifically, the robot's state information at the sample moment can be determined first based on sample observation information and sample motion state information. Then, the sample state information and sample terrain features are input into the policy network to obtain sample control commands.

[0136] Compared with the aforementioned embodiments that generate control commands based solely on sample observation information and sample motion state information, this embodiment further incorporates sample terrain features into the decision input of the policy network. This allows the policy network to learn to make decisions based on the combination of the robot's own motion state and the surrounding terrain information during the training phase. For example, when approaching steps, it can adjust the leg lift height in advance based on sample terrain features, and when approaching a slope, it can adjust the body posture and stride based on the terrain slope, thereby obtaining a walking strategy that adapts to complex terrain environments.

[0137] Step S7041: Input the sample observation information, motion state information labels, and sample terrain features into the value network to obtain state value information.

[0138] Specifically, by using sample observation information, motion state information labels, and sample terrain features as inputs to the value network, the value network can obtain a more accurate state value estimate than based solely on sample state information, thereby providing a more reliable advantage evaluation signal for updating the policy network.

[0139] Step S705: Determine the state estimation loss based on the difference between the sample motion state information and the motion state information label; determine the strategy loss based on the feedback result obtained after the robot executes the sample control command; and determine the value loss based on the difference between the state value information and the feedback result obtained after the robot executes the sample control command.

[0140] Step S706: Determine the sample loss based on the state estimation loss, policy loss, and value loss, and update the parameters of the initial walking control model based on the sample loss to obtain the walking control model.

[0141] Specifically, based on the state estimation loss, policy loss, and value loss mentioned above, the sample loss is determined, and the parameters of the state estimation network, terrain feature extraction network, policy network, and value network in the initial walking control model are updated according to the sample loss, thereby obtaining the walking control model.

[0142] Based on the above embodiments, by incorporating sample terrain features into the decision input of the policy network, the policy network learns to make decisions by combining the robot's own motion state with surrounding terrain information during the training phase. It can adjust its gait and foot placement in advance based on the undulations and landing conditions of the surrounding terrain, thereby improving the walking control model's adaptability and success rate in complex terrain environments such as steps and slopes. Furthermore, because the terrain feature extraction network, state estimation network, policy network, and value network are trained collaboratively through joint loss, the terrain feature extraction network can learn to extract effective shape features related to walking decisions, avoiding information loss caused by manually designed terrain features, further improving the model's control performance in complex terrain environments.

[0143] exist Figure 5 Based on the initial walking control model shown, another embodiment of this application further includes a terrain feature extraction network and a reconstruction network. The terrain feature extraction network performs convolutional feature extraction on the sample terrain elevation information to obtain sample terrain texture features, extracts features from the spatial coordinate information corresponding to the sample terrain elevation information to obtain sample spatial features, and obtains sample terrain features based on the sample terrain texture features and sample spatial features. The reconstruction network reconstructs the sample terrain texture features to obtain reconstructed terrain elevation information, and applies reconstruction supervision to the terrain feature extraction network accordingly. (See below for reference.) Figure 8 The training process of the initial walking control model according to another embodiment of this application is described exemplarily.

[0144] First, training sample data is acquired. This data includes the robot's observation information at the sample time, the historical observation information for each of the previous consecutive historical sample times, and the robot's motion state information labels at the sample time.

[0145] Next, the sample observation information and the sample historical observation information of multiple consecutive sample historical moments are input into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment.

[0146] Specifically, the sample observation information and the sample historical observation information corresponding to each of multiple consecutive sample historical moments can be constructed into a sample time series observation sequence and input into the state estimation network 810 in accordance with the method described in the above embodiments, so as to obtain the sample motion state information of the robot at the sample moment.

[0147] Next, obtain sample terrain elevation information of the area surrounding the robot.

[0148] Specifically, the sample terrain elevation information of the area surrounding the robot can be obtained according to the methods described in the above embodiments.

[0149] Next, a terrain feature extraction network is used to extract features from the sample terrain elevation information to obtain the sample terrain features.

[0150] For example, the terrain feature extraction network 840 may include a convolutional neural network 841 and a multilayer perceptron 842. The convolutional neural network 841 and the multilayer perceptron 842 have the same or similar structure and function as the convolutional neural network and multilayer perceptron described in the above embodiments, and will not be repeated here for brevity.

[0151] Specifically, the convolutional neural network 841 in the terrain feature extraction network 840 can be used to extract convolutional features from the sample terrain elevation information to obtain sample terrain texture features. These sample terrain texture features are used to characterize the undulating texture pattern of the terrain surrounding the robot. Additionally, the multilayer perceptron 842 in the terrain feature extraction network 840 is used to extract features from the spatial coordinate information corresponding to the sample terrain elevation information to obtain sample spatial features. These sample spatial features are used to characterize the spatial positional relationship of the sample terrain grid. Then, the sample terrain texture features and sample spatial features are concatenated along the channel dimension (…). Figure 8 Sino-Israeli symbols (This refers to a splicing operation), which yields sample terrain features. These features contain both texture information representing the terrain's undulations and orientation information representing its spatial relationships, providing the policy network 820 with a structured description of the terrain surrounding the robot.

[0152] Next, the sample observation information, sample motion state information, and sample terrain features are input into the policy network to generate sample control instructions.

[0153] Specifically, the robot's sample state information at the sample moment can be determined based on the sample observation information and sample motion state information, as described in the above embodiments. The sample state information and sample terrain features are then input into the policy network 820 to obtain the robot's sample control command at the sample moment.

[0154] Next, the sample observation information, motion state information labels, and sample terrain features are input into the value network to obtain state value information.

[0155] Specifically, based on sample observation information and motion state information labels, the robot's true state information at the sample time is determined. This true state information corresponds to the sample state information estimated in the policy network input, and it characterizes the robot's true motion state at the sample time. Then, the true state information and sample terrain features are input into the value network 830 to obtain state value information, enabling the value network 830 to perform more accurate value assessments during the training phase using the true state values.

[0156] Next, the sample terrain texture features are input into the reconstruction network 850 to obtain the reconstructed terrain elevation information.

[0157] Specifically, the reconstruction network 850 can be implemented using a deconvolutional structure symmetrical to the convolutional neural network 841. For example, the convolutional neural network 841 includes an input layer, multiple convolutional feature extraction layers, and an output layer connected in sequence. Each convolutional feature extraction layer includes at least one convolutional layer and a subsequent ReLU nonlinear activation layer, used to sequentially extract convolutional features from the sample terrain elevation information, ultimately outputting sample terrain texture features. Correspondingly, the reconstruction network 850 includes an input layer, multiple terrain reconstruction layers, and an output layer connected in sequence. Each terrain reconstruction layer includes at least one deconvolutional layer (transposed convolutional layer) and a subsequent ReLU nonlinear activation layer. The input layer of the reconstruction network 850 receives the sample terrain texture features output by the convolutional neural network 841, the multiple terrain reconstruction layers sequentially perform deconvolutional feature reconstruction on the sample terrain texture features, and finally, the output layer outputs reconstructed terrain elevation information with the same size as the sample terrain elevation information.

[0158] Next, based on the difference between the sample motion state information and the motion state information label, the state estimation loss is determined; based on the difference between the reconstructed terrain elevation information and the sample terrain elevation information, the terrain reconstruction loss is determined; based on the feedback results obtained after the robot executes the sample control commands, the policy loss is determined; and based on the difference between the state value information and the feedback results obtained after the robot executes the sample control commands, the value loss is determined.

[0159] Specifically, the state estimation loss, policy loss, and value loss can be determined in the manner described in the above embodiments, and will not be repeated here. The terrain reconstruction loss is determined based on the difference (e.g., mean squared error) between the reconstructed terrain elevation information and the sample terrain elevation information. The above-mentioned terrain reconstruction loss uses terrain reconstruction as auxiliary supervision to constrain the terrain texture features extracted by the terrain feature extraction network 840 to fully retain the geometric structure information of the original terrain, providing high-quality terrain feature input for the walking decision of the policy network 820.

[0160] Next, based on the state estimation loss, terrain reconstruction loss, policy loss, and value loss, the sample loss is determined, and the parameters of the initial walking control model are updated based on the sample loss to obtain the walking control model.

[0161] Specifically, the state estimation loss, terrain reconstruction loss, policy loss, and value loss can be weighted and summed according to preset weight coefficients to determine the sample loss. Based on this sample loss, the parameters of the state estimation network 810, policy network 820, value network 830, terrain feature extraction network 840, and reconstruction network 850 in the initial walking control model are updated to obtain the walking control model.

[0162] It should be noted that, similar to the value network 830, the reconstruction network 850 is only used to assist in optimizing the terrain feature extraction network 840 and the policy network 820 during the training phase. It does not participate in the generation of control instructions during the deployment phase, thus avoiding the introduction of additional deployment computation overhead.

[0163] Based on the above embodiments, on the one hand, the policy network learns and combines terrain information to make walking decisions, adjusting gait and foot placement according to the undulations and landing conditions of the terrain around the robot, thus improving the walking control model's adaptability to complex terrains such as steps and slopes. On the other hand, the terrain reconstruction loss uses the sample terrain elevation information itself as the supervision target, constraining the sample terrain texture features to fully preserve the terrain geometric structure information through self-supervision, providing the policy network with high-quality terrain feature basis, thereby indirectly improving the walking control model's walking control performance in complex terrains. Simultaneously, the state estimation network, value network, and the aforementioned networks are updated collaboratively, enabling the walking control model to obtain complete motion state support without relying on privileged state information during deployment, further ensuring the stability of walking on complex terrains.

[0164] Figure 9 A schematic diagram illustrating the comparison of walking control effects combining terrain reconstruction and motion state estimation is provided. Please refer to... Figure 9 , Figure 9Figures 91 and 92 illustrate the robot's walking performance during stair climbing using the "combined terrain reconstruction and motion state estimation" scheme and the "no terrain reconstruction and motion state estimation" scheme, respectively. Scheme 91 simultaneously incorporates motion state information (such as the robot's linear velocity) estimation based on sample time-series observation sequences and terrain reconstruction-assisted supervision during the training phase. Comparing Figures 91 and 92 shows that after training with both mechanisms simultaneously introduced, the robot can promptly lift its foot and select an accurate landing position regardless of whether it faces the first or last step, avoiding collisions with the steps throughout the climb (e.g., ...). Figure 9 (As shown in Figure 91). Robots not trained using the aforementioned mechanism collided both when initially climbing the stairs and when facing the last step (as shown in Figure 91). Figure 9 (As shown in Figure 92). It can be seen that terrain reconstruction-assisted supervision can enable terrain features to fully retain the geometric structure information of the real terrain, providing higher quality terrain feature input for policy network decision-making. Combined with the motion state information provided by the state estimation network, it can significantly improve the robot's ability to pass through and its walking safety on terrain with sudden height changes such as steps.

[0165] exist Figure 5 Based on the initial walking control model shown, another embodiment of this application further includes a terrain feature extraction network and an attention encoding module. The terrain feature extraction network extracts features from the sample terrain elevation information of the area surrounding the robot to obtain sample terrain features. The attention encoding module encodes the robot's sample state information at the sample time with the sample terrain features to obtain first sample attention features. These first sample attention features can be used as input to a policy network, enabling the policy network to filter key terrain information related to the robot's sample state at the sample time for decision-making.

[0166] The training process of the initial walking control model in another embodiment of this application is described below by way of example.

[0167] First, training sample data is acquired. This data includes the robot's observation information at the sample time, the historical observation information for each of the previous consecutive historical sample times, and the robot's motion state information labels at the sample time.

[0168] Next, the sample observation information and the sample historical observation information of multiple consecutive sample historical moments are input into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment.

[0169] Next, obtain sample terrain elevation information of the area surrounding the robot.

[0170] Next, a terrain feature extraction network is used to extract features from the sample terrain elevation information to obtain the sample terrain features.

[0171] Next, based on the sample observation information and sample motion state information, the sample state information of the robot at the sample time is determined. The attention encoding module is used to encode the sample state information and sample terrain features to obtain the first sample attention feature. The first sample attention feature is then input into the policy network to generate sample control instructions.

[0172] Next, based on the sample observation information and motion state information labels, the robot's true state information at the sample time is determined. The attention encoding module is used to encode the true state information and the sample terrain features to obtain the second sample attention features. The second sample attention features are then input into the value network to obtain the state value information.

[0173] Next, based on the difference between the sample motion state information and the motion state information label, the state estimation loss is determined; based on the feedback results obtained after the robot executes the sample control instructions, the policy loss is determined; and based on the difference between the state value information and the feedback results obtained after the robot executes the sample control instructions, the value loss is determined.

[0174] Next, based on the state estimation loss, policy loss, and value loss, the sample loss is determined, and the parameters of the initial walking control model are updated based on the sample loss to obtain the walking control model.

[0175] Specifically, the state estimation loss, policy loss, and value loss can be weighted and summed according to preset weight coefficients to obtain the sample loss. The parameters of the state estimation network, policy network, value network, terrain feature extraction network, and attention encoding module in the initial walking control model are then updated based on the sample loss to obtain the walking control model.

[0176] Compared to Figure 5 In another embodiment of this application, as shown in the example, the initial walking control model further incorporates a terrain feature extraction network and an attention encoding module. Through terrain feature extraction and attention encoding, the policy network can adaptively utilize key terrain information to make decisions based on its own motion state, improving the model's walking adaptability and safety in complex terrains such as steps and slopes. The value network obtains a more accurate value estimate through state truth-based attention encoding, providing a higher-quality advantage signal for policy updates.

[0177] exist Figure 8Based on the illustrated embodiment, another embodiment of the initial walking control model of this application further includes an attention encoding module. This attention encoding module is used to encode the robot's sample state information and sample terrain features at a sample time to obtain a first sample attention feature. This first sample attention feature can be used as input to a policy network, enabling the policy network to filter out key terrain information related to the robot's sample state at the sample time for decision-making.

[0178] The following is for reference. Figure 10 The training process of the initial walking control model in another embodiment of this application will be described exemplarily.

[0179] First, training sample data is acquired. This data includes the robot's observation information at the sample time, the historical observation information for each of the previous consecutive historical sample times, and the robot's motion state information labels at the sample time.

[0180] Next, the sample observation information and the sample historical observation information of multiple consecutive sample historical moments are input into the state estimation network 1010 of the initial walking control model to obtain the sample motion state information of the robot at the sample moment.

[0181] Next, obtain sample terrain elevation information of the area surrounding the robot.

[0182] Next, the terrain feature extraction network 1040 is used to extract features from the sample terrain elevation information to obtain the sample terrain features. Specifically, this can be done according to... Figure 8 The method used in the illustrated embodiment to determine the sample terrain features will not be described in detail here.

[0183] Next, based on the sample observation information and sample motion state information, the sample state information of the robot at the sample time is determined. The attention encoding module 1060 is used to encode the sample state information and sample terrain features to obtain the first sample attention feature. The first sample attention feature is then input into the policy network 1020 to generate sample control instructions.

[0184] Next, based on the sample observation information and motion state information labels, the robot's true state information at the sample time is determined. The attention encoding module 1060 is used to encode the true state information and the sample terrain features to obtain the second sample attention features. The second sample attention features are then input into the value network 1030 to obtain the state value information.

[0185] Next, the sample terrain texture features are input into the reconstruction network 1050 to obtain the reconstructed terrain elevation information.

[0186] Next, based on the difference between the sample motion state information and the motion state information label, the state estimation loss is determined; based on the difference between the reconstructed terrain elevation information and the sample terrain elevation information, the terrain reconstruction loss is determined; based on the feedback results obtained after the robot executes the sample control commands, the policy loss is determined; and based on the difference between the state value information and the feedback results obtained after the robot executes the sample control commands, the value loss is determined.

[0187] Next, based on the state estimation loss, terrain reconstruction loss, policy loss, and value loss, the sample loss is determined, and the parameters of the initial walking control model are updated based on the sample loss to obtain the walking control model.

[0188] Specifically, the state estimation loss, terrain reconstruction loss, policy loss, and value loss can be weighted and summed according to preset weight coefficients to obtain the sample loss. The parameters of the state estimation network 1010, policy network 1020, value network 1030, terrain feature extraction network 1040, reconstruction network 1050, and attention encoding module 1060 in the initial walking control model are updated based on the sample loss to obtain the walking control model.

[0189] Compared to Figure 8 In another embodiment of this application, as shown in the example, the initial walking control model further incorporates an attention encoding module. On one hand, this attention encoding module performs attention encoding on sample state information and sample terrain features, and uses the resulting first sample attention features as input to the policy network, enabling the policy network to make decisions based on key terrain information selected through the attention mechanism. On the other hand, the attention encoding module performs attention encoding on real state information and sample terrain features to obtain second sample attention features, which are then input into the value network. This allows the value network to perform value estimation based on the second sample attention features encoded with real state information, thereby obtaining a more accurate value estimate and providing the policy network with a more reliable advantage update signal.

[0190] To verify the effectiveness of the training scheme for the aforementioned walking control model, this application embodiment conducted a stair climbing comparison experiment in a simulation environment: the robot continuously climbed three steps (step height 0.15m, depth 0.30m, width 0.8m) at a commanded speed of, for example, 0.4m / s, and the test was repeated 20 times. The success criterion was whether the robot could complete the entire stair climb; any situation where the robot collided with a step and missed a step or fell down the stairs was considered a failure.

[0191] The robot employing the scheme of this application (including terrain perception, attention encoding, and terrain reconstruction-assisted supervision) can generally complete stair climbing stably. In 20 experiments, collisions occurred only 2 times, with a stair collision rate of 10%. Furthermore, after a collision, the robot can quickly adjust its footing and complete the stair climbing action (e.g., ...). Figure 11 As shown in Figure 111), the climbing success rate reached 100%. However, the robot that did not adopt the solution of this application had insufficient leg lifting height, frequently scraped the front edge of the steps, and collided 15 times in 20 experiments. Furthermore, it exhibited misstepping behavior after colliding with the stairs (such as...). Figure 11 As shown in Figure 112, the robot fell down the stairs, with a climbing success rate of only 60%. Experiments show that the technical solution of this application enables the robot to dynamically monitor the edge of the steps and safe footholds, significantly improving the safety and success rate of traversing complex terrain.

[0192] In some other embodiments, the step of inputting the first sample attention features into the policy network to generate sample control instructions includes: inputting the first sample attention features and sample state information into the policy network to generate sample control instructions. Correspondingly, inputting sample observation information and motion state information labels into the value network of the initial walking control model to obtain state value information includes: determining the robot's true state information at the sample time based on the sample observation information and motion state information labels; using an attention encoding module to perform attention encoding on the true state information and sample terrain features to obtain second sample attention features; and inputting the second sample attention features and true state information into the value network to obtain state value information.

[0193] In the above embodiments, by using the attention features and state information of the first sample as input to the policy network, the policy network can directly perceive the robot's own motion state while utilizing terrain information, avoiding the loss of ontological state information during attention encoding. This allows for the generation of control commands that match both the robot's current actual motion state and the terrain environment. Correspondingly, the value network performs value evaluation based on the real state information and the attention features of the second sample. Since the real state information contains real motion state information, the value network can evaluate the state value based on its understanding of the robot's real motion state, obtaining a more accurate value estimate. This provides a more reliable advantage signal for policy network updates, indirectly improving the convergence speed and walking control effect of the policy network.

[0194] Based on the embodiments of the training methods for the aforementioned walking control models, the walking control model training method of this application can further introduce a data augmentation mechanism to achieve mirror-symmetric constraint training. It is understood that this data augmentation mechanism does not depend on the specific network structure or specific input information in the foregoing embodiments, but can be extended to any of the aforementioned walking control model training method embodiments.

[0195] Specifically, regardless of whether the initial walking control model includes a terrain feature extraction network, an attention encoding module, or a reconstruction network, the input information related to the robot's body state in the training sample data can be mirrored using the robot's plane of symmetry as a reference. This constructs mirrored training sample data, and the policy network is constrained to output mirrored control commands in mirrored states. For example, when the input of the policy network in the training method embodiment only includes sample observation information and sample motion state information, the object of the mirror transformation is the sample observation information and the sample motion state information. When the input of the policy network in the training method embodiment also includes terrain perception input aligned with the robot's coordinate system (e.g., sample terrain elevation information), the object of the mirror transformation can be extended to this type of terrain information, ensuring that the mirrored terrain information and the mirrored robot state are spatially consistent. Thus, the above data augmentation mechanism can be arbitrarily combined with the training processes described in the aforementioned embodiments (including the training of the state estimation network, policy network, value network, terrain feature extraction network, reconstruction network, attention encoding module, etc., as well as the determination and parameter updating of each loss term). For ease of understanding, the following will use... Figure 10 The training method shown in the example introduces a method that incorporates mirror-symmetric data augmentation.

[0196] In this implementation, the robot, for example, is a humanoid robot with a body structure exhibiting significant left-right symmetry. Correspondingly, its walking gait should also possess left-right symmetry. However, if the training sample distribution is limited and no symmetry constraint is imposed on the policy output, the policy network may learn unstable or asymmetrical gaits. Therefore, a mirror transformation is performed based on the robot's plane of symmetry, which is, for example, the plane in the robot's body coordinate system formed by the forward direction axis (x-axis) and the vertical direction axis (z-axis) (see [link]). Figure 12 The x-axis represents the forward direction, the y-axis represents the left-right direction, and the z-axis represents the vertical upward direction. Specifically, the sample observation information, sample motion state information, and sample terrain elevation information can be mirrored to obtain mirrored observation information, mirrored motion state information, and mirrored terrain elevation information.

[0197] It should be noted that the mirror transformation follows transformation rules corresponding to the robot's left-right symmetry: for paired state variables, such as joint positions and velocities of symmetrical joints, they should be interchanged after mirroring; for symmetrical variables such as linear velocity components along the lateral (y-axis) and the projection of gravity onto the y-axis, they should be inverted after mirroring; for angular velocity components around the x-axis and z-axis, they should also be inverted after mirroring; while asymmetrical state variables remain unchanged. For the sample terrain elevation information, the mirror transformation is manifested as flipping the elevation map with the plane of symmetry as a mirror, that is, inverting the y-coordinates of each grid point in the elevation map (while keeping the x-coordinates and height values ​​unchanged), so that the position of any point in the mirrored terrain matches the robot's body posture after mirroring.

[0198] Next, a terrain feature extraction network is used to extract features from the mirrored terrain elevation information to obtain mirrored terrain features.

[0199] Next, the mirror observation information, mirror motion state information, and mirror terrain features are input into the policy network to generate mirror control commands for controlling the robot's movement.

[0200] It should be noted that when the implementation also includes an attention encoding module, to ensure consistency between the processing links of the mirror branch and the forward branch, the mirror state information can first be determined from the mirror observation information and the mirror motion state information. Then, using the same attention encoding module, the mirror state information is mapped to a query vector, and the mirror terrain features are mapped to key and value vectors. Attention encoding is then performed on both to obtain the mirror attention features. Subsequently, following the input method of the policy network in the implementation, the mirror attention features are input into the policy network to generate mirror control commands for controlling the robot's movement.

[0201] In other embodiments, when the input information of the policy network in the based embodiment includes state information in addition to the attention encoding result, the mirror attention features and mirror state information can be input into the policy network together to generate mirror control commands for controlling the robot's movement. Accordingly, when determining the symmetry loss, the compared mirror control commands should be based on the same policy network input as the forward branch to ensure that the symmetry constraint relationship between the mirror branch and the forward branch is consistent.

[0202] Next, the symmetry loss is determined based on the difference between the mirror control command and the mirror sample control command obtained after mirror transformation of the sample control command.

[0203] Specifically, the original sample control commands are mirrored using the same mirror transformation method as described above to obtain mirrored sample control commands. Then, based on the difference between the mirrored control commands and the mirrored sample control commands (e.g., mean squared error), a symmetric loss is determined. This symmetric loss is used to constrain the policy network to output mirrored actions in mirrored states, enabling the policy network to learn left-right symmetrical gait patterns.

[0204] Next, the sample loss is determined based on the symmetry loss, terrain reconstruction loss, state estimation loss, policy loss, and value loss. The parameters of the initial walking control model are then updated based on the sample loss to obtain the walking control model.

[0205] Specifically, when determining the sample loss, the aforementioned symmetric loss is weighted and summed with the state estimation loss, policy loss, and value loss (which are also included if the adopted embodiment also includes terrain reconstruction loss) determined according to the adopted embodiment, according to preset weight coefficients, to obtain the sample loss. With the goal of reducing the sample loss, the parameters of the corresponding networks in the initial walking control model (including the state estimation network, policy network, value network, and the adopted terrain feature extraction network, attention encoding module, reconstruction network, etc.) are simultaneously updated through backpropagation to achieve collaborative optimization of each module, thereby obtaining the walking control model.

[0206] Based on the aforementioned data augmentation mechanism, this embodiment uses the robot's left-right symmetry as a priori. Without any additional interaction with the environment, mirror sample data can be expanded from existing training sample data, broadening the distribution of training data and improving its utilization. Furthermore, by constraining the policy network with symmetric loss to output symmetrical control actions in mirror states, the robot learns a gait with good left-right symmetry. This suppresses yaw during walking, enabling the robot to walk more straight in the command direction and reducing the probability of unstable gait. This improves the generalization ability and walking stability of the policy network under different movement directions and terrain conditions. Simultaneously, the mirror transformation only requires algebraic transformation of existing samples, resulting in low computational overhead and no increase in training time, which is beneficial for efficient convergence of reinforcement learning training.

[0207] To verify the actual effect of combining state estimation based on time-series observation sequences with mirror symmetry constraints in the training method of this application embodiment, a straight-line walking comparison experiment was conducted in a simulation environment.

[0208] Specifically, the walking control model used in this embodiment includes a state estimation network and a policy network. Correspondingly, during the training phase, the initial walking control model also includes a value network; that is, the initial walking control model includes a state estimation network, a policy network, and a value network. During the model training phase, in addition to constructing the state estimation loss, policy loss, and value loss, a mirror symmetry constraint is introduced. Mirror training sample data is constructed by performing a mirror transformation on the training sample data (including sample observation information at the sample time and the sample historical observation information of multiple consecutive historical sample times before the sample time). The mirror training sample data is then used to train the model under mirror symmetry constraints to obtain the symmetry loss. The sample loss is determined based on this symmetry loss, the state estimation loss, the policy loss, and the value loss, and the parameters of the corresponding initial walking control model (including the state estimation network, policy network, and value network) are updated based on the sample loss.

[0209] In this comparative experiment, the robot's most recent five frame observations (including the current observation information and the historical observation information corresponding to the four consecutive historical moments preceding the current moment) were constructed into a temporal observation sequence. This temporal observation sequence was then input into the state estimation network of the aforementioned walking control model. The state estimation network was used to estimate the robot's motion state information at the current moment, thereby providing a reliable motion state prior for the policy network. This allows the policy network to obtain the aforementioned state quantities without relying on additional environmental perception equipment during the deployment phase. The policy network then generates control commands for controlling the robot's walking based on the current observation information and motion state information at the current moment.

[0210] Please see Figure 13 , Figure 13 The diagram illustrates a comparison of the trajectories of a robot walking along a straight line. The solid line represents the trajectory of the technical solution of this application embodiment (which employs both state estimation and mirror symmetry constraint training), while the dashed line represents the trajectory of the control scheme without the introduction of state estimation and symmetry loss.

[0211] Specifically, the robot performs a straight-line walking task at a constant commanded linear velocity v_x = 0.6 m / s, with a target walking distance D = 5 m. The robot's starting position is (x_0, y_0) = (0.0, 0.0) m, and the target position is 5 m ahead of the robot, i.e., (x_target, y_target) = (5.0, 0.0) m. The final test endpoint position using the technical solution of this application embodiment is (5.0, 0.1216) m, while the minimum offset endpoint position of the robot using the control scheme is (5.0, 0.2355) m. Compared to the control scheme, the lateral offset of the robot using the technical solution of this application embodiment is reduced by approximately 48.36%. Repeated verification after multiple walks shows that after adopting the technical solution of this application embodiment, the lateral offset of the robot is reduced by approximately 25% to 50%, which is significantly better than the control scheme without the above mechanism.

[0212] Furthermore, Figure 14 The diagram illustrates the foot placement of the robot during its first five steps as it walks in a straight line. (See also...) Figure 14 The solid-lined boxes represent the landing positions of the technical solutions in this application embodiment, while the dashed-lined boxes represent the landing positions of the comparative solutions. The robot moves forward at a speed of v_x = 0.6 m / s. In the first five steps after the robot starts moving, the landing position of the fifth step using the solution in this application embodiment is (1.1363, 0.1500) m, while the landing position of the fifth step using the comparative solution is (1.1127, 0.1761) m. It can be seen that when using the solution in this application embodiment, the difference in left and right step distances is smaller, and the mirror symmetry is better. Figure 13 The results, showing a straighter mid-trajectory and a smaller lateral offset at the endpoint, corroborate each other.

[0213] comprehensive Figure 13 and Figure 14 The results show that motion state information estimation based on time-series observation sequences improves the accuracy of the robot's motion state perception, enabling it to plan actions based on accurate motion state information when walking in a straight line along the command direction. Mirror symmetry constraints guide the policy network to generate left-right symmetrical gait during the training phase, reducing the inconsistency in step distance between the left and right sides during actual walking. The combined effect of these two factors significantly reduces the robot's yaw during straight-line walking, makes the endpoint closer to the target position, and improves the symmetry of the first few steps, thus exhibiting superior stability and straight-line accuracy in straight-line walking scenarios. This lays the foundation for subsequent accurate tracking and foot planning in complex terrain.

[0214] Based on the same inventive concept, such as Figure 15As shown, an embodiment of this application also provides a robot walking control system, which can be integrated into the robot. The robot walking control system includes: a first acquisition module 1501, a first determination module 1502, and a first generation module 1503.

[0215] The first acquisition module 1501 is used to acquire the robot's current observation information at the current moment and the historical observation information of multiple consecutive historical moments before the current moment.

[0216] The first determining module 1502 is used to determine the robot's motion state information at the current moment based on historical observation information and current observation information.

[0217] The first generation module 1503 is used to generate control commands for controlling the robot's movement based on the current observation information and motion state information.

[0218] In some embodiments, current observation information includes at least one of the robot's body state information at the current moment, the instruction information received by the robot at the current moment, and the control instruction from the previous moment; and / or, historical observation information includes at least one of the robot's body state information at a historical moment, the instruction information received by the robot at a historical moment, and the control instruction from the previous moment; and / or, motion state information includes at least one of the robot's motion speed information and foot end state information.

[0219] In some embodiments, the robot's walking control system further includes a second acquisition module for acquiring terrain elevation information of the area surrounding the robot; the first generation module 1503 is also used to generate control commands based on the current observation information, the current motion state information and the terrain elevation information.

[0220] In some embodiments, the first generation module 1503 is further configured to determine the current state information of the robot based on the current observation information and motion state information; extract terrain features based on terrain elevation information; perform attention encoding on the current state information and terrain features to obtain attention features; and generate control instructions based on the attention features.

[0221] In some embodiments, the first generation module 1503 is further configured to perform convolutional feature extraction on the terrain elevation information to obtain terrain texture features; and to perform feature extraction on the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features; and to obtain terrain features based on the terrain texture features and spatial features.

[0222] In some embodiments, the first generation module 1503 is further configured to map the current state information into a query vector, map the terrain features into a key vector and a value vector respectively; calculate the attention weight based on the query vector and the key vector, and perform weighted aggregation on the value vector based on the attention weight to obtain the attention features.

[0223] In some embodiments, the first generation module 1503 is further configured to generate control instructions based on attention characteristics and current state information.

[0224] In some embodiments, the first determining module 1502 is further configured to construct a time-series observation sequence from the historical observation information and current observation information of multiple consecutive historical moments; input the time-series observation sequence into the state estimation network of a pre-built walking control model to obtain motion state information; and / or, the first generating module 1503 is further configured to input the current observation information and motion state information into the policy network of the walking control model to generate control commands for controlling the robot to walk.

[0225] In some embodiments, the robot's walking control system further includes: a third acquisition module, used to acquire training sample data, the training sample data including the robot's sample observation information at the sample time, the sample history observation information of each of the multiple consecutive sample history times before the sample time, and the motion state information label of the robot at the sample time; and an update module, used to update the parameters of the initial walking control model according to the sample observation information, the sample history observation information of each of the multiple consecutive sample history times, and the motion state information label, so as to obtain the walking control model.

[0226] In some embodiments, the update module includes: an input unit, used to input sample observation information and sample historical observation information of each of multiple consecutive sample historical moments into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample moment; and an update unit, used to update the parameters of the initial walking control model according to the sample observation information, sample motion state information and motion state information labels to obtain the walking control model.

[0227] In some embodiments, the robot's walking control system further includes a fourth acquisition module for acquiring sample terrain elevation information of the area surrounding the robot; the update unit is further configured to update the parameters of the initial walking control model based on sample observation information, sample motion state information, sample terrain elevation information and motion state information labels to obtain the walking control model.

[0228] In some embodiments, the updating unit is further configured to: input sample observation information and sample motion state information into the policy network of the initial walking control model to generate sample control commands for controlling robot walking; input sample observation information and motion state information labels into the value network of the initial walking control model to obtain state value information; determine state estimation loss based on the difference between sample motion state information and motion state information labels; determine policy loss based on the feedback result obtained after the robot executes the sample control commands; and determine value loss based on the difference between state value information and the feedback result obtained after the robot executes the sample control commands; determine sample loss based on state estimation loss, policy loss, and value loss; and update the parameters of the initial walking control model based on sample loss to obtain the walking control model.

[0229] In some embodiments, the initial walking control model further includes a terrain feature extraction network; the updating unit is further configured to: extract features from the sample terrain elevation information using the terrain feature extraction network to obtain sample terrain features; input sample observation information, sample motion state information, and sample terrain features into the policy network of the initial walking control model to generate sample control commands for controlling robot walking; input sample observation information, motion state information labels, and sample terrain features into the value network of the initial walking control model to obtain state value information; determine state estimation loss based on the difference between sample motion state information and motion state information labels; determine policy loss based on the feedback results obtained after the robot executes the sample control commands; and determine value loss based on the difference between state value information and the feedback results obtained after the robot executes the sample control commands; determine sample loss based on state estimation loss, policy loss, and value loss; and update the parameters of the initial walking control model based on sample loss to obtain the walking control model.

[0230] In some embodiments, the updating unit is further configured to: use a terrain feature extraction network to perform convolutional feature extraction on the sample terrain elevation information to obtain sample terrain texture features; perform feature extraction on the spatial coordinate information corresponding to the sample terrain elevation information to obtain sample spatial features; and obtain sample terrain features based on the sample terrain texture features and sample spatial features; the initial walking control model further includes a reconstruction network; the updating unit is further configured to: input the sample terrain texture features into the reconstruction network to obtain reconstructed terrain elevation information; determine the terrain reconstruction loss based on the difference between the reconstructed terrain elevation information and the sample terrain elevation information; and determine the sample loss based on the terrain reconstruction loss, state estimation loss, policy loss, and value loss.

[0231] In some embodiments, the initial walking control model further includes an attention encoding module; the update unit is further configured to: determine the sample state information of the robot at the sample time based on the sample observation information and the sample motion state information; use the attention encoding module to perform attention encoding on the sample state information and the sample terrain features to obtain a first sample attention feature; input the first sample attention feature into the policy network to generate sample control instructions; and determine the true state information of the robot at the sample time based on the sample observation information and the motion state information label; use the attention encoding module to perform attention encoding on the true state information and the sample terrain features to obtain a second sample attention feature; and input the second sample attention feature into the value network to obtain state value information.

[0232] In some embodiments, the updating unit is further configured to: input the first sample attention features and sample state information into the policy network to generate sample control instructions; and input the second sample attention features and real state information into the value network to obtain state value information.

[0233] In some embodiments, the robot's walking control system further includes a mirror transformation module, used to perform mirror transformations on sample observation information, sample motion state information, and sample terrain elevation information based on the robot's plane of symmetry, to obtain mirror observation information, mirror motion state information, and mirror terrain elevation information; an extraction module, used to extract features from the mirror terrain elevation information using a terrain feature extraction network to obtain mirror terrain features; a second generation module, used to input the mirror observation information, mirror motion state information, and mirror terrain features into a policy network to generate mirror control commands for controlling the robot's walking; and a second determination module, used to determine the symmetry loss based on the difference between the mirror control commands and the mirror sample control commands obtained after mirror transformation of the sample control commands; wherein, the update unit is further used to determine the sample loss based on the symmetry loss, terrain reconstruction loss, state estimation loss, policy loss, and value loss.

[0234] Each module or unit in the above systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the robot's processor in hardware form or independent of it, or stored in the robot's memory in software form, so that the processor can call and execute the operations corresponding to each module.

[0235] In some embodiments, a robot is also provided. See also Figure 16The robot 1600 includes a memory 1601 and a processor 1602. The memory 1601 stores a computer program, which, when executed by the processor 1602, implements the steps described in the above-described method embodiments. The robot can be, for example, a humanoid robot, a quadruped robot, or other legged mobile robots suitable for the present application. The robot may also include sensing components (including but not limited to depth cameras, lidar, inertial measurement units, etc.) and execution components (e.g., drive motors for each joint). The sensing components are used to collect observation information and terrain elevation information, and the execution components are used to drive the robot to walk according to control commands.

[0236] Of course, in practical applications, such as Figure 16 As shown, the robot 1600 may further include at least one network interface 1603. Various components of the robot are coupled together via a bus system 1604. It is understood that the bus system 1604 is used to implement communication between these components. In addition to a data bus, the bus system 1604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 16Various buses are labeled as bus system 1604. The number of processors 1602 can be at least one. Network interface 1603 is used for wired or wireless communication between the robot and other devices. Memory 1601 in this embodiment is used to store various types of data to support robot operation. The methods disclosed in the above embodiments can be applied to processor 1602, or implemented by processor 1602. Processor 1602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 1602 or by instructions in software form. Processor 1602 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected in the combined execution of hardware and software modules in a microcontroller. The software module can reside in a storage medium located in memory 1601. Processor 1602 reads information from memory 1601 and, in conjunction with its hardware, completes the steps of the aforementioned method. In an exemplary embodiment, robot 1600 can be implemented using one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontroller units (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0237] This application also provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute and implement the steps in the above-described method embodiments. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0238] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the above-described method embodiments.

[0239] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), disks, or optical disks.

[0240] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, disks, or optical discs.

[0241] It should be noted that, in the data processing stage, the technical solution of this application strictly limits the scope of data collection to the minimum necessary to achieve the technical objectives, preventing the acquisition of irrelevant information. For any user information to be collected, the data subject will be clearly informed and their consent obtained. Furthermore, technologies such as encrypted storage and access control are employed to strengthen data security and ensure the security and compliance of the entire data processing process. The technical model and decision-making mechanism are based on objective technical parameters and do not introduce unnecessary parameters such as gender or age that may lead to discrimination, resolutely eliminating algorithmic discrimination and upholding public order and good morals. In addition, the specification fully describes the technical implementation methods, application scenarios, and compliance protection details. The claims are consistent with the content of the specification, key compliance designs are clear and verifiable, and the overall technical design is guided by the protection of public interests and adherence to social ethics, without any circumstances that harm public interests or violate public order and good morals.

[0242] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0243] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed the preferred embodiments as described above, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A method for controlling the walking of a robot, characterized in that, include: Obtain the robot's current observation information at the current moment and the historical observation information of each of the multiple consecutive historical moments before the current moment; Based on the historical observation information of each of the consecutive historical moments and the current observation information, the motion state information of the robot at the current moment is determined; The terrain elevation information of the area surrounding the robot is obtained, and control commands for controlling the robot's movement are generated based on the current observation information, the motion state information, and the terrain elevation information.

2. The method according to claim 1, characterized in that, The current observation information includes at least one of the robot's current body state information, the instruction information received by the robot at the current time, and the control instruction from the previous time; and / or, The historical observation information includes at least one of the robot's body state information at the historical moment, the instruction information received by the robot at the historical moment, and the control instruction from the previous moment in the historical moment; and / or, The motion state information includes at least one of the robot's motion speed information and foot end state information.

3. The method according to claim 1, characterized in that, The step of determining the robot's motion state information at the current moment based on the historical observation information of each of the consecutive historical moments and the current observation information includes: constructing a time-series observation sequence from the historical observation information of each of the consecutive historical moments and the current observation information; inputting the time-series observation sequence into the state estimation network of a pre-built walking control model to obtain the motion state information; and / or, The step of generating control commands for controlling the robot's movement based on the current observation information, the motion state information, and the terrain elevation information includes: extracting terrain features based on the terrain elevation information; inputting the current observation information, the motion state information, and the terrain features into the policy network of the walking control model to generate control commands for controlling the robot's movement.

4. The method according to claim 1, characterized in that, The step of generating the control command based on the current observation information, the motion state information, and the terrain elevation information includes: Based on the current observation information and the motion state information, the current state information of the robot is determined; Extract terrain features based on the terrain elevation information; Attention features are obtained by performing attention encoding on the current state information and the terrain features; The control instructions are generated based on the attention characteristics.

5. The method according to claim 4, characterized in that, The step of extracting terrain features based on the terrain elevation information includes: performing convolutional feature extraction on the terrain elevation information to obtain terrain texture features; and performing feature extraction on the spatial coordinate information corresponding to the terrain elevation information to obtain spatial features; obtaining the terrain features based on the terrain texture features and the spatial features; and / or, The step of performing attention encoding on the current state information and the terrain features to obtain attention features includes: mapping the current state information to a query vector, and mapping the terrain features to key vectors and value vectors respectively; calculating attention weights based on the query vectors and the key vectors, and weighting and aggregating the value vectors based on the attention weights to obtain the attention features; and / or, The step of generating the control command based on the attention features includes: generating the control command based on the attention features and the current state information.

6. The method according to claim 3, characterized in that, The walking control model was trained in the following manner: Acquire training sample data, which includes the robot's sample observation information at the sample time, the sample history observation information of each of the multiple consecutive sample history times before the sample time, and the robot's motion state information label at the sample time; Based on the sample observation information, the sample history observation information of each of the consecutive sample history times, and the motion state information label, the parameters of the initial walking control model are updated to obtain the walking control model.

7. The method according to claim 6, characterized in that, The step of updating the parameters of the initial walking control model based on the sample observation information, the sample historical observation information of each of the consecutive sample historical moments, and the motion state information label to obtain the walking control model includes: The sample observation information and the sample history observation information of each of the consecutive sample history times are input into the state estimation network of the initial walking control model to obtain the sample motion state information of the robot at the sample time. The parameters of the initial walking control model are updated based on the sample observation information, the sample motion state information, and the motion state information label to obtain the walking control model.

8. A robot, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the walking control method of the robot as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute and implement the steps of the walking control method for a robot as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the steps of the walking control method for a robot as described in any one of claims 1 to 7.

Images