Control method and system of universal foot type robot

Abstract

The invention provides a control method and system for a universal foot type robot, and relates to the technical field of robot control, and the control method comprises the steps: carrying out the unified token processing of the time sequence observation data of a multi-source sensor, adding the modality, time and sensor configuration embedding, and constructing a unified input representation compatible with a heterogeneous sensor and different sampling frequencies; a shielding mask is introduced to enable a shielding time sequence attention encoder to cope with data shielding and frame loss; robot form description information and world state latent variables are fused in a cross-form fusion module, and a single strategy network (composed of a shielding time sequence attention encoder, the cross-form fusion module and a universal strategy head) is adapted to foot type robot forms of different kinematics topologies and kinetic parameters; finally, a joint-level control instruction is output to drive the robot to move, a complete universal foot type robot control scheme is formed, and the core problems that in the prior art, specific sensors and forms are relied on, and sensing data missing is difficult to deal with are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to a control method and system for a general-purpose legged robot. Background Technology

[0002] Legged robots, due to their superior terrain adaptability, have shown broad application prospects in fields such as industrial inspection, logistics handling, disaster relief, and field exploration. In recent years, with the rapid development of deep reinforcement learning and simulation technologies, learning-based motion control of legged robots has achieved significant breakthroughs in tasks such as speed tracking, disturbance recovery, and navigating complex terrain.

[0003] However, existing legged robot control methods still face technical bottlenecks in engineering implementation. For example, current methods have high requirements for the integrity and synchronization of sensor data, typically assuming that each modal sensor is always stable and available. But in practical applications, sensor data often becomes missing or distorted due to environmental factors (such as occlusion, strong light, reflection, fog, and dust) or hardware problems (such as frame drops, latency, and disconnections). Traditional policy networks lack effective modeling of these partially observable states. When the input data is incomplete, the control performance drops sharply, and may even cause the robot to become unstable and fall.

[0004] Furthermore, current multi-source sensor fusion methods mostly employ simple splicing or weighted fusion, which makes it difficult to fully utilize the temporal dependencies and complementarities between different modes, and also makes it difficult to handle the data asynchrony problem caused by differences in sensor sampling frequencies. Summary of the Invention

[0005] The problem solved by this invention is one or more of the aforementioned related technical problems.

[0006] To address the above problems, this invention provides a control method and system for a universal legged robot.

[0007] In a first aspect, the present invention provides a control method for a universal legged robot, comprising: Acquire multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles; The time-series observation data from the multi-source sensors are uniformly tokenized to obtain a modal token sequence. Each modal token in the modal token sequence is then embedded by adding modal embedding, time embedding, and sensor configuration embedding to obtain a modal token sequence with added embeddings. Generate a masking code based on the preset masking strategy during the training phase or the quality indicators during the inference phase. The embedded modal token sequence and the occlusion mask are input into the occlusion temporal attention encoder, which outputs the world state latent variable. The morphological description information of the legged robot is obtained, and the morphological description information and the world state latent variables are input into the cross-morphological fusion module to obtain morphological conditional latent variables. The morphological conditional latent variables are then input into the general strategy head, and joint-level control commands are output to control the movement of the legged robot.

[0008] Optionally, the step of uniformly tokenizing the multi-source sensor time-series observation data to obtain a modal token sequence includes: Based on the corresponding encoder, feature extraction is performed on the time-series observation data of each mode in the multi-source sensor time-series observation data to obtain the initial features of each mode; The initial features of each modality are linearly mapped to a vector space of uniform dimension to obtain the modality token of each modality at the current time. Arrange modal tokens from different times and modalities in chronological and modal order to form a modal token sequence.

[0009] Optionally, the occlusion mask includes at least one of random modal occlusion mask, random time segment occlusion mask, and structured occlusion mask; wherein the structured occlusion mask is generated based on the occlusion distribution of the real scene, and the structured occlusion mask includes at least one of lateral view occlusion, local area occlusion, distance segment occlusion, and continuous frame drop occlusion.

[0010] Optionally, the step of inputting the embedded modal token sequence and the masking mask into the masking temporal attention encoder to output world state latent variables includes: Obtain the length information of the current sliding window, and determine the causal mask based on the length information; In each attention layer of the occlusion temporal attention encoder, the attention score is calculated using the embedded modal token sequence as the source of query, key and value, and the causal mask and the occlusion mask are superimposed on the attention score; The attention score after the overlay mask is normalized to obtain the attention weight, and the values ​​are weighted and summed according to the attention weight to obtain the attention output of the current layer. After being processed by stacking multiple attention layers, the feature representation corresponding to the current moment is extracted from the attention output of the last layer, and the world state latent variable is obtained by pooling.

[0011] Optionally, the cross-morphological fusion module includes a morphological map encoder and a feature fusion unit; the step of inputting the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables includes: The morphological graph encoder is constructed based on a graph neural network or a morphological Transformer encoder; A morphological diagram is constructed using the kinematic topology and dynamic parameters of the legged robot. The nodes of the morphological diagram represent joints or links, and the edges of the morphological diagram represent connection relationships. The node attributes include at least one of the following: link length, mass, moment of inertia, joint type, and joint limit. The morphology graph is input into the graph neural network or the morphology Transformer encoder to obtain a morphology token sequence; The morphological token is fused with the world state latent variable using the feature fusion processor to obtain the morphological conditional latent variable.

[0012] Optionally, the step of fusing the morphological token with the world state latent variable through the feature fusion processor to obtain the morphological conditional latent variable includes: Based on feature linear modulation or cross-attention mechanism, the morphological token sequence is injected into the world state latent variable to obtain the morphological conditional latent variable.

[0013] Optionally, the control method for the general-purpose legged robot further includes: Based on the quality indication and cross-modal reconstruction residuals, the reliability of each mode at the current moment is determined by a reliability estimator; Dynamically adjusting the masking code based on the reliability includes: When the reliability of any of the modalities is lower than a preset threshold, the corresponding modal token is replaced with a learnable mask token, and the weight of the modality is reduced or masked in the attention calculation.

[0014] Optionally, the occlusion temporal attention encoder and the cross-morphological fusion module are obtained through three-stage training, which includes self-supervised occlusion modeling pre-training, multi-task control learning, and cross-morphological expert-student distillation. The self-supervised occlusion modeling pre-training includes: Acquire pre-collected time-series observation data from the multi-mode multi-sensor simulation data pool as historical training data; Random and structured masking are applied to the historical training data to generate a masked modal token sequence; The masked modal token sequence is input into the initialized masked temporal attention encoder. The original observation data of the masked part is used as the supervision signal to optimize the masking reconstruction loss, dynamic consistency loss and cross-modal alignment loss, and to train the pre-trained masked temporal attention encoder. The multi-task control learning includes: The parameters of the pre-trained occlusion temporal attention encoder are connected to the initialized cross-morphological fusion module and the initialized general policy head to form a preliminary policy network. In a multi-task reinforcement learning environment, the preliminary policy network interacts online with the environment to generate experience data. Based on the occlusion temporal attention encoder in the preliminary policy network, the modal token sequence at the current moment is input to obtain temporary world state latent variables; the morphological description information of the current legged robot form is input to the cross-morphological fusion module in the preliminary policy network to generate temporary form tokens; the temporary form tokens are fused with the temporary world state latent variables to obtain temporary form conditional latent variables, which are then input into the general policy head in the preliminary policy network to output actions. Based on the reward signals from environmental feedback, construct a reinforcement learning control loss; The gradient of the reinforcement learning control loss is backpropagated to the occlusion temporal attention encoder, cross-morphology fusion module and general policy head to jointly optimize the parameters. Modal randomization perturbation is continuously applied during training to learn motion control strategies that adapt to different robot shapes, and the occlusion temporal attention encoder, cross-morphology fusion module and general policy head are obtained after preliminary joint training. The cross-morphological expert-student distillation includes: Multiple expert policy networks are obtained, and the action distributions generated by each expert policy network under various states are used as soft labels. The occlusion temporal attention encoder, cross-morphological fusion module, and general policy head, which are preliminarily jointly trained, constitute a unified student policy network. The soft label is used as a supervision signal to optimize the distillation loss so that the action distribution output by the unified student policy network is the same as the action distribution of each expert policy network. During the distillation process, the occlusion reconstruction loss is jointly optimized, and the occlusion temporal attention encoder, the cross-morphological fusion module, and the general policy head are trained.

[0015] Optionally, the occlusion-time attention encoder also outputs uncertainty; the control method for the general-purpose legged robot further includes: The weights of the desired speed command or safety constraints are adaptively adjusted based on the uncertainty.

[0016] Secondly, the present invention provides a control system for a universal legged robot, comprising: The acquisition unit is used to acquire multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles. The embedding unit is used to uniformly tokenize the time-series observation data of the multi-source sensors to obtain a modal token sequence, and to perform embedding processing on each modal token in the modal token sequence, adding modal embedding, time embedding and sensor configuration embedding to obtain a modal token sequence after adding embedding. The processing unit is used to generate a masking code based on the preset masking strategy in the training phase or the quality indicator in the inference phase; input the embedded modal token sequence and the masking code into the masking temporal attention encoder, and output the world state latent variable; The control unit is used to acquire the morphological description information of the legged robot, input the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables, input the morphological conditional latent variables into the general strategy head, and output joint-level control commands to control the movement of the legged robot.

[0017] The beneficial effects of the control method and system for the universal legged robot of the present invention are: By unifying and tokenizing multi-source sensor temporal observation data and adding modal embedding, temporal embedding, and sensor configuration embedding, a unified input representation compatible with heterogeneous sensor combinations and different sampling frequencies is constructed. By introducing occlusion masks, the occlusion temporal attention encoder can effectively cope with abnormal situations such as sensor data occlusion and frame loss, significantly improving the robot's perception robustness in complex environments. Furthermore, by fusing robot morphological description information with world state latent variables in a cross-morphological fusion module, a single policy network (composed of the occlusion temporal attention encoder, the cross-morphological fusion module, and a general policy head) is universally adapted to legged robot morphologies with different kinematic topologies and dynamic parameters, avoiding the high cost of developing separate control strategies for each robot morphology. Finally, joint-level control commands are output to directly drive robot movement, forming a complete, unified, and end-to-end deployable general legged robot control scheme. This effectively solves the core problems in existing technologies, such as control strategies heavily relying on specific sensor configurations and robot morphologies and difficulty in coping with missing perception data, and has significant industrial application value and generalization capabilities. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a control method for a general-purpose legged robot according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the control system of a general-purpose legged robot according to an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0020] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0021] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0024] In related technologies, most control policy networks are designed and trained for specific robot forms (such as quadrupeds or bipeds with fixed degrees of freedom) and fixed sensor configurations (such as a fixed combination of depth camera and IMU). When the robot form changes (such as from quadruped to wheeled or bipedal) or the sensor configuration is adjusted (such as removing the LiDAR or changing the camera model), the original policy network is difficult to reuse directly. Data needs to be collected again and extensive retraining is required, which leads to extended development cycles and high deployment costs.

[0025] Therefore, there is an urgent need for a universal legged robot control method that is compatible with different robot forms and sensor configurations, and is robust to sensor data occlusion and frame loss, in order to reduce deployment costs and improve control stability in complex environments.

[0026] like Figure 1 As shown in the figure, an embodiment of the present invention provides a control method for a general-purpose legged robot, comprising: Step S100: Obtain multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles.

[0027] Specifically, this process aims to provide the raw data foundation for subsequent unified tokenization. The acquired data covers historical observations from the current moment and several consecutive control cycles prior to it, forming a fixed-length sliding time window.

[0028] The data sources fall into two main categories: proprioceptive sensors, used to perceive the robot's own state; and external perception sensors, used to perceive the external environment. For example, proprioceptive data may include joint angles, joint angular velocities, joint torques, triaxial accelerations and angular velocities measured by the IMU, and foot contact signals, etc.; external perception data may include depth maps acquired by depth cameras, visual images acquired by RGB cameras, point cloud data acquired by LiDAR, or their rasterized representations, etc. This multi-source data is organized in a time-series manner, with observation data from each sensor recorded and cached within each control cycle, forming the input for subsequent processing steps.

[0029] By acquiring multi-source sensor time-series observation data covering the current and historical control cycles, a rich and complete original information foundation is provided for subsequent unified tokenization processing. This enables the policy network to utilize both ontological and external perception information and establish temporal dependencies with the help of historical observations. As a result, it can still maintain an effective estimation of the robot's state even when some sensor data is missing or distorted, significantly improving the perception robustness and decision stability of the control policy in complex environments.

[0030] Step S200: Unify the multi-source sensor time-series observation data into a unified token to obtain a modal token sequence, and perform embedding processing on each modal token in the modal token sequence, adding modal embedding, time embedding and sensor configuration embedding to obtain a modal token sequence after adding embedding.

[0031] Specifically, heterogeneous and asynchronous multi-source sensor data is transformed into a unified, context-rich serialized representation for subsequent processing by a temporal attention encoder.

[0032] First, an encoder is constructed for each type of sensor modality (e.g., joint angle, IMU, depth map, LiDAR point cloud, etc.) to map the raw observation data into a fixed-dimensional feature vector, thus obtaining the modality token for that modality at the current moment. For example, joint angle data can be encoded into a 128-dimensional vector using a multilayer perceptron; for depth images, features can be extracted and compressed to the same dimension using a lightweight convolutional neural network. Tokens from different moments and modalities are arranged in chronological and modal order to form a modality token sequence, thereby preserving the original structure of the time and modal dimensions.

[0033] After obtaining the basic modality tokens, three types of learnable embedding vectors are further added to each token: first, modality embedding, used to identify the sensor type to which the token belongs (such as IMU, depth camera, etc.), enabling the model to distinguish the information sources of different modalities; second, temporal embedding, used to encode the position of the token within a time window (e.g., using absolute position encoding or relative position encoding), enabling the model to perceive temporal order and time intervals; and third, sensor configuration embedding, used to describe the inherent properties of the modality, such as sampling frequency, sensor latency, field of view, ranging range, etc. This information is added to the token in vector form, allowing the model to adaptively handle the differences brought about by different sensor configurations. Through the above embedding processing, a unified token sequence rich in modality identity, temporal context, and sensor characteristic information is finally obtained, providing high-quality input for subsequent occlusion temporal attention encoding.

[0034] The sensor configuration embeds at least one or a combination of the following information: sampling frequency, delay time, calibration quality index, field of view, ranging range, and resolution.

[0035] By unifying the tokenization of multi-source sensor temporal observation data and adding modality, time, and sensor configuration embeddings, heterogeneous and asynchronous sensor data are transformed into structured unified sequence representations. This enables the subsequent temporal attention encoder to effectively fuse ontological and external perception information in the same feature space, while explicitly distinguishing different modalities, different times, and different sensor characteristics. This significantly improves the model's adaptability to changes in sensor combinations, sampling frequency differences, and data asynchrony, laying a solid input foundation for building a general and robust legged robot control strategy.

[0036] Step S300: Generate a masking code based on the preset masking strategy in the training phase or the quality indicator in the inference phase.

[0037] Specifically, the masking code is a binary or continuous value matrix corresponding to the dimension of the modality token sequence, used to indicate whether the data of each modality is available or its reliability at each time step.

[0038] During the training phase, the masking code is generated based on a preset masking strategy. The preset masking strategy is a set of artificially designed rules used to simulate the degradation of sensor data in the real world.

[0039] Preset occlusion strategies mainly include random modal occlusion: randomly disabling the entire data channel of one or more sensors (such as depth cameras).

[0040] Structured view occlusion: Simulates physical occlusion. For example: Lateral occlusion: Randomly occlude the left or right side of an image or depth map to simulate the lack of field of vision when a robot is close to a wall or enters a narrow passage.

[0041] Near and far range occlusion: Covers point cloud or depth values ​​within a specific distance range to simulate long-range failure caused by rain and fog or near-range blind spots caused by strong reflections.

[0042] Temporal dimensional occlusion: Simulates sensor frame loss or communication delay.

[0043] Continuous frame loss masking: Masking data of a certain modality for K consecutive time intervals.

[0044] Timestamp perturbation: artificially creating timestamp offsets to simulate asynchronous input.

[0045] This strategy ultimately generates a binary or continuous mask (M_occl / M_pred).

[0046] For example, the strategy could include: randomly selecting a modality at a given time (such as the depth map at time t) and marking it as unavailable; or occluding the same modality across multiple consecutive time points to simulate sensor downtime; or performing structured occlusion on specific regions of the depth image / point cloud (such as the left field of view or near-field range) to simulate the loss of vision when the robot approaches a wall, enters a narrow passage, or encounters rain or fog. These occlusion operations are applied randomly during training, forcing the model to use learnable placeholders at the occluded token locations and fill in the missing content using other available modalities and historical information.

[0047] During the inference phase, the occlusion mask is dynamically generated based on real-time quality indicators. Quality indicators are metrics extracted from the raw sensor data that reflect its current operational status. For example, for a depth camera, the quality indicator could be the effective pixel ratio (i.e., the proportion of pixels with valid depth values ​​out of the total pixels). When this ratio is too low, it indicates that the depth data is largely invalidated due to strong light, reflection, or occlusion. For LiDAR, the quality indicator could be the abnormal echo intensity ratio. For images, it could be the blurriness or overexposure level calculated by an algorithm. Furthermore, cross-modal consistency residuals within the model can be incorporated. For example, using IMU and historical information to predict the depth map that should be seen, the predicted value is compared with the actual value; the greater the difference, the more suspicious the current depth data. Based on these quality indicators, the system calculates a reliability score between 0 and 1 for each modality in real time. When the reliability score is below a preset threshold, the token location corresponding to that modality is marked as occluded, thus generating the occlusion mask for the inference phase.

[0048] By dynamically generating masking codes based on the preset masking strategy during the training phase and the quality indicators during the inference phase, the model can actively learn to cope with various degradation situations such as sensor data occlusion, frame loss, and asynchrony during training. During inference, it can adaptively mask unreliable modal information based on the real-time data quality. This significantly improves the perceptual robustness and generalization ability of the policy network in complex environments, realizing a technological leap from "passively receiving data" to "actively evaluating the credibility of information," and providing a key guarantee for the stable and reliable operation of legged robots in the real world.

[0049] Step S400: Input the embedded modal token sequence and the masking mask into the masking temporal attention encoder, and output the world state latent variable.

[0050] Specifically, the embedded modal token sequence is organized as a temporal input, where each token already contains modal identity, time location, and sensor configuration information. Simultaneously, a masking code is included as part of the input to indicate which times and which modal data are currently available.

[0051] The Temporal Occlusion Attention Encoder (MTAN) is built on the Transformer architecture. Through multi-layered self-attention computation, it enables each token to pay attention to information from other positions in the sequence. During attention computation, the model combines the indication of the occlusion mask to attenuate or mask unusable modal tokens, forcing the model to still make inferences using historical information and information from other available modalities even when information is missing. For example, when a robot (legged robot) is walking, if the depth camera loses data at some moments due to strong light, the occlusion mask marks the depth tokens at these moments as unusable. The encoder then uses the attention mechanism to comprehensively infer the terrain undulations and obstacle positions from historical depth information from adjacent moments, as well as the current IMU and joint data, thereby maintaining effective perception of the environment.

[0052] After multiple layers of attention processing, features corresponding to the current moment are extracted from the output of the last layer, and a fixed-dimensional world state latent variable is obtained through aggregation. This latent variable is a compressed representation of the robot's environment, its own state, and the interaction between the two. It contains the most relevant content for control decisions from the multimodal historical information, providing a unified information foundation for subsequent morphological fusion and action generation.

[0053] By processing the embedded modal token sequence through a masked temporal attention encoder, multimodal historical information and temporal dependencies are effectively integrated. At the same time, by actively addressing sensor data deficiencies using masking codes, stable and complete world state latent variables can still be output under certain observation conditions. This significantly improves the policy network's perception robustness to practical problems such as occlusion and frame loss, and provides high-quality state representation for subsequent general-purpose legged robot control.

[0054] Step S500: Obtain the morphological description information of the legged robot, and input the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables. Input the morphological conditional latent variables into the general strategy head and output joint-level control commands to control the movement of the legged robot.

[0055] Specifically, this process aims to integrate the robot's own physical morphology information with the latent variables of the world state extracted from multimodal perception to generate control instructions adapted to the current robot body, thereby achieving unified control of legged robots of different morphologies by a single policy network.

[0056] First, morphological description information of the legged robot is obtained. This information is a quantitative representation of the robot's physical structure and may include: the number and type of joints, link lengths and mass distribution, joint range of motion, rated torque of actuators, and foot geometry parameters. These parameters collectively characterize the robot's kinematics and dynamics, and are essential prior knowledge for generating adaptive control commands.

[0057] Next, the morphological description information and world state latent variables are input into the cross-morphological fusion module (MFM). The core function of this module is to encode the morphological description information into morphological features that match the dimensions of the latent variables and then deeply fuse them with the world state latent variables. For example, when processing a quadruped robot, the cross-morphological fusion module generates corresponding morphological features based on the topology of its four legs and the link parameters, and interacts with the current world state latent variables (which include information such as terrain, obstacles, and robot posture). When processing a bipedal robot, the module generates different morphological features based on its leg structure and center of mass distribution, and similarly fuses them with environmental information. Through this fusion mechanism, the general information in the world state latent variables that is originally related to the environment is "modulated" with the physical constraints and capability boundaries specific to the current robot, thus forming morphologically conditional latent variables.

[0058] Finally, the morphologically conditional latent variables are input into the general strategy head, which outputs joint-level control commands. The general strategy head is a unified neural network module whose input is latent variables incorporating environmental and morphological information, and whose output is specific joint control quantities, such as desired joint angles, joint velocities, or joint torques. Because the morphologically conditional latent variables already contain robot-specific information, the same general strategy head can generate commands adapted to the joint spaces of robots with different body shapes. For example, for a quadruped robot, the strategy head might output control quantities for 12 joints (3 joints per leg); for a bipedal robot, it might output control quantities for 6 or more joints. These commands are sent to the underlying actuators to drive the robot to complete the desired motion.

[0059] By fusing morphological description information with world state latent variables in a cross-morphological fusion module, and then outputting joint-level control commands through a general strategy head, a single strategy network can adapt to the morphology of legged robots with different kinematic topologies and dynamic parameters, avoiding the high cost of developing separate control strategies for each robot. At the same time, the fused morphological conditional latent variables contain both environmental perception information and robot body constraints, ensuring that the output joint commands meet both the current terrain requirements and the robot's own physical limitations. This significantly improves the versatility and deployment efficiency of the control strategy, providing key technical support for the large-scale application of legged robots in diverse scenarios.

[0060] In this embodiment, by unifying and tokenizing multi-source sensor temporal observation data and adding modal embedding, temporal embedding, and sensor configuration embedding, a unified input representation compatible with heterogeneous sensor combinations and different sampling frequencies is constructed. By introducing occlusion masks, the occlusion temporal attention encoder can effectively cope with abnormal situations such as sensor data occlusion and frame loss, significantly improving the robot's perception robustness in complex environments. Furthermore, by fusing robot morphological description information with world state latent variables in the cross-morphological fusion module, a single policy network (composed of the occlusion temporal attention encoder, the cross-morphological fusion module, and the general policy head) is universally adapted to legged robot morphologies with different kinematic topologies and dynamic parameters, avoiding the high cost of developing separate control strategies for each robot morphology. Finally, joint-level control commands are output to directly drive robot movement, forming a complete, unified, and end-to-end deployable general legged robot control scheme. This effectively solves the core problems in existing technologies, such as control strategies heavily relying on specific sensor configurations and robot morphologies and difficulty in coping with missing perception data, and has significant industrial application value and generalization ability.

[0061] Optionally, the step of uniformly tokenizing the multi-source sensor time-series observation data to obtain a modal token sequence includes: Based on the corresponding encoder, feature extraction is performed on the time-series observation data of each mode in the multi-source sensor time-series observation data to obtain the initial features of each mode; The initial features of each modality are linearly mapped to a vector space of uniform dimension to obtain the modality token of each modality at the current time. Arrange modal tokens from different times and modalities in chronological and modal order to form a modal token sequence.

[0062] Specifically, the above process aims to transform heterogeneous, asynchronous, and high-dimensional multi-source sensor data into structured, unified-dimensional sequence representations, providing a standardized input format for subsequent temporal attention coding. This includes the following three sub-processes: I. Feature extraction based on modal encoder: For different sensor modes, encoders adapted to their data characteristics are constructed to map the raw observation data into initial feature vectors. Examples include the following optional methods: For proprioceptive data (such as low-dimensional time-series data like joint angles, joint angular velocities, joint torques, and IMU measurements), a multilayer perceptron (MLP) can be used as the encoder. For example, the angles (12-dimensional) and angular velocities (12-dimensional) of 12 joints can be concatenated into a 24-dimensional vector, which is then input into a 64-dimensional MLP with two hidden layers, and the output is a 64-dimensional initial feature.

[0063] For visual / depth data (high-dimensional grid data such as images and depth maps in external perception data), convolutional neural networks (CNNs) can be used as encoders. For example, for a 224×224 RGB image, the first few layers of a lightweight MobileNet or ResNet-18 can be used to extract features, and then global average pooling can be used to obtain 128-dimensional initial features; for depth maps, a similar convolutional structure can be used for processing.

[0064] For LiDAR point cloud data, PointNet, PointNet++, or voxel convolutional networks can be used as encoders. For example, a frame of point cloud (assuming there are N points) can be input into PointNet, and 128-dimensional global features can be obtained as initial features through multilayer perceptron and max pooling.

[0065] For binary or low-dimensional discrete data such as foot-to-ground signals, an embedding layer can be used to map them into continuous vectors, or they can be processed directly through an MLP.

[0066] II. Linear mapping to a unified dimension: Since the initial feature dimensions of different modal encoders may differ (e.g., visual features are 128-dimensional, IMU features are 64-dimensional), a linear mapping (i.e., a fully connected layer) is needed to transform the initial features of all modalities to the same preset dimension d (e.g., d=128). This mapping layer can be independent for each modality or can share parameters among some modalities, depending on the similarity of the modalities and the design choices. After this step, the output of each modality at the current time step is a modality token with a unified dimension.

[0067] III. Serialization organization of temporal and modal dimensions: Modal tokens from different times and modes are arranged in a predetermined order to form a two-dimensional modal token sequence. For example, the following organization method can be used: Prioritize time order: First, arrange the tokens at each time step in chronological order, then within each time step, arrange them in modal order. For a sliding window of length L (containing time steps t-L+1 to t) and There are 3 modes, and the sequence length is L× For example, if L=10 and the modal includes 3 modes {IMU, joint, depth}, then the sequence length is 30, and the order is: (IMU at time t-9, joint at time t-9, depth at time t-9, IMU at time t-8, ..., depth at time t).

[0068] Modality-major order: First arrange by mode, then arrange within each mode by time order. This yields a length of L× The sequence is changed, but the order becomes: (IMU token from time t-9 to time t, joint token from time t-9 to time t, depth token from time t-9 to time t).

[0069] Regardless of the organization method used, the position of each token in the sequence uniquely corresponds to a specific "time-modality" combination. This structured organization method enables subsequent attention mechanisms to simultaneously capture the interaction between different modalities at the same time (cross-modal attention) and the evolution of the same modality at different times (temporal attention).

[0070] Through the aforementioned unified tokenization process, heterogeneous, asynchronous, and high-dimensional multi-source sensor data is transformed into structured, dimensionally unified sequence representations. This achieves the alignment and fusion of information from different modalities in the feature space, providing a standardized input format for subsequent temporal attention encoding. Simultaneously, by designing dedicated encoders for different modalities, key information features in the original data of each modality are preserved, avoiding information loss caused by simple splicing or downsampling. The resulting modal token sequence retains structural information in both the temporal and modal dimensions, enabling the model to flexibly learn complex dependencies across time and modalities. This lays a solid data foundation for constructing a general and robust perception and control strategy for legged robots.

[0071] Optionally, the occlusion mask includes at least one of random modal occlusion mask, random time segment occlusion mask, and structured occlusion mask; wherein the structured occlusion mask is generated based on the occlusion distribution of the real scene, and the structured occlusion mask includes at least one of lateral view occlusion, local area occlusion, distance segment occlusion, and continuous frame drop occlusion.

[0072] Specifically, the generation of masking codes aims to provide a dynamic "attention adjustment signal" for the masking temporal attention encoder, enabling the model to actively learn to cope with various sensor data degradation scenarios during the training phase and adaptively adjust based on real-time quality indicators during the inference phase. Masking codes can be categorized into three main types based on their generation strategy and simulation objectives; in practical applications, one or more of these types can be used in combination. 1. Random Modal Masking: This type of masking randomly selects certain modal tokens at a specific time with independent probability to mask them, aiming to simulate temporary sensor failures caused by hardware malfunctions, communication interruptions, or environmental interference.

[0073] For example: for a sliding window of length L and For each modality, a occlusion probability p is set (e.g., p=0.2). Each modality at each time step within the window is iterated over, and each modality token is independently marked as occluded with probability p. For example, in a window containing 10 time steps and 3 modalities (IMU, joint, depth), the depth map at time step t-5 and the IMU data at time step t-2 might be randomly occluded, while other modalities remain available. The occluded positions are replaced with learnable placeholder vectors, and the model needs to utilize other unoccluded modalities and historical information to fill in the missing content.

[0074] Application scenario simulation: Simulate situations such as temporary disconnection of a single sensor (e.g., a depth camera temporarily shuts down due to overheat protection), loss of single-frame data due to communication packet loss, or local environmental interference (e.g., strong light causing single-frame depth failure).

[0075] 2. Random Time Segment Masking: This type of masking performs continuous masking of a specific modality over multiple frames in the time dimension, aiming to simulate situations such as long-term sensor failure, communication interruption, or data delay accumulation.

[0076] For each modality, a segment occlusion probability and a maximum segment length are set. On a modal basis, a start time and a duration k (e.g., k = 3~5 frames) are randomly selected, and all k consecutive tokens for that modality starting from the start time are occluded. For example, in a sliding window, the data from the depth modality for three consecutive frames from t-7 to t-5 are randomly selected and occluded, simulating a complete failure of the depth camera during this period. The model is forced to rely on other modalities (such as IMU, joints) and historical information to maintain continuity of environmental perception.

[0077] Application scenario simulation: Simulate continuous data failure caused by prolonged sensor disconnection, communication link interruption, or robot passing through areas with strong interference (such as strong magnetic fields affecting the IMU).

[0078] III. Structured Occlusion Masks: This type of mask is generated based on occlusion patterns common in real-world scenes with specific spatial or temporal structures. Its aim is to ensure the model has seen occlusion distributions consistent with the real world during training, thereby improving generalization ability. Structured occlusion masks further include the following subtypes: 1. Lateral field of view occlusion: Simulates the loss of field of view when the robot is close to a wall, enters a narrow passage, or has an obstacle on one side.

[0079] For example, for 2D perceptual data such as depth maps or images, randomly select the left or right side and mark the feature locations corresponding to all pixels within that side region as occluded with a random width w (e.g., occupying 20% ​​to 50% of the image width). For instance, randomly occlude 30% of the left side region on the depth map to simulate a situation where the robot's left field of vision remains usable even when there is a wall on the right. During training, randomly switch between the left and right sides and the occlusion width to enable the model to learn to determine passable areas even when part of the field of vision is missing.

[0080] Application scenario simulation: The robot walks along a wall, enters a narrow passage, or faces a large obstacle on one side.

[0081] 2. Localized Occlusion: Simulates situations where the field of view is partially obscured by random objects (such as leaves, dust, and raindrops).

[0082] One or more rectangular regions (random location and size) are randomly generated on the 2D sensing data, and features within these regions are marked as occlusions. For example, a 30×30 pixel rectangular occlusion is generated slightly to the right of the center of the depth map to simulate the camera being briefly obscured by leaves. The number of regions (1~3) and their size range can be set, enabling the model to learn to make inferences using surrounding regions when local information is missing.

[0083] Application scenario simulation: the lens is obscured by splashed mud, the field of view is partially obstructed when passing through low-hanging branches, and the lens is blurred in rainy or snowy weather, etc.

[0084] 3. Range-based occlusion: Simulates the failure of sensors due to physical characteristics within a specific range, such as the near-range blind zone of lidar, the attenuation of long-range rain and fog, and the failure of depth cameras when they are too close or too far away.

[0085] For example, for modalities containing distance information, such as depth maps or point clouds, a random distance interval [d_min, d_max] is selected (e.g., a near-range blind zone of 0~0.5m, or a far-range interval of 8~10m), and all valid points within this interval are marked as occluded. For instance, to simulate data failure caused by excessive reflection at close range (<0.5m) of a LiDAR, all point clouds within that range are occluded. During training, different distance intervals are randomly selected, allowing the model to learn to rely on other cues when specific distance information is missing.

[0086] Application scenario simulation: near-field blind spots when the robot is close to a wall, failure of long-distance ranging in foggy weather, and interference from reflections at specific distances caused by transparent objects or mirrors.

[0087] 4. Continuous frame loss masking: Simulates the complete loss of multiple consecutive frames of data caused by communication delays, processing timeouts, or frame drops in the sensor.

[0088] For example, in the time dimension, a starting time and a frame drop length k (e.g., k = 5~10 frames) are randomly selected, and all modal data for k consecutive frames starting from that time are completely masked. This is the most stringent form of masking, simulating full-perception failure in extreme cases, forcing the model to rely solely on historical states and action memories to maintain control. During training, the frame drop length and frequency are gradually increased, enabling the model to learn to maintain basic stability even under prolonged periods of no sensory input.

[0089] Application scenario simulation: extreme situations such as instantaneous overload of the main control computer causing data acquisition thread blockage, long-term interruption of communication link, and simultaneous disconnection of all devices under the sensor bus.

[0090] In actual training, the above-mentioned masking codes are not used in isolation, but are applied dynamically in combination. For example, the following scheduling strategy can be adopted: Random modal masking is applied with a certain probability (e.g., 0.3) to simulate the occasional sensor failures in daily life; Apply random time-segment masking with a certain probability (e.g., 0.2) to simulate short-term communication interruption; Structured occlusion is applied with a certain probability (e.g., 0.3) (further randomly selecting sub-types such as lateral, local, and distance segments) to simulate typical occlusion in real-world scenes; Apply continuous frame-dropping occlusion with a low probability (e.g., 0.1) as adversarial training for extreme cases; The remaining probabilities (e.g., 0.1) are not masked, and the complete data is retained for training.

[0091] The occlusion ratio can be dynamically adjusted during training, for example, gradually increasing from a lower occlusion ratio (0.2) in the early stage to a higher ratio (0.6) in the later stage, forming a learning effect and enabling the model to gradually master the ability to cope with missing information.

[0092] By introducing various types of occlusion masks, such as random modal occlusion, random temporal segment occlusion, and structured occlusion, the model can systematically encounter and learn to cope with various sensor data degradation situations during the training phase, including occasional single-frame failures, continuous multi-frame interruptions, and field-of-view occlusion with real spatial structural features. In particular, the structured occlusion strategy is designed based on the occlusion distribution of real scene, so that the model has seen occlusion patterns consistent with the deployment environment during training, which significantly improves the generalization ability. The combined application and dynamic scheduling of multiple occlusion types enable the model to gradually master state completion and robust control capabilities under conditions of missing information, from simple to complex. Finally, when facing actual sensor failures or environmental occlusion during the inference phase, it can still maintain stable and reliable motion control performance.

[0093] Optionally, the step of inputting the embedded modal token sequence and the masking mask into the masking temporal attention encoder to output world state latent variables includes: Obtain the length information of the current sliding window, and determine the causal mask based on the length information. In each attention layer of the occlusion temporal attention encoder, the attention score is calculated using the embedded modal token sequence as the source of query, key and value, and the causal mask and the occlusion mask are superimposed on the attention score; The attention score after the overlay mask is normalized to obtain the attention weight, and the values ​​are weighted and summed according to the attention weight to obtain the attention output of the current layer. After being processed by stacking multiple attention layers, the feature representation corresponding to the current moment is extracted from the attention output of the last layer, and the world state latent variable is obtained by pooling.

[0094] Specifically, the core of this process lies in using a masked temporal attention encoder to perform deep temporal modeling on the modal token sequence after embedding, while combining causal masking and masking masking to integrate cross-time and cross-modal information while ensuring real-time control causality, and finally outputting a fixed-dimensional world state latent variable as a unified state representation for subsequent decisions.

[0095] I. Generation of Causal Masks: First, based on the length L of the current sliding window (i.e., the number of historical frames), generate a window of size L. The causal mask matrix is ​​a lower triangular matrix where, for the i-th position in the output sequence (corresponding to a mode at a certain time), it only allows attention to the j-th ≤ i-th position in the input sequence, while assigning a negative infinity (-∞) bias to positions j>i. This design ensures that the model does not utilize information from future times when calculating the output at the current time, satisfying the causal constraint in online control that "current decisions can only rely on historical observations." For example, if the window length L=5, the number of modes... =3, then the total length of the sequence is 15. The causal mask will form a 15×15 lower triangular matrix, with the elements on the diagonal and below being 0 (allowed to be noticed) and the elements above being -∞ (disallowed to be noticed).

[0096] II. Attention Calculation and Masking: In each layer of the occlusion-temporal attention encoder, the embedded modal token sequence X serves as the common source of query Q, key K, and value V (typically obtained through linear projection). First, the original attention score matrix is ​​computed. , where d is the token dimension. Then, two mask matrices are superimposed on A_raw: Causal mask M_causal: As mentioned earlier, it is added together to set the attention score of the future position to -∞, so that the weight in the subsequent softmax is close to 0.

[0097] The masking mask M_occl: This mask has the same dimension as the input sequence, but is expanded to the same shape as the attention score matrix through a broadcast mechanism. The masking mask is typically a binary or continuous-valued matrix. For modal tokens marked as unusable, a large negative value (e.g., -1e9) or -∞ is assigned to their corresponding positions, effectively masking these tokens during attention computation. For usable tokens, 0 or a small learnable bias is superimposed. In this way, the model automatically ignores unreliable modalities when aggregating information, instead relying on other usable modalities and historical information.

[0098] The superimposed attention score matrix is ​​A = A_raw + M_causal + M_occl. Then, A is softmax normalized along its last dimension to obtain the attention weight matrix W_att = softmax(A). Finally, the attention weights are weighted and summed with the value V to obtain the attention output Y = W_att × V for the current layer. This output, after processing through residual connections, layer normalization, and a feedforward network, serves as the input to the next layer.

[0099] III. Multi-layer stacking and feature extraction: The attention layers mentioned above can be stacked in multiple layers (e.g., 6 or 12 layers), allowing the model to progressively extract higher-level semantic features and long-range dependencies. Each layer maintains the same causality and occlusion constraints, ensuring the temporal correctness of information flow. After multiple layers of processing, the output Y_last of the last layer is also a value of length [missing information]. A sequence of dimension d, where each position corresponds to a deep representation of a certain modality at a certain time.

[0100] To obtain the global state at the current moment, we need to extract the tokens of all modalities related to the current moment t from the sequence. Since the sequence is organized primarily in time order (first by time, then by modality), the token corresponding to the current moment t is located at the end of the sequence. Each location. After retrieving these tokens, they are aggregated into a fixed-dimensional world state latent variable h_t through pooling operations. The pooling method can be selected as follows: Average pooling: Takes the average value of all modal tokens, suitable for scenarios where the importance of each modality is balanced.

[0101] Max pooling: takes the maximum value across all dimensions, which helps to capture the most significant features.

[0102] Attention pooling: Introduces a learnable query vector and performs a weighted summation of each modality token, enabling the model to dynamically select key modality information.

[0103] For example, if average pooling is used, then ,in, : World state latent variable, t represents the current time, is the final result of the occlusion temporal attention encoder, and is used for subsequent morphological fusion and action generation.

[0104] : Represents the total number of modes (i.e., the size of the mode set). This is the vector corresponding to time t and mode m in the output of the last layer of attention.

[0105] Optional implementation methods: Relative position bias: Learnable relative position bias can be added to the attention score to replace or supplement the absolute position encoding, enabling the model to better perceive the time interval between different tokens.

[0106] Sparse attention or linear attention: To reduce the computational complexity of long sequences, sparse attention mechanisms (such as local window attention) or linear attention approximations (such as Performer and Linformer) can be used to improve efficiency while maintaining causality, which is suitable for high-frequency control scenarios.

[0107] Multi-scale feature fusion: Skip connections can be introduced between different layers, or multi-layer outputs can be concatenated and then pooled to make comprehensive use of shallow details and deep semantics.

[0108] By introducing causal masks, the model is ensured to strictly follow temporal causality, meeting the real-time requirement of "relying only on historical information" in online control. By superimposing occlusion masks, the model actively ignores unreliable modal information during attention calculation, significantly improving robustness to anomalies such as sensor occlusion and frame loss. Multi-layer attention stacking enables the model to gradually fuse complex dependencies across time and modalities, extracting rich contextual representations. Finally, pooling operations are used to obtain fixed-dimensional world state latent variables, providing a unified, compact, and information-complete input for subsequent morphological fusion and action decision-making, laying a solid foundation for constructing a general and robust legged robot control strategy.

[0109] Optionally, the cross-morphological fusion module includes a morphological map encoder and a feature fusion unit; the step of inputting the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables includes: The morphological graph encoder is constructed based on a graph neural network or a morphological Transformer encoder; A morphological diagram is constructed using the kinematic topology and dynamic parameters of the legged robot. The nodes of the morphological diagram represent joints or links, and the edges of the morphological diagram represent connection relationships. The node attributes include at least one of the following: link length, mass, moment of inertia, joint type, and joint limit. The morphology graph is input into the graph neural network or the morphology Transformer encoder to obtain a morphology token sequence; The morphological token is fused with the world state latent variable using the feature fusion processor to obtain the morphological conditional latent variable.

[0110] Optionally, the step of fusing the morphological token with the world state latent variable through the feature fusion processor to obtain the morphological conditional latent variable includes: Based on feature linear modulation or cross-attention mechanism, the morphological token sequence is injected into the world state latent variable to obtain the morphological conditional latent variable.

[0111] Specifically, the cross-morphological fusion module is the core component for enabling a single policy network to adapt to different legged robot morphologies. Its function is to encode the robot's physical structure information into morphological features and deeply fuse them with latent world-state variables extracted from multimodal perception, generating morphologically conditional latent variables. This allows the subsequent general policy head to output joint-level control commands adapted to the current robot body. This module consists of two parts: a morphological map encoder and a feature fusion unit. The specific processing flow is as follows: I. Morphological Diagram Construction: First, the physical structure of the legged robot is abstracted into a graph structure representation, called a morphological graph. The definition of the morphological graph G=(V,E) is as follows: Node V: Each node represents a joint or link in the robot. For complex robot structures, you can choose to use joints as the main nodes and links as their attributes, or links as the main nodes and joints as the connection relationships. For example, for a typical quadruped robot, each leg includes a hip joint, knee joint, ankle joint, and corresponding links, plus the torso, which can form a graph of about 13-15 nodes.

[0112] Edge E: Each edge represents the physical connection between two nodes, typically a rigid connection or a rotational / sliding joint between parent and child nodes. The directionality of the edge reflects the hierarchical relationship of the kinematic tree.

[0113] Node attribute f_v: Assigns a set of quantized attributes to each node to describe its physical characteristics. For example, attributes may include: link length l (scalar), mass m (scalar), moment of inertia J (3×3 matrix or simplified diagonal matrix), joint type type (one-hot encoding of a rotary / moving / fixed joint), joint range of motion [q_min, q_max] (lower and upper limits), actuator rated torque τ_max (scalar), foot contact geometry parameters (such as foot radius, shape encoding), etc. For link nodes without joints, the corresponding attributes can be set to zero or use special markers.

[0114] For example, a simplified leg structure may have a morphological diagram containing three nodes: hip joint (with attributes: type=rotation, limit [-30°, 90°], torque 50Nm), thigh link (with attributes: length 0.3m, mass 2kg), knee joint (with attributes: type=rotation, limit [0°, 120°], torque 40Nm), lower leg link (with attributes: length 0.3m, mass 1.5kg), and foot node (with attributes: radius 0.05m, shape code "spherical"). Edges are established between nodes based on kinematic connections.

[0115] II. Morphological diagram coding: The constructed morphology graph is input into the morphology graph encoder to generate a sequence of morphology tokens. The morphology graph encoder can be implemented using either a graph neural network or a morphology Transformer, each with its own characteristics and can be used interchangeably.

[0116] Optional methods include: 1. Graph Neural Network Encoding: A graph neural network is used to encode the morphological graph, and information is propagated between nodes through a message-passing mechanism, enabling each node to aggregate structural information from its neighborhood. For example, a graph convolutional network (GCN) or a graph attention network (GAT) can be used for this purpose. Initialize node characteristics (The initial feature vector obtained by linearly mapping the original attributes of node v).

[0117] For the l-th layer (l=0,1,2,…), the feature update of node v is: ; in, Let v represent the set of neighboring nodes. Let be the learnable weight matrix of the l-th layer, AGG be the aggregation function (summation, averaging, or attention weighting), and σ be the non-linear activation function. Stacking multiple layers (such as 2-3 layers) allows information to propagate through the graph, ultimately yielding the deep features of each node. .

[0118] Arrange the features of all nodes in topological order (e.g., from torso to feet) to obtain the morphological token sequence. , where n is the number of nodes. Represents a sequence of tokens in terms of form. To represent the i-th node The feature vector obtained after encoding by an L-layer graph neural network; 2. Morphological Transformer Encoding: The Transformer architecture is used to encode the shape graph, treating nodes as sequence elements and capturing the global dependencies between all nodes through a self-attention mechanism.

[0119] For example: Linearly map the node attribute f_v to the initial node token and add a learnable positional encoding to represent the node's topological position in the graph (e.g., using depth-first traversal order encoding).

[0120] Input a standard Transformer encoder (multi-layer multi-head self-attention + feedforward network) to enable each node token to pay attention to information from all other nodes in the graph.

[0121] After multi-layer encoding, the deep features of each node are output and arranged in a predetermined order to obtain the morphological token sequence. .

[0122] The advantage of the morphological Transformer lies in its ability to capture non-local dependencies, such as cooperative relationships between distant joints; while the advantage of GNNs lies in their explicit encoding of the graph's topology, resulting in a stronger inductive bias towards local connections. The choice between the two approaches depends on specific needs.

[0123] Regardless of the method used, the final output will be a token sequence. The dimension is n×d_m, where d_m is the dimension of the morphological token, which usually matches the dimension required for subsequent fusion.

[0124] III. Feature Fusion: morphological token sequence The world state latent variable h_t (output by the occlusion temporal attention encoder) is input into the feature fusion processor to obtain the morphologically conditional latent variable h'_t. The feature fusion processor can employ the following two main mechanisms: Option 1: Characteristic linear modulation: Feature-linear modulation (FLEM) is a global conditionalization method that performs a feature-dimensional affine transformation on the latent variables of the world state by learning scaling and offset parameters from a sequence of morphological tokens. For example: morphological token sequence The vectors are aggregated into a single global morphological vector, z_morph. Aggregation can be achieved using average pooling, max pooling, or attention pooling. For example, z_morph can be input into two small MLPs to generate scaling parameter γ and offset parameter β, respectively, with the same dimensions as h_t: γ = MLP_γ(z_morph), β = MLP_β(z_morph).

[0125] Modulate h_t: h'_t = γ⊙h_t + β, where ⊙ represents element-wise multiplication.

[0126] This modulation method is equivalent to a global "recalibration" of the perceived world state based on the robot's overall morphological characteristics. For example, robots of different masses should have different center of gravity adjustment strategies when facing the same terrain, and γ and β can learn this morphological adjustment.

[0127] Option 2: Cross-attention mechanism: Cross-attention is a finer-grained fusion approach that allows latent variables of the world state to selectively focus on different nodes in a sequence of morphological tokens, capturing the importance of specific components to the current decision. For example: Using the world state latent variable h_t as the query Q, and the morphological token sequence As the key K and value V, Q = h_t × W_Q is obtained through linear projection. , Among them, W_Q, W_K, These are the learnable weight matrices.

[0128] Calculate the attention score A to obtain the attention weight of each morphological node to the current world state.

[0129] Calculate the weighted sum h_att to obtain the features aggregated from a morphological perspective.

[0130] The original h_t and h_att are merged to obtain h'_t. The fusion method can be a linear transformation after concatenation, or direct addition: h'_t = h_t + h_att.

[0131] The advantage of cross-attention is that it can dynamically select the morphological components most critical to the current movement task. For example, when a robot is preparing to cross an obstacle, attention may be focused on the joints and links of the forelegs; when turning, attention may be focused on the torso and the joints related to the turn.

[0132] If cross attention is used, the dimensions of h_t can be expanded before calculation to align with the length of the morphological token sequence (e.g., by copying it multiple times), or cross attention can be calculated directly between a single vector and the sequence.

[0133] To enhance the fusion effect, multiple layers of cross-attention modules can be stacked to gradually refine the fusion representation.

[0134] Residual connections and layer normalization can be introduced to stabilize the training process.

[0135] By constructing the robot morphology description as a graph structure and encoding it using a graph neural network or morphology Transformer, key physical information such as kinematic topology, link parameters, mass, and inertia can be fully preserved. This heterogeneous structured data is transformed into a unified-dimensional sequence of morphology tokens, enabling subsequent fusion modules to flexibly handle robot morphologies with different degrees of freedom and topologies. Through feature linear modulation or cross-attention mechanisms, morphology tokens are deeply integrated with latent world state variables, allowing perceptual and ontological information to interact organically at the feature level. Morphological features can dynamically modulate perceptual representations according to environmental context, and perceptual representations can selectively focus on morphological components related to the current motion decision. The resulting morphology-conditional latent variables simultaneously contain environmental perceptual information and the robot's own physical constraints, enabling the general strategy head to output joint-level commands adapted to different robot morphologies based on a unified representation space. This achieves universal adaptation of a single strategy network to various legged robots, including quadrupeds, bipeds, and wheeled robots, significantly reducing the deployment cost and control strategy development cycle of new robot morphologies.

[0136] Optionally, the control method for the general-purpose legged robot further includes: Based on the quality indication and cross-modal reconstruction residuals, the reliability of each mode at the current moment is determined by a reliability estimator; Dynamically adjusting the masking code based on the reliability includes: When the reliability of any of the modalities is lower than a preset threshold, the corresponding modal token is replaced with a learnable mask token, and the weight of the modality is reduced or masked in the attention calculation.

[0137] Specifically, this process aims to provide an adaptive adjustment mechanism based on real-time data quality for the masking-time attention encoder. This allows the model to dynamically evaluate the reliability of sensor information from each modality during the inference phase and adjust the masking mask accordingly, thereby maintaining stable control performance even in the event of sensor data degradation, environmental interference, or hardware failure. Specific steps include: I. Obtaining Quality Indicators: Quality indicators are quantitative metrics extracted directly from the sensor's raw data, reflecting its current operating status. For different types of sensors, quality indicators take different forms.

[0138] For example, in depth cameras, a quality indicator can be the effective pixel ratio, which is the proportion of pixels with valid depth values ​​out of the total pixels. When a robot faces strong light, highly reflective surfaces, or distant targets, the effective pixel ratio may drop significantly, indicating that the depth data is largely unreliable. Furthermore, the temporal consistency of the depth map can be calculated, i.e., the smoothness of depth changes between adjacent frames; abrupt changes may indicate data anomalies.

[0139] LiDAR: Quality indicators can include the proportion of abnormal echo intensity, i.e., the percentage of points where the echo intensity exceeds the normal range. In environments such as rain, snow, fog, haze, or on highly reflective surfaces (e.g., mirrors, water surfaces), echo intensity may become abnormal, leading to inaccurate ranging. Point cloud density can also be statistically analyzed; a sudden drop in density may indicate that part of the sensor's field of view is obstructed or that ranging has failed.

[0140] RGB cameras: Quality indicators can include image blur (estimated via Laplacian variance), the proportion of overexposed / underexposed areas (the proportion of pixel values ​​close to 0 or 255), histogram entropy, etc. For example, when the lens is obscured by mud, local areas of the image may appear blurry or lack texture; blur indicators can effectively capture such anomalies.

[0141] IMU: Quality indicators may include saturation flags (triggered when the measured value exceeds the range), temperature drift estimates, vibration amplitude, etc. For example, the IMU may temporarily saturate under severe impact, at which point the data is unreliable.

[0142] Joint sensors: Quality indicators may include encoder reading validity flags, torque sensor overload indications, communication timeout counts, etc. For example, torque data may be distorted when the joint torque exceeds the rated range.

[0143] The aforementioned quality indicators are typically calculated in real time by the sensor driver or the underlying preprocessing module and output in scalar form. The range is generally normalized to the [0,1] interval, and the higher the value, the better the data quality.

[0144] II. Calculation of cross-modal reconstruction residuals: Cross-modal reconstruction residuals are a quality metric based on internal model consistency. They assess the reliability of modal data by comparing the actual observations of a particular modality with the expected values ​​predicted based on other modalities and historical information. The specific implementation is as follows: In the forward computation of the occlusion-sequential attention encoder, for each modality at each time step, the model internally maintains a prediction value for the observations of that modality. This prediction value can be derived from two sources: History-based self-prediction: Using the same modal data from historical moments, predict the current moment's observations of that modality through time series models (such as recurrent units or attention).

[0145] Cross-modal prediction: Using data from other modalities at the current time (such as IMU and joints) and historical information, a lightweight decoder is used to predict observations of that modality at the current time (e.g., predicting the contour or key features of the depth map from IMU and joint states).

[0146] The predicted values ​​are compared with the actual observed values ​​to calculate the reconstruction residuals. The residuals can be measured in various ways: For continuous modes (such as depth, image features), mean square error or L1 distance can be used.

[0147] For discrete modes (such as ground contact signals), cross-entropy or classification loss can be used.

[0148] For high-dimensional data (such as images), residuals can be computed in the feature space rather than the pixel space. That is, features are extracted first by an encoder, and then the differences between the feature vectors are compared.

[0149] A larger residual value indicates a greater discrepancy between the actual observations of the current mode and the model's predictions based on contextual information, potentially indicating anomalies in sensor data or sudden environmental changes. This residual, along with the quality indicator, constitutes the input for reliability estimation.

[0150] III. Construction and Reasoning of Reliability Estimator: The reliability estimator is a lightweight neural network module that integrates quality indicators and cross-modal reconstruction residuals to output the reliability score for each modality at the current time.

[0151] For example, the following structure can be used: Input layer: Quality indicators Reconstruction of residuals The time offset of this mode from the master clock (Used to perceive the degree of asynchrony) and the latent variable h_t-1 from the previous time step (providing context) are concatenated into an input vector. .

[0152] Hidden layers: Single or double MLPs are used, each containing 64 to 128 neurons, with ReLU as the activation function.

[0153] Output layer: The sigmoid activation function is used to map the output to the [0,1] interval to obtain the reliability. , where 1 represents completely trustworthy and 0 represents completely untrustworthy.

[0154] The reliability estimator can be jointly optimized with the entire policy network during the training phase, and its supervision signals can come from: Self-supervised signal: For occlusion imposed manually during training, the reliability of the occluded position should be close to 0, and that of the unoccluded position should be close to 1.

[0155] Auxiliary loss: A reliability prediction loss is introduced to align the reliability of the estimator output with the backpropagation signal based on the performance of subsequent tasks.

[0156] IV. Reliability-based dynamic masking adjustment: Based on the reliability output by the reliability estimator, the masking code M_pred is dynamically adjusted, specifically including the following two aspects: 1. Modal token replacement: When the reliability of a certain mode is lower than a preset threshold θ (e.g., θ = 0.3), the mode token corresponding to that mode is... Replace it with a learnable mask token vector x_mask. This mask token has the same dimension as the modality token and is optimized along with other network parameters during training. Through this replacement, the model will receive a placeholder indicating "this modality is unavailable" in subsequent attention calculations, rather than potentially misleading invalidation data.

[0157] 2. Adjustment of attention weights: Even if the reliability is not lower than the threshold, a soft adjustment mechanism based on reliability can be introduced into the attention calculation. Specifically, when superimposing the masking code on the attention score, the reliability can be added as a bias term in logarithmic form. When the reliability is close to 0, the weight approaches 0; when the reliability is close to 1, the bias approaches 0 and does not affect the original score.

[0158] Furthermore, for modalities that are replaced with mask markers, their corresponding masking mask positions are directly set to -∞, completely shielding them from participation in attention calculation.

[0159] It should be noted that the optional implementation methods and variations include: reliability fusion method: in addition to the above logarithmic bias, reliability can also be directly applied as a multiplicative factor to the attention weight, or as a gating signal to control the inflow of modal information.

[0160] Threshold Adaptive: The preset threshold can be dynamically adjusted according to the modal type or task stage. For example, the threshold can be increased during high-speed motion, which places higher demands on the quality of the sensor.

[0161] Time series smoothing: To prevent control jitter caused by drastic changes in reliability at adjacent time points, the reliability series can be smoothed by exponential moving average.

[0162] Multi-scale reliability: For high-dimensional modalities (such as images), region-level reliability can be calculated (e.g., dividing the image into blocks and evaluating each block independently), rather than a single global reliability, enabling finer-grained masking adjustment.

[0163] Reliability feedback: Reliability is added as an additional feature to the latent variables of the world state, enabling subsequent policy heads to perceive the credibility of the currently perceived information and thus make more conservative or more aggressive decisions.

[0164] Taking a robot operating in a dusty environment as an example: due to dust interference, the proportion of abnormal echo intensity of the lidar increases, and the quality indication q drops from 0.9 to 0.4; at the same time, the difference between the lidar point cloud predicted by the model based on IMU and depth history information and the actual point cloud increases, and the reconstruction residual... The reliability estimator combines both metrics, outputting a laser mode reliability r = 0.25, below the threshold of 0.3. The laser token at that moment is replaced with a learnable mask marker, and its weights are masked during attention calculation. The model then relies on depth camera and IMU information to maintain environmental awareness. When the robot leaves the dusty area, laser data recovers, reliability rebounds, and the system automatically reactivates the laser mode.

[0165] By introducing a reliability estimator based on quality indicators and cross-modal reconstruction residuals, the model can evaluate the reliability of sensor information from each modality in real time during the inference phase and dynamically adjust the masking code accordingly. Unreliable modal tokens are replaced with learnable masking tags, and their weights are reduced or masked during attention calculation, achieving adaptive robust control against sensor data degradation. This mechanism enables the policy network to automatically reduce the weight of failed modes in various real-world scenarios such as depth camera overexposure, LiDAR interference, IMU saturation, and image blurring, relying on the remaining reliable information to maintain stable operation, significantly improving the robot's survivability and control reliability in complex environments. At the same time, the reliability estimator is jointly optimized with the main network, enabling it to learn the correlation between data quality and task performance, providing contextual information on perceived reliability for subsequent decisions, further enhancing the system's security and interpretability.

[0166] Optionally, the occlusion temporal attention encoder and the cross-morphological fusion module are obtained through three-stage training, which includes self-supervised occlusion modeling pre-training, multi-task control learning, and cross-morphological expert-student distillation. The self-supervised occlusion modeling pre-training includes: Acquire pre-collected time-series observation data from the multi-mode multi-sensor simulation data pool as historical training data; Random and structured masking are applied to the historical training data to generate a masked modal token sequence; The masked modal token sequence is input into the initialized masked temporal attention encoder. The original observation data of the masked part is used as the supervision signal to optimize the masking reconstruction loss, dynamic consistency loss and cross-modal alignment loss, and to train the pre-trained masked temporal attention encoder. The multi-task control learning includes: The parameters of the pre-trained occlusion temporal attention encoder are connected to the initialized cross-morphological fusion module and the initialized general policy head to form a preliminary policy network. In a multi-task reinforcement learning environment, the preliminary policy network interacts online with the environment to generate experience data. Based on the occlusion temporal attention encoder in the preliminary policy network, the modal token sequence at the current moment is input to obtain temporary world state latent variables; the morphological description information of the current legged robot form is input to the cross-morphological fusion module in the preliminary policy network to generate temporary form tokens; the temporary form tokens are fused with the temporary world state latent variables to obtain temporary form conditional latent variables, which are then input into the general policy head in the preliminary policy network to output actions. Based on the reward signals from environmental feedback, construct a reinforcement learning control loss; The gradient of the reinforcement learning control loss is backpropagated to the occlusion temporal attention encoder, cross-morphology fusion module and general policy head to jointly optimize the parameters. Modal randomization perturbation is continuously applied during training to learn motion control strategies that adapt to different robot shapes, and the occlusion temporal attention encoder, cross-morphology fusion module and general policy head are obtained after preliminary joint training. The cross-morphological expert-student distillation includes: Multiple expert policy networks are obtained, and the action distributions generated by each expert policy network under various states are used as soft labels. The occlusion temporal attention encoder, cross-morphological fusion module, and general policy head, which are preliminarily jointly trained, constitute a unified student policy network. The soft label is used as a supervision signal to optimize the distillation loss so that the action distribution output by the unified student policy network is the same as the action distribution of each expert policy network. During the distillation process, the occlusion reconstruction loss is jointly optimized, and the occlusion temporal attention encoder, the cross-morphological fusion module, and the general policy head are trained.

[0167] Specifically, this three-stage training process aims to systematically train the occlusion-sequential attention encoder, cross-morphology fusion module, and general strategy head, ultimately enabling them to form a unified, cross-morphology, and cross-sensor generalized legged robot control strategy. The three stages focus on perceptual representation learning, control strategy optimization, and knowledge transfer generalization, respectively, progressing layer by layer and co-optimizing to achieve end-to-end mapping from raw perceptual input to joint-level control commands.

[0168] I. First Stage: Self-Supervised Occlusion Modeling Pre-training: Phase objective: To enable the masked temporal attention encoder to learn to recover complete state representations under partial observation conditions, establish cross-modal and cross-temporal perception and understanding capabilities, and provide high-quality initialization parameters for subsequent control learning.

[0169] Data source: Pre-collected time-series observation data from a multi-morphology, multi-sensor simulation data pool. This data pool contains sensor records of various legged robot morphologies (such as quadrupeds, bipeds, and wheeled robots) walking on various terrains (flat ground, slopes, steps, obstacles, etc.). Data modalities cover joint states, IMU, depth maps, laser point clouds, foot contact data, etc. This data requires no manual annotation and can be obtained simply by random collection in the simulation environment.

[0170] Processing procedure: 1. Data Augmentation and Masking: Random and structured masking are applied to historical time-series observation data sampled from the data pool to generate masked modal token sequences. Masking strategies include: Random modal occlusion: Occludes certain modalities at certain times with a certain probability.

[0171] Random time-segment occlusion: Occlusion of the same mode across multiple consecutive frames simulates sensor disconnection.

[0172] Structured occlusion: Simulates real-world scene occlusion, such as lateral view occlusion, local area occlusion, and distance segment occlusion.

[0173] For example, for a trajectory containing 10 time points and 3 modalities (IMU, joint, depth), the depth map at time t-5 and the IMU data from t-2 to t-4 for three consecutive frames may be randomly occluded. The occluded locations will be replaced with learnable mask markers, and the model will need to use the unoccluded information to complete the mapping in subsequent processing.

[0174] 2. Forward Computation and Loss Optimization: The masked modal token sequence is input into the initialized masked temporal attention encoder. After multi-layer causal attention computation, the world state latent variable h_t is output, and the masked parts are reconstructed and predicted using multiple decoders. The optimization objective includes three types of loss: Covering and rebuilding losses For occluded modal tokens, calculate the difference between the model's predicted values ​​and the original occluded observations. For example, for an occluded depth map region, the model needs to predict the features of that region and calculate the L2 loss against the actual features.

[0175] Dynamic consistency loss This requires latent variables to satisfy short-term predictive consistency. For example, predicting latent variables for the next time step based on h_t and the current action a_t. _{t+1}, making it close to the true h_{t+1}: L dyn =|| _{t+1}-stopgrad(h_{t+1})||².

[0176] Cross-modal alignment loss This aligns the representations of different modalities at the same time in the latent space. For example, for IMU and joint features at the same time, contrastive learning can be used to bring them closer together in the latent space, enhancing semantic consistency between modalities.

[0177] 3. Stage Output: Through self-supervised training on a large amount of simulation data, a pre-trained occlusion-based temporal attention encoder with robust perception capabilities was obtained. This encoder has learned to complete the state representation using historical information and other available modalities when some observations are missing, laying the foundation for subsequent control learning.

[0178] II. Second Stage: Multi-task Control Learning Phase goal: Under the reinforcement learning framework, enable joint optimization of the occlusion temporal attention encoder, cross-morphology fusion module and general policy head to learn motion control strategies that can adapt to robots of different shapes, while maintaining robustness to perception loss.

[0179] Network initialization and connection: Occlusion Temporal Attention Encoder: Loads the parameters obtained from the first stage of training as initialization.

[0180] Cross-morphological fusion module: Random initialization.

[0181] General strategy header: Random initialization.

[0182] Connect the three in sequence to form a preliminary policy network. Its data flow is as follows: modal token sequence → MTAN → world state latent variable → MFM (fused with morphological token) → morphological conditional latent variable → policy head → action.

[0183] Training environment and data generation: In a multi-task reinforcement learning environment, an initial policy network is deployed. In each training round, a robot form (sampled from the form set) and a task type (such as velocity tracking, step crossing, perturbation recovery, etc.) are randomly selected.

[0184] The policy network interacts online with the environment: In each control cycle, the current observation is acquired, and actions are generated through MTAN and MFM, which are then sent to the environment for execution. The environment returns the next time-instance observation and immediate reward.

[0185] The experience data (state, action, reward, next state) generated during the interaction is stored in the experience replay buffer.

[0186] Forward computation details: Based on the occlusion-time attention encoder in the initial policy network, the modal token sequence at the current time is input to obtain the latent variables of the world state (at this time, the encoder parameters are pre-trained values, but are still updated during training).

[0187] The morphological description information of the current legged robot form is input into the cross-morphological fusion module in the preliminary policy network to generate a morphological token (at this time, the MFM is randomly initialized and is learning how to encode morphological information).

[0188] The morphological tokens and world state latent variables are fused in MFM (through FiLM or cross attention) to obtain morphological conditional latent variables.

[0189] Input the morphologically conditional latent variables into the general strategy head, and output the action (action distribution parameters or deterministic action).

[0190] Loss calculation and parameter update: Based on the reward signals from environmental feedback, a reinforcement learning control loss is constructed using reinforcement learning algorithms (such as PPO and SAC). This loss typically comprises two parts: strategy loss and value loss, and is designed to maximize the cumulative expected reward.

[0191] Will The gradient is backpropagated to the occlusion-temporal attention encoder, the cross-morphological fusion module, and the general policy head, and the parameters of the three are jointly optimized. This means that controlling the loss simultaneously drives the perception module (MTAN) to extract features that are more conducive to decision-making, the morphological fusion module (MFM) to learn more effective morphological representations, and the policy head to output better actions.

[0192] During training, modal randomization perturbations are continuously applied: certain modalities are randomly masked in the input with a certain probability, or noise is added to the sensor data, so that the policy network is continuously exposed to the distribution that is unavailable to the sensor during training, thereby learning to control robustly under conditions of missing information.

[0193] Phase outputs: After multi-task reinforcement learning training, a preliminary jointly trained occlusion temporal attention encoder, cross-morphological fusion module, and general policy head were obtained. The three components have initially formed the ability to work collaboratively, enabling effective control in various morphologies and tasks, and exhibiting a certain degree of robustness to sensor absence.

[0194] III. Third Stage: Cross-Morphology Expert-Student Distillation: Phase goal: Through knowledge distillation, transfer the knowledge of multiple high-performance expert policies trained for specific morphologies and sensor combinations to a unified student policy network, further improving the generalization ability and performance ceiling of the student policy, while maintaining cross-morphology and cross-sensor versatility.

[0195] Expert policy preparation: High-performance expert policy networks are trained or acquired for different robot forms (e.g., quadruped A, quadruped B, bipedal C, wheel-legged D) and different sensor combinations (e.g., depth only, depth + laser, body only, etc.). These expert policies can be trained using privileged information (e.g., terrain ground truth, contact ground truth, future trajectory, etc.), thus achieving near-optimal performance. For example, training one expert for quadruped robot R1 can use a complete height map and high-precision state estimation; training another expert for bipedal robot R2 can use multi-view depth and accurate foot force information.

[0196] By leveraging expert policy networks to perform forward reasoning across multiple states, action distributions are generated as soft labels. Compared to hard labels (such as a single manually labeled action), soft labels contain richer probabilistic information and can convey the expert's uncertainty about the state and action preferences. For example, on a dataset containing 5000 different terrains and robot states, each expert outputs parameters of their action distribution (such as the mean and variance of a Gaussian distribution, or a probability vector of discrete actions).

[0197] Student network initialization: The occlusion-sequential attention encoder, cross-morphological fusion module, and general policy head obtained from the initial joint training in the second stage constitute a unified student policy network. The structure of this network may differ from that of the expert policy (the student network must use a uniform input format, while the experts may use different sensor combinations), but the output space of the student network is consistent with that of each expert (both output joint-level control commands).

[0198] Distillation process: For each training sample (containing multi-source sensor observations, morphological description information, and task instructions), the student network performs forward computation and outputs its action distribution. .

[0199] Simultaneously, based on the expert type corresponding to the sample (i.e., the type and sensor combination from which the sample originates), the action distribution of the expert in that state is obtained. As a monitoring signal.

[0200] Constructing distillation loss The KL divergence is typically used to measure the difference between two distributions. ; in, Knowledge Distillation Loss. Summation of sample index or feature dimension. The expectation operator typically represents the expectation under a state distribution or trajectory. KL divergence is a measure of the difference between two probability distributions. The expert network, or teacher policy model, is in a state... The distribution of action outputs. Student Policy model. Besides observing the state... It also received additional contextual information memory encoding. and environment embedding .

[0201] By minimizing this loss, the output distribution of the student network can be made to approximate the output distribution of the expert network.

[0202] Total loss function: ; in, To reinforce learning control loss and measure the policy's performance on a task, it is typically composed of policy gradient loss and value network loss, aiming to maximize cumulative reward. To mitigate reconstruction loss, during the self-supervised pre-training phase, the model needs to predict the occluded sensor observations, forcing the model to learn to fill in the missing data from the remaining information. To ensure dynamic consistency loss, we must ensure that the latent variables of the world state can accurately predict the latent variables at the next moment, so that the representation has temporal coherence. Cross-modal alignment loss is used to bring the representations of different modalities (such as vision and IMU) closer to each other in the latent space at the same time, thereby promoting modality fusion. To mitigate the distillation loss, as described above, the student policy is made to approximate the output distribution of the expert policy. To mitigate safety risks, penalties or projections are applied to movements to ensure that joint positions, speeds, torques, etc., meet safety boundaries and prevent collisions or falls. Regularization terms are used to prevent overfitting and typically include parameter norm penalties.

[0203] Weight Phased scheduling: Phase A (pre-training): Improvement The focus is on learning perceptual representation.

[0204] Phase B (Controlled Learning): Improvement and It focuses on control performance and safety.

[0205] Stage C (Distillation): Improvement At the same time, maintain a certain Prevent forgetting sensory abilities.

[0206] This dynamic adjustment allows the network to focus on different learning objectives at different stages, ultimately achieving a unified strategy that balances perception, control, and generalization.

[0207] The loss function is a mathematical expression of the optimization objective during training, with each component constrained for a specific capability. Through multi-stage weight scheduling, progressive learning is achieved, moving from basic perception to advanced control and then to knowledge transfer. Distillation loss is key to achieving cross-morphology / cross-sensor generalization; it leverages privileged expert knowledge to guide a unified student network, enabling it to achieve expert-level performance even with limited perception.

[0208] Joint optimization of shading reconstruction loss during distillation process That is, while distilling, the masking reconstruction task continues to be applied to the masking temporal attention encoder of the student network to prevent it from forgetting the perceptual completion ability learned in the first stage during the distillation process. This is achieved through an auxiliary decoding head, so that the encoder can still maintain its ability to reconstruct missing information when updating parameters.

[0209] Distillation Details and Techniques: Modal Alignment: Since experts may use different sensor combinations, and student networks need to uniformly process all possible inputs, expert inputs need to be aligned during distillation. For example, if expert A uses depth + laser, while the student network only has depth input (laser is masked), then when calculating distillation loss, only the student network output should be compared with the expert's output under the same observation conditions.

[0210] Course distillation: Start with simple expert samples (such as walking on flat ground) and gradually introduce expert samples with complex terrain and challenging scenarios to enable students to learn in a step-by-step manner online.

[0211] Multi-expert fusion: When there are multiple experts (such as experts of different types) in the same state, a weighted average or attention mechanism can be used to fuse the expert distribution as a supervisory signal for the student network.

[0212] Regularization and stability: In the early stages of distillation, the weight of distillation loss can be appropriately reduced to allow the student network to retain its original exploration ability, and then the distillation intensity can be gradually increased in the later stages.

[0213] Phase outputs: After distillation and optimization, the final trained occlusion-based temporal attention encoder, cross-morphological fusion module, and general policy head are obtained. This unified policy network can output high-quality actions close to expert level under various morphologies and sensor configurations, while maintaining robustness to perceptual loss and environmental changes.

[0214] Among them, the optional implementation methods and variations are as follows: Two-stage simplification: If resources are limited, the first stage can be omitted and training can start directly from the second stage, but some perceptual robustness may be sacrificed; or the third stage can be omitted and the network trained in the second stage can be deployed directly, but the performance may be insufficient on some complex tasks.

[0215] Alternating training: In the second stage, reinforcement learning updates and self-supervised updates can be performed alternately (simultaneously optimizing). and This enables the perception representation to continuously adapt to the control task.

[0216] Progressive morphological expansion: In the second and third stages, a course-based learning approach can be used to gradually increase the variety and difficulty of morphological types. For example, first train quadrupedalism, then add wheeled legs, and finally add bipedalism, so that the network can gradually learn a wider range of morphological adaptations.

[0217] Online distillation: After the second stage, expert strategies and student strategies can interact with the environment in parallel, generating supervision signals in real time for online distillation, enabling the distillation process to adapt to the environment.

[0218] Meta-learning initialization: The first stage of pre-training can be regarded as a kind of meta-learning initialization, so that the subsequent second stage can adapt to new tasks and new forms more quickly.

[0219] Through a three-stage training process, a systematic capability building process was achieved, encompassing perceptual representation learning, control strategy optimization, and knowledge transfer. The first stage, self-supervised occlusion pre-training, enables the model to learn to complete state information from partial observations without manual annotation, providing a high-quality perceptual foundation for subsequent control learning. The second stage, multi-task control learning, jointly optimizes the perception and control modules, allowing the model to learn to map environmental perception into specific actions in real-world interactions, and enhances robustness against sensor failure through continuous modal perturbation. The third stage, cross-morphology expert-student distillation, compresses the knowledge of multiple high-performance expert strategies into a unified student network, enabling the student to achieve near-expert-level performance under various morphological and sensor configurations, while jointly optimizing occlusion reconstruction loss to prevent perceptual degradation. These three progressively advancing and collaboratively optimized stages result in a unified policy network that not only possesses universal adaptability across morphologies and sensors but also maintains robust control in complex environments such as occlusion, frame drops, and noise. This significantly reduces the development and deployment costs of legged robot control strategies, providing crucial technical support for the large-scale application of robots in diverse scenarios such as industrial inspection, disaster relief, and logistics handling.

[0220] Optionally, the occlusion-time attention encoder also outputs uncertainty; the control method for the general-purpose legged robot further includes: The weights of the desired speed command or safety constraints are adaptively adjusted based on the uncertainty.

[0221] Specifically, the occlusion-sequential attention encoder outputs not only the latent variables of the world state but also a corresponding uncertainty estimate, which quantifies the confidence level of the current latent variables. Based on this uncertainty, the weights of the desired velocity command or safety constraints are adaptively adjusted, enabling the robot to proactively adopt conservative strategies when perceptual information is ambiguous, the environment is complex, or the state estimate is unreliable, significantly improving the system's safety and robustness.

[0222] I. Uncertainty Estimation and Output: Uncertainty Uncertainty is a quantitative measure of the confidence level of the latent world state variable h_t, reflecting the reliability of the model's estimate of the current state. Uncertainty can be estimated in several ways: Option 1: Probability Output Header After the output layer of the masked temporal attention encoder, a separate uncertainty prediction branch is added. For example, the pooled features from the last layer are fed into two parallel fully connected layers, which output the mean of the latent variables respectively. and logarithmic variance : ; This branch can be optimized during training using negative log-likelihood loss, enabling the model to estimate its own prediction confidence intervals.

[0223] Option 2: Entropy based on attention distribution: The entropy of attention weights is used as a proxy for uncertainty. When attention weights are evenly distributed across modalities or historical time points, the entropy is large, indicating that the model lacks clear sources of information and has high uncertainty; when attention is highly concentrated on a few reliable modalities, the entropy is small, indicating that the model has strong confidence in the current state.

[0224] Option 3: Heuristic estimation based on reconstruction error: The cross-modal reconstruction residuals or occlusion reconstruction losses from step S300 are used as a measure of uncertainty. For example, the weighted average of the reconstruction errors of each mode at the current time is used as the uncertainty of the latent variable: Option 4: Integrating Uncertainty This method acquires multiple latent variable samples through multiple forward propagations (such as Monte Carlo sampling using dropout) and calculates their variance as uncertainty. While computationally intensive, this method effectively captures cognitive uncertainty in the model.

[0225] Regardless of the method used, the uncertainty of the final output Typically, it is a non-negative scalar or a vector corresponding to the dimension of the latent variable, used for subsequent adaptive adjustment.

[0226] II. Adjustment of expected speed command based on uncertainty: When uncertainty increases, it indicates that the model's estimation of the current environment or its own state is unreliable. In this case, the robot's movement speed should be reduced to allow more reaction time and reduce potential risks. Exemplary adjustment methods include: linear decay adjustment, piecewise adjustment, or exponential decay adjustment; the adjusted speed command. It is fed into the general strategy head as part of the high-level input, replacing the original expected velocity in action generation.

[0227] In some embodiments, This formula is used to adaptively adjust the desired speed command based on the uncertainty.

[0228] ; in, This represents the original expected velocity (from high-level mission planning). : The uncertainty at the current moment (output by the masking temporal attention encoder). The larger the value, the less certain the model is about the current state estimate. Adjusting the intensity coefficient controls the degree of influence of uncertainty on the speed. The minimum permissible speed for the robot (safety lower limit).

[0229] To be Limited to [ Within the range.

[0230] In this way, when uncertainty increases, the robot automatically reduces its speed and operates in a more conservative manner; when uncertainty decreases, the speed returns to normal. This helps improve safety when perception is ambiguous or the environment is complex.

[0231] III. Uncertainty-based weight adjustment of safety constraints: In addition to speed adjustment, the stringency of safety constraints can be dynamically adjusted based on uncertainty. When uncertainty increases, the weight of safety constraints should be increased to make the robot take more conservative actions and avoid risks such as collisions and falls. Safety constraints are usually implemented through quadratic programming or joint-by-joint saturation functions, projecting the original actions output by the strategy head into a set of safe actions.

[0232] Among them, safety constraint projection is used to project the raw joint commands output by the policy network. Revised to meet hardware safety restrictions and collision avoidance requirements for actual execution instructions. .

[0233] ; in, The variables representing safety actions to be solved; The square of the Euclidean distance between the original action and the safe action is used as the optimization objective, aiming to make the safe action as close as possible to the intent of the original strategy. A set of safe actions defined by hardware specifications (such as joint limits, maximum speed, and maximum torque) and real-time collision detection (such as self-collision and environmental obstacles); The goal is to find the smallest possible 'a' within the safe set, which is the feasible solution that is closest to the original action and satisfies all constraints.

[0234] The projection can be solved using numerical methods such as quadratic programming (QP) or joint-by-joint saturation functions to ensure that the robot always stays within the safety boundary when executing instructions, while preserving the control effect of the original strategy as much as possible.

[0235] The adjustment methods for constraint weights may include: dynamic adjustment of safety constraint weights. ; in, The basic safety constraint weights are β, which is the adjustment coefficient. These are the constraint weights used in practice. In quadratic programming solutions, these weights can be applied to the constraint relaxation term in the objective function or used as scaling factors for the constraint boundaries. Larger weights result in projected actions that are closer to the safety boundary, sacrificing some task performance for higher safety.

[0236] Adaptive tightening of joint limits: , ; in, , To limit the original joint position, , This is the contraction amplitude coefficient. When uncertainty increases, the joint limit actually used contracts inward to prevent the joint from moving to its limit position, reserving buffer space for emergencies. , It is the adjusted joint limit.

[0237] Torque limit adjustment: ; Where η is the adjustment coefficient. The original maximum allowable torque, This is the adjusted maximum permissible torque. When uncertainty increases, the maximum output torque is limited to prevent over-excitation caused by misjudgment of the state.

[0238] Soft constraint of foot contact force: In foot contact force control, the confidence interval of the desired contact force can be adjusted according to the uncertainty, or the foot landing margin in the foot trajectory generator can be adjusted so that the robot tends to choose a more conservative landing position.

[0239] In some embodiments, uncertainty and reliability are jointly adjusted: modal reliability can be combined with global uncertainty, for example, when the reliability of a critical mode (such as a depth camera) is low and the global uncertainty is high, a more aggressive deceleration strategy can be adopted.

[0240] Task-adaptive adjustment coefficients: Adjustment coefficients α, β, γ, etc. can be dynamically adjusted according to the current task type. For example, reduce deceleration sensitivity in high-speed running tasks and increase safety constraint weights in precision operation tasks.

[0241] Learning-based adjustment strategy: Uncertainty is used as an additional input feature, along with velocity instructions and safety constraints, and fed into a small policy network, allowing the model to learn how to adaptively adjust under different uncertainty levels, rather than relying on a fixed formula.

[0242] Time series smoothing: To prevent drastic changes in instructions caused by instantaneous fluctuations in uncertainty, an exponential moving average filter can be applied to the uncertainty sequence. . For the corrected uncertainty, This is the corresponding adjustment coefficient.

[0243] Multidimensional uncertainty: When uncertainty is a vector (corresponding to different dimensions of latent variables), different dimensions can be adjusted separately. For example, higher safety constraint weights can be assigned to attitude-related dimensions.

[0244] Taking a robot walking on unknown terrain as an example: when the robot enters a sloping area covered with fallen leaves from a flat surface, the depth camera's depth estimation becomes unstable due to the blurred edges of the fallen leaves, increasing the cross-modal reconstruction residual and uncertainty. It rose from 0.1 to 0.6. According to... The desired speed was linearly reduced from 1.0 m / s to 0.5 m / s; simultaneously, the safety constraint weight was increased from 1.0 to 1.8, causing the joint limits to contract inward by 5% and the maximum torque limit to decrease by 20%. The robot traversed the area at a slower speed and with a more conservative posture. Although the passage efficiency decreased slightly, it successfully avoided slipping or missteps caused by perceptual ambiguity. When the robot re-entered the flat area, the uncertainty dropped to 0.2, and the speed and constraint weights automatically returned to normal levels.

[0245] By introducing uncertainty estimation and adaptive adjustment mechanisms, the robot can proactively adopt conservative control strategies when perceptual information is ambiguous, the environment is complex, or state estimation is unreliable: reducing movement speed to allow more reaction time, while increasing safety constraint weights to limit the amplitude and force of movements, thereby significantly reducing the risk of collisions and falls in high-uncertainty scenarios; this mechanism does not rely on manual rules, but is based on the model's real-time assessment of its own confidence level, achieving a dynamic balance between safety and task efficiency; when uncertainty decreases, the system automatically returns to normal control mode, avoiding unnecessary conservative behavior; uncertainty itself can also serve as an interpretability indicator to diagnose perceptual degradation or environmental abrupt changes, providing a basis for subsequent fault analysis and strategy optimization, significantly improving the deployment safety and robustness of legged robots in complex real-world environments.

[0246] It should be noted that, without departing from the core idea of ​​this invention, this invention may also have the following various optional embodiments or extended modifications: (1) Encoder structure deformation: The occluded temporal attention encoder (MTAN) can adopt a variety of temporal modeling architectures, such as pure Transformer, a hybrid structure of Transformer and state space model, or a hierarchical temporal attention mechanism (combining short-term sliding window attention and long-term memory module) to adapt to different computing resources and real-time requirements.

[0247] (2) Modal encoder deformation: The encoder can be flexibly replaced for different types of sensor data: visual / depth modal encoders can use convolutional neural networks (CNN), visual transformers (ViT), or lightweight MobileViT; point cloud modal encoders can use PointNet, voxel convolution, or sparse convolution; proprioceptive modal encoders can use multilayer perceptrons (MLP) or temporal convolutional networks. In addition, encoders of different modalities can share some low-level parameters to reduce model size and improve computational efficiency.

[0248] (3) Latent motion space expansion: To further enhance cross-morphological sharing capabilities, a unified latent motion space can be introduced. After mapping the latent variables of the world state to this latent motion space, they can be converted into specific joint commands through a morphological conditional decoder. This approach is compatible with recent research directions in cross-morphological world models and latent motion learning, and helps to improve the transfer efficiency of strategies between robots with different degrees of freedom.

[0249] (4) Online adaptive capability: Under the premise of ensuring safety, a small number of parameters in the network (such as sensor configuration embedding layer, gating network) can be fine-tuned online or periodically to compensate for sensor calibration drift, environmental changes or differences in ground conditions. This adaptive process can use self-supervised reconstruction loss as the optimization objective, without the need for manual data labeling, which enhances the robustness of the system in long-term deployment.

[0250] (5) Task and Modal Expansion: The method of this invention is not limited to walking on flat ground, but can also be extended to more complex legged robot tasks such as running and jumping, crossing obstacles, climbing, and carrying objects (i.e., loco-manipulation). At the same time, the force / tactile sensor of the end effector can be added as a new modality and processed in a unified token sequence to further enrich the perception dimension and improve the execution capability of complex tasks.

[0251] In some specific embodiments, the performance of the method of the present invention in different scenarios is systematically verified through three progressive embodiments. Example 1 (Quadruped + Depth Camera / LiDAR Combination): A combined terrain including steps, slopes, stepping stones, narrow paths, and variable friction surfaces was set up in the simulation environment. The quadruped robot R_A was configured with S1=proprioception only, S2=proprioception + depth, S3=proprioception + LiDAR, and S4=proprioception + depth + LiDAR. The comparison methods were: (a) a conventional Transformer strategy without occlusion modeling; (b) a general strategy with only morphological conditionalization but without sensor gating; and (c) the method of this invention. Evaluation metrics included: success rate SR, average velocity tracking error E_v, energy consumption per unit distance C, number of falls F, and performance degradation rate Δ under different occlusion intensities.

[0252] The example evaluation results are shown in Table 1. In the simulation example, the occlusion ratio p_mask=0.4. The better the value, the higher the value is marked with ↑ / ↓.

[0253] Table 1 Example Evaluation Results Table

[0254] As shown in Table 1, under the same sensor combination, this invention significantly reduces the performance degradation caused by occlusion through occlusion modeling and reliability gating. When the sensor combination is reduced (only depth or blind walking), this invention still maintains acceptable throughput, demonstrating robustness across sensor combinations. Simultaneously, it achieves higher stability (fewer falls) under S4 full-sensor conditions. When unseen occlusion patterns or sensor frame drops are added to the test, this invention exhibits stronger extrapolation capabilities due to structured occlusion pre-training.

[0255] like Figure 2 As shown, an embodiment of the present invention provides a control system for a universal legged robot, comprising: The acquisition unit is used to acquire multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles. The embedding unit is used to uniformly tokenize the time-series observation data of the multi-source sensors to obtain a modal token sequence, and to perform embedding processing on each modal token in the modal token sequence, adding modal embedding, time embedding and sensor configuration embedding to obtain a modal token sequence after adding embedding. The processing unit is used to generate a masking code based on the preset masking strategy in the training phase or the quality indicator in the inference phase; input the embedded modal token sequence and the masking code into the masking temporal attention encoder, and output the world state latent variable; The control unit is used to acquire the morphological description information of the legged robot, input the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables, input the morphological conditional latent variables into the general strategy head, and output joint-level control commands to control the movement of the legged robot.

[0256] This invention provides a control device for a universal legged robot, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the control method for the universal legged robot as described above when the computer program is executed.

[0257] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the control method for a general-purpose legged robot as described above.

[0258] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A control method for a general-purpose legged robot, characterized in that, include: Acquire multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles; The time-series observation data from the multi-source sensors are uniformly tokenized to obtain a modal token sequence. Each modal token in the modal token sequence is then embedded by adding modal embedding, time embedding, and sensor configuration embedding to obtain a modal token sequence with added embeddings. Generate a masking code based on the preset masking strategy during the training phase or the quality indicators during the inference phase. The embedded modal token sequence and the occlusion mask are input into the occlusion temporal attention encoder, which outputs the world state latent variable. The morphological description information of the legged robot is obtained, and the morphological description information and the world state latent variables are input into the cross-morphological fusion module to obtain morphological conditional latent variables. The morphological conditional latent variables are then input into the general strategy head, and joint-level control commands are output to control the movement of the legged robot.

2. The control method for a universal legged robot according to claim 1, characterized in that, The step of unifying and tokenizing the multi-source sensor time-series observation data to obtain a modal token sequence includes: Based on the corresponding encoder, feature extraction is performed on the time-series observation data of each mode in the multi-source sensor time-series observation data to obtain the initial features of each mode; The initial features of each modality are linearly mapped to a vector space of uniform dimension to obtain the modality token of each modality at the current time. Arrange modal tokens from different times and modalities in chronological and modal order to form a modal token sequence.

3. The control method for a universal legged robot according to claim 1, characterized in that, The occlusion mask includes at least one of random modal occlusion mask, random time segment occlusion mask, and structured occlusion mask; wherein the structured occlusion mask is generated based on the occlusion distribution of the real scene, and the structured occlusion mask includes at least one of lateral view occlusion, local area occlusion, distance segment occlusion, and continuous frame drop occlusion.

4. The control method for a universal legged robot according to claim 1, characterized in that, The step of inputting the embedded modal token sequence and the masking mask into the masking temporal attention encoder, and outputting world state latent variables, includes: Obtain the length information of the current sliding window, and determine the causal mask based on the length information; In each attention layer of the occlusion temporal attention encoder, the attention score is calculated using the embedded modal token sequence as the source of query, key and value, and the causal mask and the occlusion mask are superimposed on the attention score; The attention score after the overlay mask is normalized to obtain the attention weight, and the values ​​are weighted and summed according to the attention weight to obtain the attention output of the current layer. After being processed by stacking multiple attention layers, the feature representation corresponding to the current moment is extracted from the attention output of the last layer, and the world state latent variable is obtained by pooling.

5. The control method for a universal legged robot according to claim 1, characterized in that, The cross-morphological fusion module includes a morphological map encoder and a feature fusion unit; The step of inputting the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables includes: The morphological graph encoder is constructed based on a graph neural network or a morphological Transformer encoder; A morphological diagram is constructed using the kinematic topology and dynamic parameters of the legged robot. The nodes of the morphological diagram represent joints or links, and the edges of the morphological diagram represent connection relationships. The node attributes include at least one of the following: link length, mass, moment of inertia, joint type, and joint limit. The morphology graph is input into the graph neural network or the morphology Transformer encoder to obtain a morphology token sequence; The morphological token is fused with the world state latent variable using the feature fusion processor to obtain the morphological conditional latent variable.

6. The control method for a universal legged robot according to claim 5, characterized in that, The process of fusing the morphological token with the world state latent variables through the feature fusion processor to obtain the morphological conditional latent variables includes: Based on feature linear modulation or cross-attention mechanism, the morphological token sequence is injected into the world state latent variable to obtain the morphological conditional latent variable.

7. The control method for a universal legged robot according to claim 1, characterized in that, The control method for the general-purpose legged robot also includes: Based on the quality indication and cross-modal reconstruction residuals, the reliability of each mode at the current moment is determined by a reliability estimator; Dynamically adjusting the masking code based on the reliability includes: When the reliability of any of the modalities is lower than a preset threshold, the corresponding modal token is replaced with a learnable mask token, and the weight of the modality is reduced or masked in the attention calculation.

8. The control method for a universal legged robot according to claim 1, characterized in that, The occlusion temporal attention encoder and the cross-morphological fusion module are obtained through three-stage training, which includes self-supervised occlusion modeling pre-training, multi-task control learning, and cross-morphological expert-student distillation. The self-supervised occlusion modeling pre-training includes: Acquire pre-collected time-series observation data from the multi-mode multi-sensor simulation data pool as historical training data; Random and structured masking are applied to the historical training data to generate a masked modal token sequence; The masked modal token sequence is input into the initialized masked temporal attention encoder. The original observation data of the masked part is used as the supervision signal to optimize the masking reconstruction loss, dynamic consistency loss and cross-modal alignment loss, and to train the pre-trained masked temporal attention encoder. The multi-task control learning includes: The parameters of the pre-trained occlusion temporal attention encoder are connected to the initialized cross-morphological fusion module and the initialized general policy head to form a preliminary policy network. In a multi-task reinforcement learning environment, the preliminary policy network interacts online with the environment to generate experience data. Based on the occlusion temporal attention encoder in the preliminary policy network, the modal token sequence at the current moment is input to obtain temporary world state latent variables; the morphological description information of the current legged robot form is input to the cross-morphological fusion module in the preliminary policy network to generate temporary form tokens; the temporary form tokens are fused with the temporary world state latent variables to obtain temporary form conditional latent variables, which are then input into the general policy head in the preliminary policy network to output actions. Based on the reward signals from environmental feedback, construct a reinforcement learning control loss; The gradient of the reinforcement learning control loss is backpropagated to the occlusion temporal attention encoder, cross-morphology fusion module and general policy head to jointly optimize the parameters. Modal randomization perturbation is continuously applied during training to learn motion control strategies that adapt to different robot shapes, and the occlusion temporal attention encoder, cross-morphology fusion module and general policy head are obtained after preliminary joint training. The cross-morphological expert-student distillation includes: Multiple expert policy networks are obtained, and the action distributions generated by each expert policy network under various states are used as soft labels. The occlusion temporal attention encoder, cross-morphological fusion module, and general policy head, which are preliminarily jointly trained, constitute a unified student policy network. The soft label is used as a supervision signal to optimize the distillation loss so that the action distribution output by the unified student policy network is the same as the action distribution of each expert policy network. During the distillation process, the occlusion reconstruction loss is jointly optimized, and the occlusion temporal attention encoder, the cross-morphological fusion module, and the general policy head are trained.

9. The control method for a universal legged robot according to claim 1, characterized in that, The occlusion-sequential attention encoder also outputs uncertainty; the control method for the general-purpose legged robot further includes: The weights of the desired speed command or safety constraints are adaptively adjusted based on the uncertainty.

10. A control system for a universal legged robot, characterized in that, include: The acquisition unit is used to acquire multi-source sensor time-series observation data of the legged robot in the current control cycle and historical control cycles. The embedding unit is used to uniformly tokenize the time-series observation data of the multi-source sensors to obtain a modal token sequence, and to perform embedding processing on each modal token in the modal token sequence, adding modal embedding, time embedding and sensor configuration embedding to obtain a modal token sequence after adding embedding. The processing unit is used to generate a masking code based on the preset masking strategy during the training phase or the quality indication during the inference phase. The embedded modal token sequence and the occlusion mask are input into the occlusion temporal attention encoder, which outputs the world state latent variable. The control unit is used to acquire the morphological description information of the legged robot, input the morphological description information and the world state latent variables into the cross-morphological fusion module to obtain morphological conditional latent variables, input the morphological conditional latent variables into the general strategy head, and output joint-level control commands to control the movement of the legged robot.

Images