A monocular video-based robot gait control method, device, equipment and storage medium

Abstract

This disclosure relates to the field of robotics, and more particularly to a method, apparatus, device, and storage medium for robot gait control based on monocular video, used to learn and deploy physically feasible, terrain-adaptive robot gait control strategies from monocular video. The robot gait control method includes: acquiring monocular video data and preprocessing it to obtain video segments in units of gait cycles; aligning and decomposing the segments into motion primitives to extract gait primitives; labeling each gait primitive with a corresponding terrain type; storing the labeled gait primitives in a gait library; reconstructing a three-dimensional posture sequence from the video segments; generating dynamic data including joint torques and foot forces through joint kinematic and dynamic optimization; mapping the dynamic data to an executable redirection trajectory for the robot using kinematic and dynamic feasibility as a hard constraint; matching gait primitives with corresponding terrain labels from the gait library based on terrain features extracted from the video segments; splicing the gait primitives according to the temporal requirements of the redirection trajectory to generate a gait reference sequence; validating the gait reference sequence in a digital twin simulation environment; and deploying the gait control strategy corresponding to the valid gait reference sequence to the robot.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and in particular to a method, apparatus, device, and storage medium for robot gait control based on monocular video. Background Technology

[0002] Quadruped robots, with their excellent terrain adaptability and dynamic stability, have broad application prospects in fields such as industrial inspection, security patrol, and emergency rescue. The acquisition and deployment of motor skills are core to quadruped robots achieving autonomous movement in complex environments. Currently, mainstream quadruped robot motion control strategies mainly rely on technologies such as imitation learning, learning complex gaits by having the robot imitate expert trajectories (such as motion data from animals or robots).

[0003] However, existing methods face significant technical bottlenecks in data acquisition and deployment: acquiring high-precision motion data heavily relies on expensive optical motion capture systems, resulting in high data acquisition costs; and ordinary videos cannot recover key dynamic information such as joint torques, leading to poor feasibility of gait generation. Furthermore, the success rate of transferring gait data from simulation to real machines is low, and the gait's generalization ability to unfamiliar terrain is weak. Summary of the Invention

[0004] This disclosure provides a robot gait control method, apparatus, device, and storage medium based on monocular video, which can learn and deploy physically feasible and terrain-adaptive robot gait control strategies from monocular video without motion capture and manual annotation.

[0005] Firstly, a method for robot gait control based on monocular video is provided, comprising: acquiring monocular video data and preprocessing it to obtain video segments in units of gait cycles; performing motion primitive decomposition on the video segments to extract gait primitives, and labeling each gait primitive with a corresponding terrain type, storing the labeled gait primitives in a gait library; reconstructing a three-dimensional posture sequence from the video segments; generating dynamic data containing joint torques and foot forces based on the three-dimensional posture sequence through joint kinematic and dynamic optimization, and mapping the dynamic data to an executable redirection trajectory of the robot with kinematic and dynamic feasibility as a hard constraint; matching gait primitives with corresponding terrain labels from the gait library based on terrain features extracted from the video segments, and splicing them according to the temporal requirements of the redirection trajectory to generate a gait reference sequence; verifying the gait reference sequence in a digital twin simulation environment reconstructed from the video segments, and deploying the gait control strategy corresponding to the verified gait reference sequence to the robot to perform real-time gait control.

[0006] In one embodiment, acquiring monocular video data and preprocessing it includes: selecting valid video segments from the monocular video data based on a semantic classification model and a temporal consistency detection algorithm; and after performing frame rate upscaling, resolution normalization, and noise reduction on the valid video segments, dividing them into video segments based on complete gait cycles.

[0007] In one implementation, the video clip is decomposed into motion primitives to extract gait primitives. Each gait primitive is labeled with a corresponding terrain type, and the labeled gait primitives are stored in a gait library. This includes: based on the video clip, labeling the start and end times of the support phase, the start and end times of the swing phase, and the foot-ground contact switching nodes for each gait cycle to obtain gait temporal tags; based on the gait temporal tags, using a temporal causal discovery algorithm to mine the causal relationships between terrain semantic features, gait phases, joint movements, and foot force; based on the causal relationships, decomposing the continuous motion data in the video clip into flat ground gait primitives, slope gait primitives, stair gait primitives, disturbance-resistant balance primitives, and high-speed motion primitives; and labeling each gait primitive with a corresponding terrain type, movement speed range, joint torque constraints, and foot-ground contact constraints to obtain gait primitives carrying physical attributes; and storing the gait primitives with physical attribute labels in a multi-dimensional classification according to terrain type, movement mode, and gait type in a gait library.

[0008] In one embodiment, reconstructing a three-dimensional pose sequence from the video clip includes: extracting the contour mask of the moving subject in the video clip using a visual segmentation model; performing dense visual feature alignment using a visual feature alignment model within the region defined by the contour mask to extract key point features of the quadrupedal torso and limbs; performing temporal optimization and filling in occluded joints using a temporal optimization network based on the key point features to obtain a smooth two-dimensional joint temporal sequence; performing initial fitting of the three-dimensional skeleton using a parametric skeleton model based on the two-dimensional joint temporal sequence, with joint reprojection error as the optimization objective, to obtain a coarsely reconstructed three-dimensional pose sequence; performing fine alignment of gait cycles using a dynamic time warping algorithm based on the coarsely reconstructed three-dimensional pose sequence to obtain an aligned three-dimensional pose sequence; and performing global optimization using a temporal optimization network with built-in gait cycle priors and left-right leg symmetry constraints based on the aligned three-dimensional pose sequence, and performing interpolation and filtering on the optimized three-dimensional pose sequence to obtain a processed three-dimensional pose sequence.

[0009] In one implementation, based on the three-dimensional posture sequence, dynamic data including joint torques and foot forces are generated through joint kinematic and dynamic optimization. This includes: based on the three-dimensional posture sequence, outputting joint angle sequences, foot trajectory sequences, and fuselage pose sequences through the kinematic branch of the joint kinematic and dynamic optimization network; based on the three-dimensional posture sequence, calculating the foot force and torque sequences and joint torque sequences through the dynamic branch of the joint kinematic and dynamic optimization network, using rigid body dynamics equations as hard constraints; embedding a differentiable physics simulator within the dynamic branch; and introducing a contact loss function to constrain the non-penetration and non-slip conditions between the foot and the ground based on the joint angle sequence, foot trajectory sequence, fuselage pose sequence, foot force and torque sequence, and joint torque sequence, thereby obtaining dynamic data that satisfies physical consistency.

[0010] In one implementation, the dynamic data is mapped to an executable redirection trajectory for the robot, with kinematic and dynamic feasibility as hard constraints. This includes: based on the dynamic data, and considering the differences in joint configuration between the moving subject and the robot in the video clip, mapping the dynamic data to the robot's joint space with hard constraints such as consistency of foot-to-ground contact timing, upper limit of joint torque, and body balance stability. The mapping process includes correcting for issues such as excessive joint range of motion, foot suspension, foot penetration, and body tipping, while preserving the natural dynamic characteristics of the original gait, thus obtaining an executable redirection trajectory for the robot.

[0011] In one implementation, based on terrain features extracted from the video clip, gait primitives with corresponding terrain annotations are matched from the gait library and spliced ​​according to the temporal requirements of the redirection trajectory to generate a gait reference sequence. This includes: matching gait primitives with corresponding terrain annotations from the gait library based on terrain features extracted from the video clip, splicing and physically verifying them according to the temporal requirements of the redirection trajectory to obtain an initial gait reference sequence; performing end-to-end optimization of the initial gait reference sequence using a dual-layer imitation learning architecture to obtain multiple candidate gait sequences; the dual-layer imitation learning architecture includes a foot-ground interaction perception motion tracking module and a vision-and-terrain dual-conditional gait diffusion model; and comprehensively scoring the multiple candidate gait sequences using a multi-dimensional scoring network, selecting the optimal gait sequence based on the comprehensive scoring results, and using the optimal gait sequence for subsequent simulation verification and real-device deployment.

[0012] In one implementation, the foot-ground interaction sensing motion tracking module is constructed according to the following steps: Based on the initial gait reference sequence, a reinforcement learning algorithm is used, with the joint relative poses, foot contact states, fuselage inertial measurement data, and tracking errors of the initial reference gait sequence as the observation space, to learn an end-to-end mapping from the initial gait reference sequence to joint torque commands; a reward function is formed by superimposing foot-ground contact consistency rewards, fuselage stability rewards, and energy efficiency rewards, supplemented by joint limit penalties, self-collision penalties, and joint torque over-limit penalties, using the tracking error reward as the base reward, and supplemented by joint limit penalties, self-collision penalties, and joint torque over-limit penalties; a low-level motion tracking strategy is trained based on the reward function; the low-level motion tracking strategy is deployed in the foot-ground interaction sensing motion tracking module, so that the foot-ground interaction sensing motion tracking module has the end-to-end mapping capability from the reference gait sequence to joint torque commands.

[0013] In one implementation, the visual and terrain-based dual-conditional gait diffusion model is constructed according to the following steps: obtaining the conditional input and observation input of the gait diffusion model; wherein, the embedding vector output by the terrain semantic encoder after encoding real-time visual semantic features and terrain point clouds is used as the conditional input, and the aircraft state is used as the observation input; based on the conditional input and observation input, a future gait sequence covering at least two complete gait cycles is generated through the gait diffusion model; based on the future gait sequence, a classifier-guided optimization is performed through terrain and gait causal graphs to ensure that the generated future gait sequence conforms to the physical causal constraints between terrain and gait, thereby obtaining an upper-level gait planning strategy with terrain generalization capability; the upper-level gait planning strategy is deployed on the visual and terrain-based dual-conditional gait diffusion model to enable the visual and terrain-based dual-conditional gait diffusion model to perform end-to-end optimization of the initial gait reference sequence and generate multiple sets of candidate gait sequences.

[0014] In one implementation, the gait reference sequence is validated in a digital twin simulation environment reconstructed from the video clip. This includes: reconstructing the geometric structure and visual texture of the real terrain scene in the video clip using a 3D reconstruction algorithm, and replicating the reconstructed terrain scene onto a digital twin simulation platform to obtain a digital twin simulation environment; identifying core causal variables affecting gait through causal inference based on the digital twin simulation environment, and performing adversarial randomization on the core causal variables in the gait reference sequence based on the real-world physical parameter distribution recovered from the video clip, thus completing the simulation-to-real-world domain adaptation training of the gait reference sequence; and conducting parallel simulation tests on the gait reference sequence after domain adaptation training in the digital twin simulation environment, generating extreme adversarial scenarios including lateral impact, frictional abrupt changes, and sensor noise for robustness verification, and automatically feeding back simulation failure cases to the training set for incremental fine-tuning to obtain a validated gait reference sequence.

[0015] In one implementation, a validated gait reference sequence is deployed to the robot for real-time gait control. This includes: performing lightweight optimization on the gait strategy corresponding to the validated gait reference sequence. The lightweight optimization includes structured pruning, model quantization, and operator fusion for the robot's onboard hardware, and completing hardware-specific compilation optimization to obtain a lightweight gait strategy. Based on the lightweight gait strategy, a terrain semantic encoder and a gait diffusion model are run at a first frequency, with real-time video streams from the onboard camera as input, and a gait planning sequence adapted to the current terrain as output. A motion tracking model is run at a second frequency, with inertial measurement data, joint encoder data, and foot force sensor data as input, and joint torque commands as output, to complete gait tracking and real-time balance control. The second frequency is greater than the first frequency. During real-time control, a safety fallback module monitors the robot's pitch and roll angles, joint torques, and foot slippage. When any parameter exceeds a preset safety threshold, the robot switches to a safe gait or performs an emergency stop.

[0016] In one implementation, the method further includes: using the gait control strategy deployed to the robot as an edge strategy; collecting monocular video data and execution feedback data in a new scenario during the robot's task execution; uploading the video data and execution feedback data in the new scenario to the cloud; re-executing the steps of acquiring monocular video data and performing preprocessing, motion primitive decomposition, 3D pose reconstruction, dynamic data generation, and redirected trajectory mapping in the cloud to generate incremental gait primitives and store them in a gait library; incrementally fine-tuning the edge strategy based on the incremental gait primitives; constraining the update magnitude of the weights corresponding to the original gait skills through an elastic weight consolidation algorithm; and distributing the updated gait strategy to the robot.

[0017] Secondly, a robot gait control device based on monocular video is provided, comprising: an acquisition module for acquiring monocular video data and preprocessing it to obtain video segments in units of gait cycles; and a gait library building module for performing motion primitive decomposition on the video segments, decomposing them into gait primitives, labeling each gait primitive with a corresponding terrain type, and storing the labeled gait primitives into a gait library. A 3D pose reconstruction module is used to reconstruct a 3D pose sequence from the video clip; a redirection trajectory generation module is used to generate dynamic data containing joint torques and foot forces based on the 3D pose sequence through joint kinematic and dynamic optimization, and map the dynamic data into an executable redirection trajectory for the robot with kinematic and dynamic feasibility as a hard constraint; a gait reference sequence generation module is used to match gait primitives with corresponding terrain annotations from the gait library based on terrain features extracted from the video clip, and splice them according to the temporal requirements of the redirection trajectory to generate a gait reference sequence; a gait control strategy deployment module is used to verify the gait reference sequence in a digital twin simulation environment reconstructed from the video clip, and deploy the gait control strategy corresponding to the verified gait reference sequence to the robot to perform real-time gait control.

[0018] Thirdly, an apparatus is provided, comprising a processor and a memory, wherein computer instructions are stored in the memory, and the processor is configured to execute the computer instructions to perform the steps of the robot gait control method described in any of the above embodiments.

[0019] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the robot gait control method described in any of the above embodiments.

[0020] This disclosed embodiment eliminates the need for expensive optical motion capture systems, enabling gait data acquisition, primitive decomposition, and dynamic reconstruction solely through monocular video, effectively reducing data acquisition costs. By performing motion primitive decomposition on the video and constructing a gait library containing terrain annotations, a structured foundation is provided for subsequent terrain matching and gait stitching. Simultaneously, joint torques, foot forces, and other dynamic information are recovered through joint kinematic and dynamic optimization, and closed-loop verification is performed in a digital twin environment to ensure the physical feasibility of the gait. This solves the problems of instability and poor adaptation of simulated gait on real machines, significantly improving the zero-sample transfer success rate from simulation to real robots. Furthermore, by extracting terrain features from the video and matching gait primitives with corresponding terrain annotations from the gait library for stitching, the generated gait can actively adapt to the current terrain, effectively solving the problem of weak generalization ability of existing technologies for unfamiliar terrains and improving the robot's terrain adaptability in unstructured environments.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0022] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a robot gait control method based on monocular video provided in an exemplary embodiment of this disclosure is shown.

[0025] Figure 2 This illustration shows an overall logical diagram of a robot gait control method provided in an exemplary embodiment of the present disclosure; Figure 3 A schematic diagram of a robot gait control device 300 provided in an embodiment of this disclosure is shown; Figure 4 A schematic diagram of the structure of a device 400 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0028] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0030] Research has revealed the following bottlenecks in existing quadruped robot gait control methods: First, they rely on optical motion capture systems to acquire motion data, which is costly and limited by application scenarios. Second, while ordinary videos are easy to acquire, they lack the ability to map visual signals to physical signals, failing to recover dynamic information such as joint torques and foot forces, resulting in poor physical feasibility of the generated gait. Third, traditional methods have weak generalization ability to unfamiliar terrains, low success rate of simulation-to-real-machine transfer, and lack a fully automated solution. Therefore, this disclosure provides a robot gait control method based on monocular video, aiming to achieve fully automated gait control from data acquisition, dynamics reconstruction, terrain adaptation to real-machine deployment using only monocular video input.

[0031] Figure 1 This is a flowchart illustrating a robot gait control method based on monocular video, provided as an exemplary embodiment of this disclosure. Figure 1 As shown, the robot gait control method includes: S101: Acquire monocular video data and preprocess it to obtain video segments in units of gait cycles.

[0032] For example, firstly, monocular video data (i.e., video captured by a standard single camera) is acquired and preprocessed. The purpose of preprocessing is to convert the raw video into a standard format suitable for subsequent analysis, specifically including operations such as frame rate unification, resolution normalization, and noise reduction. Based on this, according to the movement characteristics of the moving subjects in the video, such as quadrupeds, the video is segmented into standardized video segments based on gait cycles.

[0033] A gait cycle refers to a complete movement cycle from the moment one foot touches the ground to the next, typically consisting of a stance phase and a swing phase. The stance phase is when the foot contacts the ground and bears the body weight, providing propulsion and stability; the swing phase is when the foot lifts off the ground and swings forward in the air, preparing for the next landing. By segmenting the gait cycle, uniform and standardized input data can be provided for subsequent posture reconstruction and gait analysis.

[0034] In one embodiment, acquiring monocular video data and preprocessing it includes: selecting valid video segments from the monocular video data based on a semantic classification model and a temporal consistency detection algorithm; and after performing frame rate upscaling, resolution normalization, and noise reduction on the valid video segments, dividing them into video segments based on complete gait cycles.

[0035] In practice, the original monocular video often contains a large amount of invalid content, such as static images, severe occlusion, motion blur, and incomplete gait cycles. To improve the efficiency and quality of subsequent processing, it is necessary to automatically filter out valid video segments. Specifically, firstly, a semantic classification model, such as a Contrastive Language-Image Pre-training (CLIP) model, is used to perform semantic understanding on video frames to identify whether the video contains a target moving subject. For example, in this embodiment, the robot is a quadruped robot, and the target moving subject can be a quadruped animal (such as a dog, horse, or deer). It also identifies whether the movement matches the expected motion scene (such as running or climbing). This model aligns images with text descriptions through contrastive learning, enabling it to determine whether a target moving subject and reasonable actions exist in the video frame. Based on this, a temporal consistency detection algorithm is used to analyze consecutive frames of the video segment. This algorithm detects the motion coherence between adjacent frames, the degree of occlusion (e.g., occlusion rate greater than 30% is discarded), the degree of blur, and the completeness of the gait cycle to ensure that the selected video segments have a smooth, continuous, and uninterrupted motion trajectory in the temporal dimension. The combination of these two methods can automatically filter out invalid segments such as static images, excessive occlusion, severe motion blur, and incomplete gait cycles, without the need for manual annotation.

[0036] The selected valid video segments still need to be standardized to meet the input requirements of subsequent pose reconstruction and dynamics analysis. Specifically, for frame rate upscaling: the original video frame rate is usually low (e.g., 30fps or 60fps), while quadruped robot control requires higher temporal resolution (e.g., 100fps or higher) to capture rapidly changing gait phases. Methods such as spline interpolation, optical flow interpolation, or temporal super-resolution networks are used to insert intermediate frames between existing frames, increasing the video frame rate to the target frequency (e.g., 100fps), allowing for more precise capture of the transition nodes between the support and swing phases in the gait cycle. For resolution normalization: videos from different sources have varying resolutions (e.g., 720p, 1080p, 4K), requiring uniform scaling to a preset resolution (e.g., 640×640 or 512×512). Normalization reduces the computational complexity of subsequent models and ensures the consistency of input features. For denoising: videos may contain interference information such as shooting noise, compression artifacts, and illumination fluctuations. Video frames are smoothed by using Gaussian filtering, median filtering, or more advanced deep learning denoising algorithms (such as autoencoder-based denoising networks) to reduce high-frequency noise while preserving key edge and structural information of gait motion.

[0037] After the above processing, the standardized video is then segmented into units of complete gait cycles (i.e., truncated according to key event points such as forefoot landing or hindfoot takeoff), outputting a series of standardized, high-quality video clips, providing unified and reliable input data for subsequent posture reconstruction and gait primitive decomposition.

[0038] S102: Perform motion primitive decomposition on the video segment, decompose gait primitives, label each gait primitive with the corresponding terrain type, and store the labeled gait primitives in the gait library.

[0039] Here, motion primitives are decomposed into motion primitives after the video segments are segmented. Motion primitives refer to the basic action units that constitute complex movements, such as jogging on flat ground, climbing on a slope, and going up and down stairs—local motion patterns with clear functionality and reusability. Using a causal discovery algorithm, continuous motion data is decomposed into independently combinable gait primitives. Then, each gait primitive is labeled with its corresponding terrain type (e.g., flat ground, slope, stairs, gravel ground), and the labeled gait primitives are stored in a gait database. The gait database is a structured database used to store and manage various gait primitives and their attributes, supporting subsequent rapid retrieval and retrieval based on terrain features.

[0040] In one implementation, the video clip is decomposed into motion primitives to extract gait primitives. Each gait primitive is labeled with a corresponding terrain type, and the labeled gait primitives are stored in a gait library. This may include: based on the video clip, labeling the start and end times of the support phase, the start and end times of the swing phase, and the foot-ground contact switching nodes for each gait cycle to obtain gait temporal tags; based on the gait temporal tags, using a temporal causal discovery algorithm to mine the causal relationships between terrain semantic features, gait phases, joint movements, and foot forces; based on the causal relationships, decomposing the continuous motion data in the video clip into flat ground gait primitives, slope gait primitives, stair gait primitives, disturbance-resistant balance primitives, and high-speed motion primitives; and labeling each gait primitive with a corresponding terrain type, movement speed range, joint torque constraints, and foot-ground contact constraints to obtain gait primitives carrying physical attributes; and storing the gait primitives with physical attribute labels in a multi-dimensional classification according to terrain type, movement mode, and gait type in a gait library.

[0041] In the above implementation, the first step is to perform fine-grained temporal annotation on each gait cycle in the standardized video clip. A gait cycle consists of alternating support and swing phases. Specific annotations include: the start and end times of the support phase (i.e., the moment the foot begins to contact the ground and the moment it is about to leave the ground), the start and end times of the swing phase (i.e., the moment the foot leaves the ground and begins to swing forward and the moment it is about to touch the ground again), and the foot-ground contact switching point (i.e., the critical point of foot contact / leave). By annotating these key time points, structured gait temporal tags are formed. This step is equivalent to marking the continuous motion video with a timeline, clearly defining the start and end boundaries of each phase in each gait cycle.

[0042] Based on the obtained gait temporal labels, temporal causal discovery algorithms (such as NOTEARS-Dyn, based on non-combinatorial optimization of augmented Lagrange traces) are used to conduct in-depth data analysis. Causal relationships between terrain semantic features (such as flat ground, slopes, stairs, etc.), gait phases (stance / swing phase), joint movement sequences, and foot forces are automatically mined. For example, a causal association can be found between "sloping terrain" leading to "prolonged forelimb stance phase" and "increased hindlimb joint torque." Based on the mined causal relationships, continuous motion data is decomposed along causal boundaries, segmenting it into independent gait primitives with clear functional meanings. For example, these gait primitives include: flat ground gait primitives (such as trotting, pacing), slope gait primitives (crawling uphill / downhill), stair gait primitives (going up / down stairs), disturbance resistance balance primitives (responding to lateral impacts or uneven ground), and high-speed motion primitives (such as running, jumping). Each gait primitive is essentially a basic motion unit that can be independently reused and combined.

[0043] For each decomposed gait primitive, its corresponding physical attributes are further labeled, including: terrain type (flat ground, slope, stairs, gravel, etc.), movement speed range (e.g., 0.5-1.0 m / s), joint moment constraints (upper and lower limits of allowable moments for each joint), and foot-ground contact constraints (e.g., prohibiting foot sliding or penetration of the ground). These physical attributes ensure that the gait primitive not only contains movement trajectory information but also carries executable physical feasibility constraints. The final result is a complete labeling of gait primitives and physical attributes, providing quantifiable constraints for subsequent terrain matching and gait stitching.

[0044] Gait primitives labeled with physical attributes are categorized and stored according to multiple dimensions, such as by terrain type (flatland, slope, stairs, etc.), by movement mode (crawling, jogging, running, etc.), and by gait type (diagonal trot, same-side walking pace, gallop, etc.). After categorization, they are stored in a gait database, and a searchable index system is established. This allows subsequent steps to quickly match and retrieve the most suitable gait primitive from the database based on the currently identified terrain features, achieving terrain-driven gait generation.

[0045] In one optional embodiment of this disclosure, the constructed gait library can employ a distributed storage architecture to achieve persistent data storage and highly available access. Simultaneously, to address the incremental updates and iterative optimization needs of gait primitives, the gait library incorporates a version management mechanism, supporting version marking, backtracking queries, and one-click rollback operations for gait primitives. Specifically, each time a gait primitive is added, modified, or deleted, a new version record is automatically generated, and a complete snapshot of historical versions is retained. When incremental training experiences performance degradation or catastrophic forgetting, the version management mechanism allows for rapid rollback to a stable version, ensuring the reliability and maintainability of the gait library. This mechanism provides data-level storage and iterative support for subsequent lifelong learning loops.

[0046] S103: Reconstruct a three-dimensional pose sequence from the video clip.

[0047] In practice, a three-dimensional pose sequence can be reconstructed from the preprocessed standardized video clips. This three-dimensional pose sequence includes kinematic information such as joint angles, limb positions, and fuselage orientation of the moving subject in space. Since monocular videos only contain two-dimensional image information, parametric skeleton models, such as the Skinned Multi-Animal Linear Model (SMAL), and temporal optimization networks (such as the Temporal Transformer) are needed to infer the pose in three-dimensional space. Through this step, the two-dimensional video can be converted into three-dimensional pose data containing complete motion trajectories, providing a foundation for subsequent dynamic analysis.

[0048] In one implementation, a three-dimensional pose sequence can be reconstructed from the video clip as follows: Based on the video clip, the contour mask of the moving subject in the video is extracted using a visual segmentation model; within the region defined by the contour mask, dense visual feature alignment is performed using a visual feature alignment model to extract key point features of the quadrupedal torso and limbs; based on the key point features, temporal optimization is performed using a temporal optimization network to complete occluded joints, resulting in a smooth two-dimensional joint temporal sequence; based on the two-dimensional joint temporal sequence, with joint reprojection error as the optimization objective, an initial three-dimensional skeleton fitting is performed using a parametric skeleton model to obtain a coarsely reconstructed three-dimensional pose sequence; based on the coarsely reconstructed three-dimensional pose sequence, a gait cycle fine alignment is performed using a dynamic time warping algorithm to obtain an aligned three-dimensional pose sequence; based on the aligned three-dimensional pose sequence, global optimization is performed using a temporal optimization network with built-in gait cycle priors and left-right leg symmetry constraints, and the optimized three-dimensional pose sequence is interpolated and filtered to obtain a processed three-dimensional pose sequence.

[0049] Specifically, firstly, based on video clips, each frame of the image is processed using a visual segmentation model. This model can automatically identify moving subjects in the video, such as the outline of a quadruped (correspondingly, the robot described in this embodiment can refer to a quadruped robot), and segment it from the background to generate a corresponding contour mask (i.e., a binary image that marks the pixel region where the moving subject is located). The purpose of this step is to focus subsequent processing on the moving subject itself and eliminate background interference. Within the region defined by the contour mask, dense visual feature alignment is performed using a visual feature alignment model, such as a DIstillation with NO labels (DINO) model. The DINO model is a self-supervised visual Transformer that can extract deep semantic features of each pixel in the image. By aligning features between adjacent frames, the positions of key points (such as shoulder joints, hip joints, knee joints, ankle joints, etc.) on the torso and limbs of the moving subject, such as a quadruped, can be identified in the image. Due to problems such as occlusion and motion blur in monocular videos, the two-dimensional joint points extracted directly from a single frame image may be missing or jittery. To address this, a temporal optimization network (such as the Temporal Transformer network) is used to temporally optimize the extracted keypoint features. This network utilizes the correlation information between consecutive frames to automatically complete occluded joints and smooth their motion trajectories, ultimately outputting a stable and smooth 2D joint temporal sequence. Based on this 2D joint temporal sequence, using a parametric skeleton model (such as the SMAL model, i.e., Skinned Multi-Animal Linear Model) as a foundation, the 2D joint reprojection error (i.e., the positional deviation between the 3D skeleton projected back to the 2D image and the detected 2D joints) is used as the optimization objective. By minimizing this error, the pose parameters of the 3D skeleton are fitted, outputting an initial, coarsely reconstructed 3D pose sequence, including basic information such as joint angles and fuselage pose. The coarsely reconstructed 3D pose sequence may contain temporal misalignments, such as the start and end points of the gait cycle not being perfectly aligned with the actual video. Therefore, a Dynamic Time Warping (DTW) algorithm is used to temporally align the pose sequence. The DTW algorithm can flexibly stretch or compress the time axis, so that the reconstructed gait period can be accurately matched with the gait phase (the switching node between the support phase and the swing phase) in the real video, thus solving the timing misalignment problem caused by low frame rate videos or uneven gait speed.

[0050] Taking quadrupeds as the primary locomotion subject, a specialized temporal optimization network (such as the Quad-PoseFormer network) is constructed to address the gait characteristics of quadrupeds (e.g., periodicity and left-right leg symmetry). This network incorporates quadrupedal gait periodic priors (i.e., the phase relationship between the forelimbs and hindlimbs in the gait cycle) and left-right leg symmetry constraints (i.e., the movement of opposite limbs should remain symmetrical). Global optimization of the aligned 3D pose sequence using this network effectively completes the leg joint trajectories in occluded scenes, significantly improving pose reconstruction accuracy (e.g., reducing joint reprojection errors by more than 60%). Finally, the optimized 3D pose sequence undergoes interpolation and filtering. Spline interpolation adds intermediate frames in the temporal dimension, increasing the temporal resolution of the pose sequence to the target frequency (e.g., above 100fps); low-pass filtering then smooths and removes high-frequency jitter, making the pose sequence smoother and more natural. The final output is a high-precision, high-frame-rate 3D pose sequence, meeting the input requirements for subsequent dynamic reconstruction and control deployment.

[0051] S104: Based on the three-dimensional posture sequence, dynamic data containing joint torques and foot forces are generated through joint kinematic and dynamic optimization. The dynamic data is then mapped to an executable retargeting trajectory for the robot, with kinematic and dynamic feasibility as a hard constraint.

[0052] For example, kinematics primarily describes the geometric and temporal relationships of motion, such as joint angles, positions, and velocities. Dynamics, on the other hand, further involves physical properties such as force, torque, mass, and inertia.

[0053] This step involves embedding a differentiable physics simulator, using rigid body dynamics equations as hard constraints, to regress dynamic data such as joint torques and foot forces from the 3D posture. Because the simulator is differentiable, the network can perform end-to-end optimization of the output through gradient backpropagation, ensuring the dynamic data matches the input kinematic data. Subsequently, using kinematic feasibility (e.g., joint angles not exceeding limits) and dynamic feasibility (e.g., torque feasibility, no foot-to-ground contact penetration) as constraints, the dynamic data from the biological video is mapped into a redirected trajectory executable by the target robot. This process resolves the adaptation problem arising from the difference in configuration between the moving subject in the video and the robot performing gait control.

[0054] In one implementation, based on the three-dimensional posture sequence, dynamic data including joint torques and foot forces are generated through joint kinematic and dynamic optimization. This includes: based on the three-dimensional posture sequence, outputting joint angle sequences, foot trajectory sequences, and fuselage pose sequences through the kinematic branch of the joint kinematic and dynamic optimization network; based on the three-dimensional posture sequence, calculating the foot force and torque sequences and joint torque sequences through the dynamic branch of the joint kinematic and dynamic optimization network, using rigid body dynamics equations as hard constraints; embedding a differentiable physics simulator within the dynamic branch; and introducing a contact loss function to constrain the non-penetration and non-slip conditions between the foot and the ground based on the joint angle sequence, foot trajectory sequence, fuselage pose sequence, foot force and torque sequence, and joint torque sequence, thereby obtaining dynamic data that satisfies physical consistency.

[0055] Here, the dual-branch joint optimization network comprises two parallel channels: a kinematics branch and a dynamics branch. The kinematics branch is responsible for extracting geometric motion information from the 3D pose sequence (i.e., indicating how the object moves in space), while the dynamics branch is responsible for calculating the forces and torques required to drive these movements (i.e., indicating why the movement is possible). The two branches work together to optimize, ensuring that the output motion data conforms to both the observed trajectory and the laws of physics.

[0056] The kinematics branch takes the 3D pose sequence reconstructed in step S103 as input and uses a neural network to regress standardized kinematic data, including joint angle sequences, foot trajectory sequences, and body pose sequences. Specifically, the joint angle sequence represents the angle values ​​of each robot joint (hip, knee, ankle, etc.) over time; the foot trajectory sequence represents the trajectory of the four feet in 3D space over time; and the body pose sequence represents the sequence of the robot's torso position (x, y, z) and orientation (roll, pitch, yaw) over time. These data describe the robot's purely geometric motion state without considering forces.

[0057] The dynamics branch also takes a three-dimensional posture sequence as input, but it embeds a differentiable physics simulator. This differentiable physics simulator is a differentiable physics engine that uses rigid body dynamics equations (such as the Newton-Euler equations) as hard constraints (i.e., physical rules that must be strictly followed) to back-calculate the physical quantities required to drive the motion from the motion data. These physical quantities include sequences of foot forces and torques, and sequences of joint torques. The sequence of foot forces and torques represents the three-dimensional forces and torques generated when each foot contacts the ground; the sequence of joint torques represents the torque values ​​that each joint actuator needs to output. Because the simulator is differentiable, the network can continuously optimize the calculation results through gradient backpropagation, ensuring that the output dynamics data matches the input kinematic data.

[0058] Furthermore, to avoid potential physical inconsistencies in the forces and moments calculated based on the dynamics branch, such as the foot penetrating the ground (ground penetration) or sliding after landing (slippage), this embodiment introduces a contact loss function as an additional optimization objective. This loss function guides the generation of complete dynamic data that satisfies physical consistency by applying constraints such as no-penetration constraints and no-slip constraints. The no-penetration constraint means that the foot position should always be above the ground plane or terrain surface and cannot sink below the ground; the no-slip constraint means that the foot remains relatively stationary with respect to the ground during the support phase and cannot exhibit non-physical sliding displacement. By optimizing the above loss function, the network iteratively adjusts the kinematic and dynamic outputs, ultimately generating complete dynamic data that satisfies physical consistency. This data simultaneously includes kinematic information (joint angles, foot trajectory, fuselage pose) and dynamic information (foot force / moment, joint torque), and these data conform to the laws of rigid body motion and foot-ground contact constraints. Experimental verification shows that the physical accuracy can reach over 90%.

[0059] In one implementation, the dynamic data is mapped to an executable redirection trajectory for the robot, with kinematic and dynamic feasibility as hard constraints. This includes: based on the dynamic data, and considering the differences in joint configuration between the moving subject and the robot in the video clip, mapping the dynamic data to the robot's joint space with hard constraints such as consistency of foot-to-ground contact timing, upper limit of joint torque, and body balance stability. The mapping process includes correcting for issues such as excessive joint range of motion, foot suspension, foot penetration, and body tipping, while preserving the natural dynamic characteristics of the original gait, thus obtaining an executable redirection trajectory for the robot.

[0060] In practice, there are often configurational differences between the moving subject in the video and the target deployment object. Taking a quadrupedal animal (such as a dog, horse, or deer) as the moving subject and a quadrupedal robot as an example, there are significant configurational differences between the two. These differences manifest in several dimensions: the number and distribution of joint degrees of freedom (biological joints are more flexible, while robot joints are more restricted), the range of motion of joints (biological joints typically have a larger range of motion than robot joints), body proportions (the ratio of leg length to torso length differs between biological and robot joints), and the actuation method (biological joints are driven by muscles, while robot joints are driven by motors). Therefore, biologically reconstructed motion data cannot be directly used as control commands for the robot; it must undergo redirection mapping.

[0061] To ensure the mapped trajectory is executable and stable on a real robot, hard constraints can be set, such as consistency of foot-to-ground contact timing, upper limit constraints on joint torque, and constraints on robot balance stability. Consistency of foot-to-ground contact timing means that the contact / removal timing of the robot's feet from the ground must strictly match the gait phase in the monocular video. For example, if the right forefoot enters the support phase (ground contact) at 0.5 seconds in the original video, the robot's right forefoot must also touch the ground at the same time. This constraint ensures the fidelity of the gait rhythm. Upper limit constraints on joint torque mean that each joint motor of the robot has a maximum output torque limit. During mapping, any joint torque command exceeding this limit must be clipped or redistributed to prevent motor overload damage. Consistency on robot balance stability means that during gait execution, the robot's pitch angle, roll angle, and center of mass projection point must remain within a stable threshold (e.g., the overturning angle does not exceed 25 degrees) to prevent rollover or forward tilting during gait.

[0062] Under the premise of satisfying the above hard constraints, when mapping dynamic data from the biological configuration to the robot's joint space, some corrective measures can be automatically executed, such as correcting excessive joint range of motion, correcting foot suspension, correcting foot penetration, and correcting body tipping. Correcting excessive joint range of motion refers to the fact that the range of motion of biological joints is usually greater than that of robots (for example, a horse's shoulder joint can swing backward at a greater angle). During mapping, the excess joint angle is compressed proportionally to within the robot's joint limits, while optimizing the coordinated movement of adjacent joints to preserve the original motion characteristics as much as possible. Correcting foot suspension refers to the fact that due to differences in body proportions, the feet may be raised too high or too low in the swing phase of the original trajectory. During mapping, the height of the foot trajectory is adjusted to ensure that the foot's clearance from the ground is reasonable during the swing phase (neither too high to waste energy nor too low to cause tripping). Correcting foot penetration refers to the possibility that the feet may sink into the ground when the ground height does not match the robot's leg length. During mapping, the hip / knee joint angles are adjusted or inverse kinematics recalculation is used to ensure that the feet just touch the ground during the support phase without penetration. Correcting body tilt refers to the situation where direct migration of the original gait may cause the robot's center of mass to deviate from the supporting polygon; during mapping, body posture compensation or the addition of angular momentum control is used to ensure the overall balance and stability of the robot.

[0063] The aforementioned modifications do not involve simply and crudely cutting or suppressing gait data. Instead, they preserve the natural characteristics of the original biological gait as much as possible while ensuring physical feasibility. These characteristics include, for example, the smooth curvature of the leg swing, the trajectory shape after the foot leaves the ground, and the changing trends of joint angles during the gait cycle. The final output is an executable retargeting trajectory that satisfies both robot hardware constraints and retains the original stylistic features. This trajectory can be directly used for subsequent gait reference sequence generation and simulation verification.

[0064] S105: Based on the terrain features extracted from the video clip, match gait primitives with corresponding terrain annotations from the gait library, and splice them according to the temporal requirements of the redirection trajectory to generate a gait reference sequence.

[0065] In practice, terrain features (such as slope, roughness, and terrain type) are extracted from the original video clips and transformed into semantic representations using a visual encoder. Then, based on the extracted terrain features, gait primitives with corresponding terrain annotations are matched from the gait library constructed in step S102. Following the temporal requirements specified in the redirection trajectory generated in step S104 (such as the order of gait cycles and the duration of each primitive), the matched gait primitives are spliced ​​and optimized to generate a complete gait reference sequence. This sequence serves as the target action reference for subsequent simulation verification and real-device deployment.

[0066] In one implementation, in step S105 above, gait primitives extracted from a gait library and matched with the terrain of the video clip can be spliced ​​together according to the following steps to generate a gait reference sequence: based on the terrain features extracted from the video clip, gait primitives with corresponding terrain annotations are matched from the gait library and spliced ​​and physically verified according to the temporal requirements of the redirection trajectory to obtain an initial gait reference sequence; the initial gait reference sequence is optimized end-to-end through a dual-layer imitation learning architecture to obtain multiple sets of candidate gait sequences; the dual-layer imitation learning architecture includes a foot-ground interaction perception motion tracking module and a vision and terrain dual-condition gait diffusion model; the multiple sets of candidate gait sequences are comprehensively scored through a multi-dimensional scoring network, the optimal gait sequence is selected based on the comprehensive scoring results, and the optimal gait sequence is used for subsequent simulation verification and real device deployment.

[0067] In this implementation, terrain features (such as slope, roughness, and terrain type) are first extracted from the video clips and matched with pre-stored terrain annotations in the gait library to select gait primitives suitable for the current terrain. Then, according to the temporal requirements of the redirection trajectory (i.e., the order and duration of each gait primitive on the time axis), these primitives are spliced ​​together, and transition optimization (such as interpolation smoothing) is performed at the splicing points to avoid abrupt joint changes. Finally, the spliced ​​initial sequence is subjected to rigid body dynamics verification to filter out infeasible sequences with joint overruns, torque overruns, and overturning angles exceeding limits, and outputs an initial gait reference sequence that conforms to the robot's physical constraints.

[0068] Subsequently, the initial gait reference sequence is input into a dual-layer imitation learning architecture for end-to-end optimization. This architecture comprises two core modules: a bottom-layer foot-ground interaction-aware motion tracking module responsible for accurately tracking the reference gait sequence and outputting joint torque commands; and an upper-layer vision-and-terrain dual-conditional gait diffusion model responsible for long-term gait planning and outputting a terrain-adapted gait sequence. The two modules work together to iteratively optimize the initial sequence, generating multiple sets of different candidate gait sequences (e.g., generating multiple variants by changing the sampling noise or the number of diffusion steps).

[0069] Finally, a multi-dimensional gait scoring network is used to comprehensively score each generated candidate gait sequence. The scoring dimensions typically include: tracking accuracy (similarity to the reference sequence), physical feasibility (whether torque exceeds limits, whether contact is reasonable), stability (amplitude of fuselage pitch / roll undulations), energy efficiency (motor power consumption), and terrain adaptability (expected success rate on the target terrain). The scoring network is trained using supervised learning on real motion capture data, simulation verification data, and real-aircraft deployment data. Here, real motion capture data serves as a ground truth reference for gait accuracy; simulation verification data is used to evaluate the feasibility of the gait in the simulation environment; and real-aircraft deployment data is used to align the scoring results with the actual deployment effect. Causal constraints are embedded during training to ensure a strong correlation between the scoring results and the gait's real-aircraft deployment effect. Verification shows that the scoring prediction error of this scoring network is less than 3%. Finally, the optimal gait sequence with the highest score is automatically selected based on the comprehensive scoring results for subsequent simulation verification and real-aircraft deployment.

[0070] In one implementation, the foot-ground interaction sensing motion tracking module can be constructed according to the following steps: Based on the initial gait reference sequence, a reinforcement learning algorithm is employed, using the joint relative poses, foot contact states, fuselage inertial measurement data, and tracking errors of the initial reference gait sequence as the observation space, to learn an end-to-end mapping from the initial gait reference sequence to joint torque commands. A reward function is formed by superimposing foot-ground contact consistency rewards, fuselage stability rewards, and energy efficiency rewards, supplemented by joint limit penalties, self-collision penalties, and joint torque over-limit penalties. The underlying motion tracking strategy is trained based on this reward function. This underlying motion tracking strategy is then deployed in the foot-ground interaction sensing motion tracking module, enabling the module to possess the end-to-end mapping capability from the reference gait sequence to joint torque commands.

[0071] This module employs reinforcement learning algorithms, such as Proximal Policy Optimization (PPO), for training. The observation space (i.e., the input information that the agent can perceive) includes joint relative pose, foot contact state, fuselage inertial measurement data, and tracking error. Specifically, joint relative pose represents the deviation of each joint from the reference pose; foot contact state indicates whether each foot is currently touching the ground (a binary signal); fuselage inertial measurement data includes fuselage angular velocity and linear acceleration acquired through the Inertial Measurement Unit (IMU); and tracking error indicates the difference between the current state and the reference gait sequence.

[0072] In designing the reward function, this implementation adds several reward and penalty items to the original tracking error reward of BeyondMimic. The basic reward is the tracking error reward, which encourages the robot to closely match the reference gait. New rewards include foot-ground contact consistency rewards, body stability rewards, and energy efficiency rewards. The foot-ground contact consistency reward rewards the robot's foot contact / lift-off timing for consistency with the reference sequence; the body stability reward rewards minimal fluctuations in body pitch and roll angles; and the energy efficiency reward rewards low joint torque output (energy saving). Penalty items include joint limit penalties, self-collision penalties, and torque over-limit penalties. The joint limit penalty applies a negative reward when the joint angle approaches its limit; the self-collision penalty applies a negative reward when limbs collide with each other or when limbs collide with the body; and the torque over-limit penalty applies a negative reward when the joint torque exceeds the motor's rated value.

[0073] Based on the aforementioned observation space and reward function, an end-to-end policy network is trained using the PPO algorithm. This network can directly map from a reference gait sequence to joint torque commands. After training, the policy is deployed in the foot-ground interaction sensing motion tracking module, enabling it to track the reference gait in real time and output torque commands.

[0074] In one implementation, the visual and terrain-based dual-conditional gait diffusion model can be constructed according to the following steps: The conditional input and observational input of the gait diffusion model are obtained. The embedding vector output by the terrain semantic encoder after encoding real-time visual semantic features and terrain point clouds is used as the conditional input, and the aircraft state is used as the observational input. Based on the conditional and observational inputs, a future gait sequence covering at least two complete gait cycles is generated through the gait diffusion model. Based on the future gait sequence, a classifier-guided optimization is performed using terrain and gait causal graphs to ensure that the generated future gait sequence conforms to the physical causal constraints between terrain and gait, resulting in an upper-level gait planning strategy with terrain generalization capabilities. The upper-level gait planning strategy is deployed on the visual and terrain dual-conditional gait diffusion model, enabling the model to perform end-to-end optimization of the initial gait reference sequence and generate multiple sets of candidate gait sequences.

[0075] In the above embodiments, the vision-terrain dual-conditional gait diffusion model adopts a conditional diffusion model architecture. The conditional input of this model is an embedding vector output by the terrain semantic encoder after encoding real-time visual semantic features and terrain point clouds. This embedding vector represents the terrain constraints that must be satisfied to generate the gait. The observation input of the model is the robot's state, including motion information such as the robot's current position, orientation, and speed. Furthermore, the model's input may further include 3D point cloud data of the terrain ahead, as well as real-time visual semantic features such as terrain type and obstacles extracted from images from the airborne camera via the visual encoder.

[0076] Based on the aforementioned conditional and observational inputs, the gait diffusion model outputs a future gait sequence covering at least two complete gait cycles (e.g., approximately 2 seconds). Compared to conventional methods that typically predict only a short gait of 0.64 seconds, the embodiments of this disclosure possess a longer-term gait planning capability, enabling early prediction of terrain changes and thus achieving superior terrain adaptability and gait smoothness.

[0077] After generating future gait sequences, a classifier-guided optimization is further performed using terrain and gait causal graphs to ensure the generated sequences conform to the physical causal constraints between terrain and gait (e.g., in sloping terrain, the forelimb stance phase is prolonged and the hip joint moment increases). This ultimately yields a higher-level gait planning strategy with terrain generalization capabilities. Deploying this strategy to the aforementioned vision-and-terrain dual-conditional gait diffusion model enables the model to perform end-to-end optimization of the initial gait reference sequence and generate multiple sets of candidate gait sequences. It can generate diverse gait schemes based on different terrain conditions, providing a rich candidate set for the subsequent scoring network.

[0078] S106: The gait reference sequence is verified in a digital twin simulation environment reconstructed from the video clip, and the gait control strategy corresponding to the verified gait reference sequence is deployed to the robot to perform real-time gait control.

[0079] Here, a digital twin environment refers to a 1:1 virtual simulation scene reconstructed from the input video (e.g., using 3D Gaussian splashing or Neural Radiation Field (NeRF) technology to recreate the real terrain in the video). In this environment, a gait reference sequence is simulated to verify its physical feasibility, stability, and safety under real terrain conditions. After successful verification, the corresponding gait control strategy is deployed to a real robot, and real-time gait control is executed through a real-time control system (such as a fast-slow loop control architecture), completing the entire process from monocular video input to real-machine gait implementation.

[0080] In one implementation, to ensure the physical feasibility and robustness of the gait reference sequence on real terrain, the verification of the gait reference sequence in a digital twin simulation environment reconstructed from the video clip in step S106 can be achieved using the following scheme: Based on the video clips, the geometric structure and visual texture of the real terrain scene in the video are reconstructed using a 3D reconstruction algorithm. The reconstructed terrain scene is then replicated onto a digital twin simulation platform to obtain a digital twin simulation environment. Based on the digital twin simulation environment, core causal variables affecting gait are identified through causal inference. Based on the distribution of real-world physical parameters recovered from the video clips, the core causal variables in the gait reference sequence are subjected to adversarial randomization to complete the simulation-to-real-world domain adaptation training of the gait reference sequence. Based on the gait reference sequence after domain adaptation training, parallel simulation tests are conducted in the digital twin simulation environment. Extreme adversarial scenarios, including lateral impact, frictional abrupt changes, and sensor noise, are generated for robustness verification. Simulation failure cases are automatically fed back to the training set for incremental fine-tuning to obtain a verified gait reference sequence.

[0081] This implementation constructs a digital twin environment that maps to the real-world scene in a 1:1 ratio, where domain adaptation and robustness verification are performed. Specifically, firstly, based on the original video clips, 3D reconstruction algorithms (such as 3D Gaussian splash or NeRF) are used to automatically reconstruct the geometric structure (e.g., ground undulation, slope) and visual texture (e.g., material, color) of the real terrain scene in the video. The reconstructed scene is then completely replicated onto the digital twin simulation platform, resulting in a simulation sandbox highly consistent with the video shooting environment. Secondly, causal inference techniques are used to identify core causal variables affecting quadrupedal gait stability (e.g., ground friction coefficient, ground stiffness). Based on the distribution of real-world physical parameters recovered from the video, these core variables are subjected to adversarial randomization. This means that parameter values ​​under extreme or boundary conditions are actively generated in the simulation, allowing the model to adapt to potential dynamic deviations in the real world during the simulation phase, thus completing the domain adaptation training from simulation to reality. Finally, large-scale parallel simulation tests are conducted in the adapted digital twin environment, automatically generating extreme adversarial scenarios such as sudden lateral impacts, sudden drops in ground friction, and sensor noise, comprehensively verifying the robustness of the gait reference sequence.

[0082] After completing parallel simulation testing in the digital twin simulation environment, all failure cases (including gait instability, foot slippage, and fuselage tilting) are automatically collected. These failure cases are automatically fed back into the training set for incremental fine-tuning of the gait strategy. Through this mechanism, a closed-loop iteration of training-validation-optimization is formed, enabling the model to continuously improve during the simulation phase and further narrowing the gap between simulation and reality.

[0083] In one implementation, in order to efficiently deploy the verified gait reference sequence to a real robot and achieve real-time stable control, step S106 deploys the gait control strategy corresponding to the verified gait reference sequence to the robot, which can be specifically implemented using the following scheme: The gait strategy corresponding to the verified gait reference sequence is optimized using lightweight optimization. This lightweight optimization includes structured pruning, model quantization, and operator fusion for the robot's onboard hardware, and hardware-specific compilation optimization to obtain a lightweight gait strategy. Based on the lightweight gait strategy, the terrain semantic encoder and gait diffusion model are run at a first frequency, with real-time video stream from the onboard camera as input, and a gait planning sequence adapted to the current terrain as output. The motion tracking model is run at a second frequency, with inertial measurement data of the robot body, joint encoder data, and foot force sensor data as input, and joint torque commands as output, to complete gait tracking and real-time balance control. The second frequency is greater than the first frequency. During real-time control, the robot's pitch and roll angles, joint torques, and foot slippage status are monitored. When any parameter exceeds a preset safety threshold, the robot switches to a safe gait or performs an emergency stop.

[0084] The above implementation mainly focuses on lightweight model optimization and real-time control architecture design. First, addressing the computational resource limitations of the target robot's onboard hardware, the validated gait strategy is optimized for lightweight performance. This includes structured pruning (removing redundant network channels), model quantization (compressing floating-point parameters to INT8 precision), and operator fusion (merging consecutive computational operations). Hardware-specific compilation optimization is also performed, significantly reducing inference latency while ensuring accuracy loss is less than 1.5%. Second, a dual-frequency real-time control architecture with fast and slow loops is designed: the slow loop runs the terrain semantic encoder and gait diffusion model at a lower frequency (e.g., 50Hz), receiving real-time video streams from the onboard camera and outputting a long-term gait planning sequence adapted to the current terrain; the fast loop runs the underlying motion tracking model at a higher frequency (e.g., 1kHz), receiving real-time data from the robot's IMU, joint encoders, and foot force sensors, outputting joint torque commands to achieve accurate gait tracking and dynamic balance control. The fast and slow loops work together, balancing terrain perception planning capabilities with the real-time performance of underlying control. Finally, during real-time control, the embedded safety fallback module continuously monitors the robot's pitch and roll angles, joint torques, and foot slippage. If any parameter exceeds the preset safety threshold, a safety strategy is immediately triggered, such as switching to a static stable gait or performing an emergency stop, ensuring the robot's operational safety throughout the process.

[0085] In some embodiments, in order to overcome the limitations of traditional single-round offline deployment and enable robots to have the ability to continuously and autonomously evolve, some embodiments of this disclosure also construct a self-supervised lifelong learning closed loop of edge-cloud collaboration.

[0086] Specifically, the gait control strategy deployed to the robot is used as an edge strategy. During the robot's task execution, monocular video data and execution feedback data (including gait tracking errors, energy consumption data, foot slippage and instability records, etc.) in new scenarios are collected. The video data and execution feedback data in the new scenarios are uploaded to the cloud. In the cloud, the steps of acquiring monocular video data and performing preprocessing, motion primitive decomposition, 3D posture reconstruction, dynamic data generation, and redirected trajectory mapping are re-executed to generate incremental gait primitives and store them in the gait library. Based on the incremental gait primitives, the edge strategy is incrementally fine-tuned, and the update magnitude of the weights corresponding to the original gait skills is constrained by the Elastic Weight Consolidation (EWC) algorithm to protect the learned core skills. Finally, the updated gait strategy is distributed to the robot. For example, the updated gait strategy can be remotely distributed to the robot via OTA (Over-The-Air) to complete the online upgrade of the edge strategy, forming a lifelong learning closed loop of deployment-collection-learning-update, enabling the robot to continuously evolve in practical applications.

[0087] Specifically, this implementation constructs a self-supervised lifelong learning mechanism that integrates edge and cloud environments. First, the gait control strategy already deployed on the real robot is used as the edge strategy. During actual task execution, the robot continuously collects monocular video data (i.e., new terrain and working conditions) and execution feedback data (such as gait tracking errors, energy consumption data, and whether slippage or instability occurs). This data is uploaded to a cloud server. In the cloud, the complete training process is re-executed, including video preprocessing, motion primitive decomposition, 3D pose reconstruction, dynamics data generation, and redirected trajectory mapping. Incremental gait primitives are extracted from the new data and stored in a gait library for knowledge accumulation. Subsequently, based on these incremental gait primitives, the original edge gait strategy is incrementally fine-tuned (i.e., the model parameters are updated without complete retraining). During fine-tuning, the Elastic Weight Consolidation (EWC) algorithm is used to protect learned core skills and prevent catastrophic forgetting by constraining the update magnitude of the weights corresponding to the original gait skills. Finally, the updated gait strategy is downloaded to the robot via over-the-air (OTA) or wirelessly, completing the online upgrade of the edge strategy. This achieves a closed-loop learning process of deployment, data collection, learning, and updating, enabling the robot to continuously evolve in practical applications.

[0088] In this embodiment, to ensure the stable operation and long-term maintainability of the system, several risk control measures can be implemented, such as catastrophic forgetting suppression: During incremental model training, the EWC algorithm is used to apply protective constraints to the network weights corresponding to the original gait skills, limiting their variation during incremental updates and preventing significant performance degradation of old skills due to new task learning; simultaneously, the gait library has a built-in version management mechanism, supporting one-click rollback to any historical stable version, ensuring rapid recovery in the event of performance degradation. Another example is the hardware adaptation abstraction layer: A standardized hardware abstraction layer is constructed to address the hardware differences between different quadruped robot models, pre-completing adaptation for mainstream quadruped robot models and providing standardized configuration interfaces to support rapid expansion to new robot models. Furthermore, data compliance auditing is implemented: Addressing copyright issues related to internet video data, a data compliance auditing mechanism is constructed, using only open-source academic datasets, copyright-free publicly available videos, and custom-shot videos as input sources to ensure all training data complies with relevant regulations.

[0089] By adopting the EWC algorithm, the model can retain over 95% of its original gait skills when incrementally learning gaits for new scenarios, effectively avoiding catastrophic forgetting. Combined with the gait library version management mechanism, performance degradation can be detected and rolled back in seconds, ensuring the long-term stability and reliability of the deployed model. Through this standardized hardware abstraction layer, the adaptation time for adding a new quadruped robot model can be shortened from several weeks to 2-3 days, reducing adaptation costs by over 80%, significantly lowering the engineering implementation threshold of the technical solution. The data compliance review mechanism ensures the traceability of all training data sources, eliminating the commercial risks of technical solutions due to data copyright issues.

[0090] Through experimental verification and data statistical calculation, compared with general methods, the above-mentioned method of this disclosure embodiment can significantly reduce data acquisition costs and joint reprojection errors, greatly improve physical accuracy and Sim2Real transfer success rate, and effectively reduce scoring prediction errors and end-to-end inference latency.

[0091] Figure 2 This is a schematic diagram illustrating the overall logic of gait control for a quadruped robot, as described in an exemplary embodiment of this disclosure. Figure 2 As shown, using monocular video as input, a quadruped robot can learn physically feasible gaits from videos in an unsupervised manner without the need for optical motion capture systems, manual annotation, or low cost, and possess the ability to evolve autonomously throughout its life.

[0092] Specifically, based on monocular video data, a terrain-gait causal gait library is constructed, decomposing continuous motion into reusable gait primitives with terrain labels. Simultaneously, neurophysical reconstruction is performed on the monocular video to recover 3D pose and dynamic information, and cross-species redirection is completed, mapping biological motion to robot-executable trajectories. Building upon this, a two-layer hierarchical imitation learning architecture is employed for gait generation and optimization. The upper layer handles terrain perception and long-term gait planning, while the lower layer handles high-frequency motion tracking and torque control. Together, they generate physically feasible and terrain-adaptive gait reference sequences. Subsequently, based on the generated gait reference sequences, 1:1 scene reconstruction and adversarial randomization verification are performed in a digital twin simulation environment. After successful verification, the system is deployed to the robot's edge, achieving stable gait control on the real machine through a real-time fast-slow loop control architecture (50Hz slow loop planning, 1kHz fast loop execution), with a safety fallback mechanism to ensure operational safety. In addition, it supports incremental learning through edge and cloud collaboration: the robot continuously collects new data during actual operation, uploads it to the cloud to generate incremental gait primitives and fine-tunes the model, and then remotely distributes updates via OTA to form a lifelong learning closed loop, enabling the robot to continuously evolve.

[0093] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0094] like Figure 3 As shown in the illustration, this disclosure also provides a schematic diagram of a robot gait control device 300 based on monocular video, comprising: The acquisition module 31 is used to acquire monocular video data and perform preprocessing to obtain video segments in units of gait cycles. The gait library establishment module 32 is used to decompose the video clip into motion primitives, extract gait primitives, label each gait primitive with the corresponding terrain type, and store the labeled gait primitives into the gait library. The three-dimensional pose reconstruction module 33 is used to reconstruct a three-dimensional pose sequence from the video clip; The redirection trajectory generation module 34 is used to generate dynamic data containing joint torques and foot forces based on the three-dimensional posture sequence by performing joint kinematic and dynamic optimization, and to map the dynamic data into an executable redirection trajectory of the robot with kinematic and dynamic feasibility as a hard constraint. The gait reference sequence generation module 35 is used to match gait primitives with corresponding terrain annotations from the gait library based on the terrain features extracted from the video clip, and splice them according to the temporal requirements of the redirected trajectory to generate a gait reference sequence. The gait control strategy deployment module 36 is used to verify the gait reference sequence in a digital twin simulation environment reconstructed from the video clip, and to deploy the gait control strategy corresponding to the verified gait reference sequence to the robot to perform real-time gait control.

[0095] Optionally, the acquisition module 31 acquires monocular video data and performs preprocessing, including: selecting valid video segments from the monocular video data based on a semantic classification model and a temporal consistency detection algorithm; and after performing frame rate upscaling, resolution normalization, and noise reduction on the valid video segments, dividing them into video segments based on complete gait cycles.

[0096] Optionally, the gait library establishment module 32 performs motion primitive decomposition on the video segment, decomposes it into gait primitives, and labels each gait primitive with the corresponding terrain type. The labeled gait primitives are then stored in the gait library. This includes: based on the video segment, labeling the start and end times of the support phase, the start and end times of the swing phase, and the foot-ground contact switching nodes for each gait cycle to obtain gait temporal labels; based on the gait temporal labels, mining the causal relationship between terrain semantic features, gait phases, joint movements, and foot force using a temporal causal discovery algorithm; based on the causal relationship, decomposing the continuous motion data in the video segment into flat ground gait primitives, slope gait primitives, stair gait primitives, disturbance-resistant balance primitives, and high-speed motion primitives, and labeling each gait primitive with the corresponding terrain type, movement speed range, joint torque constraints, and foot-ground contact constraints to obtain gait primitives carrying physical attributes; and storing the gait primitives with physical attribute labels in the gait library after multi-dimensional classification according to terrain type, movement mode, and gait type.

[0097] Optionally, the 3D pose reconstruction module 33 reconstructs a 3D pose sequence from the video clip, including: extracting the contour mask of the moving subject in the video clip using a visual segmentation model based on the video clip; performing dense visual feature alignment using a visual feature alignment model in the region defined by the contour mask to extract key point features of the quadrupedal torso and limbs; performing temporal optimization and completing occluded joints using a temporal optimization network based on the key point features to obtain a smooth 2D joint temporal sequence; performing initial fitting of the 3D skeleton using a parametric skeleton model based on the 2D joint temporal sequence, with joint reprojection error as the optimization target, to obtain a coarsely reconstructed 3D pose sequence; performing fine alignment of gait cycles using a dynamic time warping algorithm based on the coarsely reconstructed 3D pose sequence to obtain an aligned 3D pose sequence; and performing global optimization using a temporal optimization network with built-in gait cycle priors and left and right leg symmetry constraints based on the aligned 3D pose sequence, and performing interpolation and filtering on the optimized 3D pose sequence to obtain a processed 3D pose sequence.

[0098] Optionally, the redirection trajectory generation module 34 generates dynamic data containing joint torques and foot forces based on the three-dimensional posture sequence by performing joint kinematic and dynamic optimization. This includes: based on the three-dimensional posture sequence, outputting joint angle sequences, foot trajectory sequences, and fuselage pose sequences through the kinematic branch of the joint kinematic and dynamic optimization network; based on the three-dimensional posture sequence, calculating the foot force and torque sequences and joint torque sequences through the dynamic branch of the joint kinematic and dynamic optimization network, using rigid body dynamics equations as hard constraints; the dynamic branch embeds a differentiable physics simulator; and based on the joint angle sequences, foot trajectory sequences, fuselage pose sequences, foot force and torque sequences, and joint torque sequences, introducing a contact loss function to constrain the non-penetration and non-slip conditions between the foot and the ground, thereby obtaining dynamic data that satisfies physical consistency.

[0099] Optionally, the redirection trajectory generation module 34 uses kinematic and dynamic feasibility as hard constraints to map the dynamic data into an executable redirection trajectory for the robot. This includes: based on the dynamic data, and considering the differences in joint configuration between the moving subject and the robot in the video clip, using foot-to-ground contact timing consistency, joint torque upper limit constraints, and body balance stability as hard constraints, mapping the dynamic data to the robot's joint space. The mapping process includes correcting for joint range of motion exceeding limits, foot suspension, foot penetration, and body tipping issues, while preserving the natural dynamic characteristics of the original gait, thus obtaining an executable redirection trajectory for the robot.

[0100] Optionally, the gait reference sequence generation module 35, based on the terrain features extracted from the video clip, matches gait primitives with corresponding terrain annotations from the gait library, and splices them according to the temporal requirements of the redirection trajectory to generate a gait reference sequence. This includes: matching gait primitives with corresponding terrain annotations from the gait library based on the terrain features extracted from the video clip, splicing and physically verifying them according to the temporal requirements of the redirection trajectory to obtain an initial gait reference sequence; performing end-to-end optimization of the initial gait reference sequence through a dual-layer imitation learning architecture to obtain multiple sets of candidate gait sequences; the dual-layer imitation learning architecture includes a foot-ground interaction perception motion tracking module and a vision-and-terrain dual-conditional gait diffusion model; and comprehensively scoring the multiple sets of candidate gait sequences through a multi-dimensional scoring network, selecting the optimal gait sequence based on the comprehensive scoring results, and using the optimal gait sequence for subsequent simulation verification and real device deployment.

[0101] Optionally, the construction process of the foot-ground interaction sensing motion tracking module includes: based on the initial gait reference sequence, using a reinforcement learning algorithm, with the joint relative pose, foot contact state, fuselage inertial measurement data, and tracking error of the initial reference gait sequence as the observation space, learning the end-to-end mapping from the initial gait reference sequence to joint torque commands; using the tracking error reward as the base reward, superimposing foot-ground contact consistency rewards, fuselage stability rewards, and energy efficiency rewards, and supplemented by joint limit penalties, self-collision penalties, and joint torque over-limit penalties, forming a reward function; training the underlying motion tracking strategy based on the reward function; and deploying the underlying motion tracking strategy in the foot-ground interaction sensing motion tracking module, so that the foot-ground interaction sensing motion tracking module has the end-to-end mapping capability from the reference gait sequence to joint torque commands.

[0102] Optionally, the construction process of the visual and terrain-based dual-conditional gait diffusion model includes: obtaining the conditional input and observational input of the gait diffusion model; wherein, the embedding vector output by the terrain semantic encoder after encoding real-time visual semantic features and terrain point clouds is used as the conditional input, and the aircraft state is used as the observational input; based on the conditional input and observational input, a future gait sequence covering at least two complete gait cycles is generated through the gait diffusion model; based on the future gait sequence, a classifier-guided optimization is performed through terrain and gait causal graphs to ensure that the generated future gait sequence conforms to the physical causal constraints between terrain and gait, thereby obtaining an upper-level gait planning strategy with terrain generalization ability; the upper-level gait planning strategy is deployed on the visual and terrain-based dual-conditional gait diffusion model to enable the visual and terrain-based dual-conditional gait diffusion model to perform end-to-end optimization of the initial gait reference sequence and generate multiple sets of candidate gait sequences.

[0103] Optionally, the gait control strategy deployment module 36 verifies the gait reference sequence in a digital twin simulation environment reconstructed from the video clip, including: reconstructing the geometric structure and visual texture of the real terrain scene in the video using a 3D reconstruction algorithm based on the video clip, and replicating the reconstructed terrain scene to the digital twin simulation platform to obtain a digital twin simulation environment; based on the digital twin simulation environment, identifying the core causal variables affecting gait through causal inference, and performing adversarial randomization processing on the core causal variables in the gait reference sequence based on the distribution of real-world physical parameters recovered from the video clip, to complete the simulation-to-real-world domain adaptation training of the gait reference sequence; based on the gait reference sequence after completing the domain adaptation training, performing parallel simulation tests in the digital twin simulation environment, generating extreme adversarial scenarios including lateral impact, frictional abrupt changes, and sensor noise for robustness verification, and automatically feeding back simulation failure cases to the training set for incremental fine-tuning to obtain a verified gait reference sequence.

[0104] Optionally, the gait control strategy deployment module 36 deploys the verified gait reference sequence to the robot for real-time gait control, including: performing lightweight optimization on the gait strategy corresponding to the verified gait reference sequence, wherein the lightweight optimization includes structured pruning, model quantization and operator fusion for the robot's onboard hardware, and completing hardware-specific compilation optimization to obtain a lightweight gait strategy; based on the lightweight gait strategy, running the terrain semantic encoder and gait diffusion model at a first frequency, inputting the real-time video stream from the onboard camera, and outputting a gait planning sequence adapted to the current terrain; running the motion tracking model at a second frequency, inputting the inertial measurement data of the robot body, joint encoder data and foot force sensor data, and outputting joint torque commands to complete gait tracking and real-time balance control; the second frequency is greater than the first frequency; during real-time control, the pitch and roll angles of the robot body, joint torques and foot slippage status are monitored by the safety fallback module, and when any parameter is detected to exceed a preset safety threshold, switching to a safe gait or performing an emergency stop is performed.

[0105] Optionally, the gait control strategy deployment module 36 is further configured to: use the gait control strategy deployed to the robot as an edge strategy; collect monocular video data and execution feedback data in a new scenario during the robot's task execution; upload the video data and execution feedback data in the new scenario to the cloud; re-execute the steps of acquiring monocular video data and performing preprocessing, motion primitive decomposition, three-dimensional posture reconstruction, dynamic data generation, and redirected trajectory mapping in the cloud; generate incremental gait primitives and store them in the gait library; perform incremental fine-tuning of the edge strategy based on the incremental gait primitives; constrain the update magnitude of the weights corresponding to the original gait skills through an elastic weight consolidation algorithm; and distribute the updated gait strategy to the robot.

[0106] The detailed implementation process of each module of the robot gait control device 300 is essentially the same as the aforementioned method, and can be found in the description of the aforementioned method content, so it will not be repeated here.

[0107] Reference Figure 4 The diagram shown is a structural schematic of a device 400 according to an embodiment of this disclosure. The device 400 can be a computer device, a control chip, etc., and can be deployed on a server, a terminal, a robot, a remote control device, etc. It may include a processor 410, a memory 420, and a bus 430. The memory 420 is used to store execution instructions and includes main memory 421 and external memory 422. The main memory 421, also called internal memory, is used to temporarily store computational data in the processor 410 and data exchanged with external memory 422 such as a hard disk. The processor 410 exchanges data with the external memory 422 through the main memory 421.

[0108] In this embodiment, the memory 420 is specifically used to store application code executing the scheme of this disclosure, and its execution is controlled by the processor 410. That is, when the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430, or the processor 410 communicates with the memory 420 through other means, so that the processor 410 executes the application code stored in the memory 420, thereby executing the steps of the robot gait control method described in any of the foregoing embodiments. The memory 420 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 410 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor.

[0109] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot gait control method described in any of the above embodiments. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as DVD-ROM, DVD-RAM, DVD-RW, DVD+RW, CD-ROM, CD-RW, CD-RW, and MO (magneto-optical) storage media; and semiconductor storage media, such as flash memory, EEPROM, Dynamic Random Access Memory (DRAM), and Static Random Access Memory (SRAM).

[0110] The computer program can be written in various computer programming languages, including but not limited to C, C++, Python, and custom messages and services under the ROS framework. When the computer program is executed by the processor, it implements the various steps of the robot gait control method in the embodiments of this disclosure.

[0111] This disclosure also provides a computer program product storing a computer program. When executed by a processor, the computer program performs the steps of the robot gait control method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here. The computer program product can be implemented using hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and apparatuses described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A monocular video based robot gait control method, characterized in that, include: Acquire monocular video data and preprocess it to obtain video segments in units of gait cycles; The video clip is decomposed into motion primitives to extract gait primitives. Each gait primitive is labeled with the corresponding terrain type, and the labeled gait primitives are stored in the gait library. A three-dimensional pose sequence was reconstructed from the video clip; Based on the three-dimensional posture sequence, dynamic data containing joint torques and foot forces are generated through joint kinematic and dynamic optimization. The dynamic data is then mapped to the robot's executable retargeting trajectory with kinematic and dynamic feasibility as hard constraints. Based on the terrain features extracted from the video clips, gait primitives with corresponding terrain annotations are matched from the gait library and spliced ​​according to the temporal requirements of the redirection trajectory to generate a gait reference sequence. The gait reference sequence is validated in a digital twin simulation environment reconstructed from the video clip, and the gait control strategy corresponding to the validated gait reference sequence is deployed to the robot to perform real-time gait control.

2. The method of claim 1, wherein, Acquire monocular video data and perform preprocessing, including: Based on a semantic classification model and a temporal consistency detection algorithm, valid video segments are selected from the monocular video data; After performing frame rate upscaling, resolution normalization, and noise reduction on the effective video segments, they are divided into video segments based on complete gait cycles.

3. The method of claim 1, wherein, The video clip is decomposed into motion primitives to extract gait primitives. Each gait primitive is labeled with its corresponding terrain type, and the labeled gait primitives are stored in a gait library, including: Based on the video clips, the start and end times of the support phase, the start and end times of the swing phase, and the foot contact switching nodes of each gait cycle are labeled to obtain gait timing labels. Based on the gait temporal tags, a temporal causal discovery algorithm is used to mine the causal relationship between terrain semantic features, gait phase, joint movements and foot force. Based on the causal relationship, the continuous motion data in the video segment is decomposed into flat gait primitives, slope gait primitives, stair gait primitives, disturbance-resistant balance primitives and high-speed motion primitives. Each gait primitive is labeled with the corresponding terrain type, movement speed range, joint torque constraints and foot contact constraints to obtain gait primitives carrying physical attributes. The gait primitives labeled with physical attributes are classified into multiple dimensions according to terrain type, movement pattern and gait type and then stored in the gait library.

4. The method of claim 1, wherein, Reconstructing a three-dimensional pose sequence from the video clip includes: Based on the video clip, the contour mask of the moving subject in the video is extracted by a visual segmentation model; in the area defined by the contour mask, dense visual feature alignment is performed by a visual feature alignment model to extract key point features of the quadrupedal torso and limbs. Based on the key point features, a temporal optimization network is used to optimize the temporal sequence and fill in the occluded joints, resulting in a smooth two-dimensional joint temporal sequence. Based on the two-dimensional joint time sequence, with the joint reprojection error as the optimization target, the initial fitting of the three-dimensional skeleton is completed through the parameterized skeleton model to obtain the coarsely reconstructed three-dimensional pose sequence. Based on the coarsely reconstructed 3D pose sequence, the gait cycle is finely aligned using a dynamic time warping algorithm to obtain the aligned 3D pose sequence. Based on the aligned 3D pose sequence, global optimization is performed through a temporal optimization network with built-in gait cycle prior and left and right leg symmetry constraints. The optimized 3D pose sequence is then interpolated and filtered to obtain the processed 3D pose sequence.

5. The method of claim 1, wherein, Based on the aforementioned three-dimensional posture sequence, dynamic data including joint torques and foot forces are generated through joint kinematic and dynamic optimization, including: Based on the three-dimensional posture sequence, the kinematic branch of the kinematic and dynamic dual-branch joint optimization network is used to output the joint angle sequence, foot trajectory sequence and fuselage pose sequence. Based on the three-dimensional posture sequence, the dynamics branch of the kinematics and dynamics dual-branch joint optimization network is used to calculate the foot force and torque sequence and the joint torque sequence with rigid body dynamics equations as hard constraints; the dynamics branch embeds a differentiable physics simulator. Based on the joint angle sequence, foot trajectory sequence, fuselage pose sequence, foot force and torque sequence, and joint torque sequence, a contact loss function is introduced to constrain the non-penetration condition and non-slip condition between the foot and the ground, thereby obtaining dynamic data that satisfies physical consistency.

6. The method of claim 1, wherein, Using kinematic and dynamic feasibility as hard constraints, the dynamic data is mapped to an executable rerouting trajectory for the robot, including: Based on the dynamic data, and considering the differences in joint configuration between the moving subject and the robot in the video clip, the dynamic data is mapped to the robot's joint space with hard constraints such as consistency of foot-ground contact timing, upper limit constraint of joint torque, and body balance stability. The mapping process includes correcting issues such as excessive joint range of motion, foot suspension, foot penetration, and body tipping, while preserving the natural dynamic characteristics of the original gait, to obtain a redirected trajectory that the robot can execute.

7. The method of claim 1, wherein, Based on the terrain features extracted from the video clips, gait primitives with corresponding terrain annotations are matched from the gait library and spliced ​​according to the temporal requirements of the redirection trajectory to generate a gait reference sequence, including: Based on the terrain features extracted from the video clips, gait primitives with corresponding terrain annotations are matched from the gait library and spliced ​​and physically verified according to the temporal requirements of the redirected trajectory to obtain an initial gait reference sequence. The initial gait reference sequence is optimized end-to-end using a dual-layer imitation learning architecture to obtain multiple candidate gait sequences. The dual-layer imitation learning architecture includes a foot-ground interaction perception motion tracking module and a vision-and-terrain dual-conditional gait diffusion model. The candidate gait sequences are comprehensively scored using a multi-dimensional scoring network. The optimal gait sequence is then selected based on the comprehensive scoring results and used for subsequent simulation verification and real device deployment.

8. The method of claim 7, wherein, The foot-ground interaction sensing motion tracking module is constructed according to the following steps: Based on the initial gait reference sequence, a reinforcement learning algorithm is used to learn the end-to-end mapping from the initial gait reference sequence to joint torque commands, using the joint relative pose, foot contact state, fuselage inertial measurement data and tracking error of the initial reference gait sequence as the observation space. The tracking error reward is used as the base reward, and the foot contact consistency reward, fuselage stability reward and energy efficiency reward are superimposed. In addition, joint limit penalty, self-collision penalty and joint torque over-limit penalty are added to form a reward function. The underlying motion tracking strategy is trained based on the reward function. The underlying motion tracking strategy is deployed in the foot-ground interaction sensing motion tracking module so that the foot-ground interaction sensing motion tracking module has end-to-end mapping capability from reference gait sequence to joint torque command.

9. The method of claim 7, wherein, The visual and terrain-based dual-conditional gait diffusion model is constructed according to the following steps: The conditional input and observational input of the gait diffusion model are obtained; the embedding vector output by the terrain semantic encoder after encoding real-time visual semantic features and terrain point cloud is used as the conditional input, and the fuselage state is used as the observational input. Based on the conditional input and observational input, a future gait sequence covering at least two complete gait cycles is generated using a gait diffusion model; Based on the future gait sequence, a classifier-guided optimization is performed using terrain and gait causal graphs to ensure that the generated future gait sequence conforms to the physical causal constraints between terrain and gait, thus obtaining an upper-level gait planning strategy with terrain generalization capability. The upper-level gait planning strategy is deployed on the visual and terrain-based dual-conditional gait diffusion model, so that the visual and terrain-based dual-conditional gait diffusion model has the ability to perform end-to-end optimization of the initial gait reference sequence and generate multiple sets of candidate gait sequences.

10. The method of claim 1, wherein, Validating the gait reference sequence in a digital twin simulation environment reconstructed from the video clips includes: Based on the video clip, the geometric structure and visual texture of the real terrain scene in the video are reconstructed using a 3D reconstruction algorithm, and the reconstructed terrain scene is replicated onto a digital twin simulation platform to obtain a digital twin simulation environment. Based on the digital twin simulation environment, the core causal variables affecting gait are identified through causal inference. Based on the distribution of real-world physical parameters recovered from the video clips, the core causal variables in the gait reference sequence are subjected to adversarial randomization processing to complete the simulation-to-real-world domain adaptation training of the gait reference sequence. Based on the gait reference sequence after domain adaptation training, parallel simulation tests are performed in the digital twin simulation environment to generate extreme adversarial scenarios including lateral impact, frictional abrupt changes, and sensor noise for robustness verification. Simulation failure cases are automatically fed back to the training set for incremental fine-tuning to obtain the verified gait reference sequence.

11. The method of claim 1, wherein, Deploying the validated gait reference sequence to the robot for real-time gait control includes: The gait strategy corresponding to the verified gait reference sequence is subjected to lightweight optimization. The lightweight optimization includes structured pruning, model quantization and operator fusion for the robot's onboard hardware, and hardware-specific compilation optimization to obtain a lightweight gait strategy. Based on the lightweight gait strategy, the terrain semantic encoder and gait diffusion model are run at a first frequency, the real-time video stream from the airborne camera is input, and the gait planning sequence adapted to the current terrain is output. The motion tracking model is run at a second frequency, the fuselage inertial measurement data, joint encoder data and foot force sensor data are input, and the joint torque command is output to complete gait tracking and real-time balance control. The second frequency is greater than the first frequency. During real-time control, the safety fallback module monitors the pitch and roll angles, joint torques, and foot slippage. When any parameter exceeds the preset safety threshold, the system switches to a safe gait or performs an emergency stop.

12. The method of claim 1, wherein, Also includes: The gait control strategy deployed to the robot is used as an edge strategy to collect monocular video data and execution feedback data in new scenarios during the robot's task execution. The video data and execution feedback data in the new scenario are uploaded to the cloud. The steps of acquiring monocular video data and performing preprocessing, motion primitive decomposition, 3D posture reconstruction, dynamic data generation and redirected trajectory mapping are re-executed in the cloud to generate incremental gait primitives and store them in the gait library. The edge strategy is incrementally fine-tuned based on the incremental gait primitives, and the update magnitude of the weights corresponding to the original gait skills is constrained by the elastic weight consolidation algorithm. The updated gait strategy is then sent to the robot.

13. A monocular video based robot gait control apparatus, characterized by, include: The acquisition module is used to acquire monocular video data and preprocess it to obtain video segments in units of gait cycles. The gait library building module is used to decompose the video clip into motion primitives, extract gait primitives, label each gait primitive with the corresponding terrain type, and store the labeled gait primitives into the gait library. A 3D pose reconstruction module is used to reconstruct a 3D pose sequence from the video clip; The redirection trajectory generation module is used to generate dynamic data containing joint torques and foot forces based on the three-dimensional posture sequence by performing joint kinematic and dynamic optimization, and to map the dynamic data into an executable redirection trajectory for the robot with kinematic and dynamic feasibility as a hard constraint. The gait reference sequence generation module is used to match gait primitives with corresponding terrain annotations from the gait library based on the terrain features extracted from the video clips, and splice them according to the temporal requirements of the redirected trajectory to generate a gait reference sequence. The gait control strategy deployment module is used to verify the gait reference sequence in a digital twin simulation environment reconstructed from the video clip, and to deploy the gait control strategy corresponding to the verified gait reference sequence to the robot to perform real-time gait control.

14. An apparatus, comprising: It includes a processor and a memory, wherein the memory stores computer instructions, and the processor executes the computer instructions to perform the steps of the robot gait control method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the robot gait control method as described in any one of claims 1 to 12.

Images