Humanoid robot full-body motion generation method and system based on monocular video

Abstract

The application discloses a human-shaped robot full-body motion generation method and system based on monocular video, relates to the technical field of robot control, and comprises the following steps: pre-processing and multi-modal feature extraction are performed on the monocular video, each modal feature is fused to form a time sequence feature vector, and a three-dimensional motion sequence is obtained; the three-dimensional motion sequence is mapped to a target robot kinematics model to obtain an initial robot motion sequence; systematic post-processing and stability optimization are performed on the initial robot motion sequence, and an optimized robot motion sequence is output; kinematics quantities are calculated and converted into a motion data matrix, and standardized interface data used for simulation verification, reinforcement learning training and real robot control deployment are generated. The application can generate a full-body motion sequence from a monocular video, so that the generated robot motion can meet the kinematics and dynamics constraints of the robot while keeping the original human motion characteristics, thereby being more stably applied to robot simulation verification and real robot control deployment.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a method and system for generating full-body motion of a humanoid robot based on monocular video. Background Technology

[0002] The generation and control of motion in full-size, multi-degree-of-freedom humanoid robots has always been an important research direction in the field of robotics. In recent years, utilizing visual perception technology to recover human motion trajectories from ordinary videos and further generate robot actions has become an important way to reduce data acquisition costs and expand the sources of robot training data. Existing technologies mainly adopt the following approach: reconstructing the three-dimensional pose of the human body from the video using a monocular vision algorithm, and then mapping the human motion to the robot's kinematic model using a retargeting algorithm. However, the above approach has some shortcomings in practical applications: 1. Physical inconsistencies exist in the visual reconstruction results: The human posture trajectory obtained based on monocular vision estimation lacks strict alignment with the real physical space, especially with significant drift in the vertical direction, making the reconstructed trajectory unusable for robot control. Specifically, in non-jumping movements, the height of the human body root continuously rises, causing a "floating" phenomenon; or the feet penetrate the ground, causing physical penetration; in jumping movements, the aerial trajectory does not conform to the law of gravitational acceleration, exhibiting non-physical sawtooth fluctuations.

[0003] 2. Lack of physical trajectory reconstruction mechanism for jumping actions: Existing methods usually use interpolation or smoothing when dealing with actions with an airborne phase, such as jumping and somersaults, without introducing gravity constraints. This results in the generated aerial trajectory violating the laws of rigid body kinematics, which cannot guarantee the stability of the robot's landing posture.

[0004] 3. Lack of systematic and automated processing flow: Existing solutions often require manual completion of multiple steps, including human data conversion, motion redirection, trajectory repair, filtering stabilization and simulation verification. The process is complex and highly repetitive, making it difficult to form a stable engineering workflow.

[0005] 4. Lack of stable post-processing mechanism: Robot trajectories obtained directly from redirection often have problems such as root drift, abnormal foot contact, cross gait or posture imbalance. Without systematic repair, they are difficult to use for simulation verification or real deployment.

[0006] Based on this, a method and system for generating full-body motion of humanoid robots based on monocular video are now provided, which can eliminate the drawbacks of existing technical solutions. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for generating full-body motion of a humanoid robot based on monocular video, in order to solve the problems in the prior art where there is physical inconsistency in the visual reconstruction results, and the lack of a physical trajectory reconstruction mechanism, a systematic automatic processing flow and a stability post-processing mechanism for jumping movements.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for generating full-body motion of a humanoid robot based on monocular video includes the following steps: Step S1: Preprocess the monocular video and extract multimodal features. By linear mapping and element-wise addition, the features of each modality are fused to form a unified temporal feature vector. Based on the gravity view coordinate system, the three-dimensional motion sequence of the human body in the world coordinate system is recovered. The three-dimensional motion sequence includes the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient. Step S2: Map the three-dimensional motion sequence to the kinematic model of the target robot, establish the joint mapping relationship between the human body and the target humanoid robot, perform posture pre-alignment of the human body joint rotation, use a local non-uniform scaling strategy to match the bone segment ratio of the human body and the robot, obtain the target three-dimensional position of the corresponding body part of the robot, and then take the consistency of the rotation of the corresponding joints of the human body and the robot and the spatial alignment of the end effector position as the optimization objectives, combine the physical limit constraints of the robot joint angle, construct the differential inverse kinematics constraint optimization problem and solve iteratively to obtain the initial robot motion sequence; Step S3: Perform systematic post-processing and stability optimization on the initial robot motion sequence, including root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation, and output the optimized robot motion sequence; Step S4: Calculate the kinematic quantities through the optimized robot motion sequence. The kinematic quantities include root velocity, joint velocity, and motion velocity of various parts of the body. Convert the kinematic quantities into a unified motion data matrix to generate standardized interface data for simulation verification, reinforcement learning training, and real machine control deployment.

[0009] Further, step S1 specifically includes: Step S11: Receive monocular video from a monocular camera in real time or input video file to form a continuous video frame sequence. Use a target detection algorithm to perform human detection on each frame and generate the corresponding human bounding box. Normalize and enhance the human bounding box to obtain the preprocessed video frame sequence. Step S12: Extract multimodal features from the preprocessed video frame sequence. The multimodal features include two-dimensional keypoint features, image semantic features, camera motion features, and bounding box features. Step S13: Linearly map each modal feature using an independent multilayer perceptron to transform it into a unified dimensional feature space, and fuse each modal feature into a unified temporal feature vector by adding elements one by one. Step S14: Recover the global motion trajectory of the human body in the real world based on the gravity view coordinate system, and obtain and output a three-dimensional motion sequence containing the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient.

[0010] Further, step S2 specifically includes: Step S21: Receive the three-dimensional motion sequence, establish the mapping relationship between the human joint set and the robot joint set, perform pre-alignment processing on the human joint rotation, and eliminate the systematic deviation caused by the difference between the default resting posture of the human body and the robot. Step S22: Adopt a local non-uniform scaling strategy, define scaling factors independently for different body parts of the human body, perform local scaling on key parts of the human body, and match the proportion of robot bone segments. Step S23: Construct a differential inverse kinematics constraint optimization problem and solve iteratively. In each iteration, calculate the position and rotation errors at the current joint angle. Use the robot Jacobian matrix to establish a linear approximate relationship between the joint angle change and the end position / rotation change. Calculate the joint angle increment that reduces the error and update the current joint angle. Perform constraint projection on the updated joint angle to obtain the robot joint angle configuration that satisfies the kinematic constraints. Step S24: Perform motion redirection processing on each frame of the motion sequence, using the optimized solution of the previous frame as the initial value for the current frame optimization solution. After all frames are redirected, perform ground consistency correction to eliminate overall floating or ground penetration phenomena, and obtain the initial robot motion sequence.

[0011] Furthermore, step S3 specifically includes: Step S31: Perform root horizontal plane drift correction on the corrected initial robot motion sequence. Decompose the robot root horizontal plane trajectory into low-frequency trend term and high-frequency motion term. Extract the low-frequency trend term of the robot root horizontal plane trajectory through sliding window averaging filter. Then correct the original robot root horizontal plane trajectory to remove low-frequency drift. Step S32: Perform root height segment repair on the robot motion sequence after root horizontal plane drift correction. Based on the robot's left and right foot horizontal plane movement speed and preset speed threshold, divide the robot motion sequence into ground contact phase and airborne phase. In the ground contact phase, calculate the lowest point height of the robot's body parts and adjust the root height by overall translation so that the lowest point contacts the ground. In the airborne phase, reconstruct the robot's root height trajectory into a parabola according to the law of gravity field motion so that the airborne trajectory satisfies the free fall motion formula. Step S33: Perform joint angle smoothing on the robot motion sequence after segmented repair of the root height. Use the Savitzky-Golay filtering method to smooth each joint angle sequence. Perform polynomial fitting in the local window through the Savitzky-Golay smoothing operator to achieve the data filtering effect. Step S34: Perform leg crossing detection and stability compensation on the robot motion sequence after joint angle smoothing. Calculate the horizontal distance between the robot's left and right feet. When the horizontal distance is less than the preset minimum safe distance threshold, it is determined that there is a risk of leg crossing. The leg crossing problem is corrected by adjusting the side swing angle of the left and right hip joints through a proportional coefficient. Step S35: After root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation, the optimized robot motion sequence is output. The optimized robot motion sequence satisfies physical consistency, motion smoothness and kinematic constraints.

[0012] Further, in step S14, the recovery of the human body's global motion trajectory in the real world based on the gravity view coordinate system is specifically expressed as follows: constructing a gravity view coordinate system with the Y-axis aligned with the world gravity direction, the Z-axis aligned with the camera view direction, and the X-axis orthogonally preserved through cross product; calculating the rotation transformation matrix from the camera coordinate system to the gravity view coordinate system based on the human body's posture relationship in different coordinate systems; transforming the human body's root node velocity vector in the camera coordinate system to the gravity view coordinate system and compensating for the influence of camera motion; accumulating the rotation changes in each frame to obtain the global orientation of the human body; calculating the human body's translation trajectory in the world coordinate system based on the velocity of the human body's root node in the local coordinate system to obtain the position of the human body's root node; and recovering the human body's continuous motion trajectory in the world coordinate system through velocity integration.

[0013] Furthermore, step S22 employs a local non-uniform scaling strategy, specifically as follows: Scaling factors are independently defined for key parts of the human body, and a local non-uniform scaling matrix is ​​constructed, which is expressed as: ; in, Indicates application to key parts of the human body The three-dimensional scaling matrix is ​​the locally non-uniform scaling matrix. , , These represent key parts of the human body. Scaling factors in the X, Y, and Z axes, and , , All are real numbers greater than zero; Based on the original positions of key human body parts and the position of the human body root node, the target space position of the key human body parts after scaling is calculated as follows: ; in, Key parts of the human body The target's 3D coordinates after scaling and coordinate system transformation. This is the global scaling factor, used to adjust the relative magnitude of the robot's motion to the original human motion. Key parts of the human body The original three-dimensional position in the original human body coordinate system This represents the three-dimensional position of the human root node in the original human coordinate system. The scaling transformation matrix for the root node; like The calculation formula is: .

[0014] Furthermore, the differential inverse kinematics constrained optimization problem in step S23 is expressed as follows:

[0015] in, For robot joint angles, For human joints With the target robot kinematic model The correspondence, This represents the mapping relationship between the set of human joints and the set of robot joints. These are the weighting coefficients for the rotation error term. Indicates the rotation of human joints. This indicates the rotation of the robot's joints. These are the weighting coefficients for the position error term. The target position of the end effector. The operator is used to calculate the end-effector position of the robot in the forward kinematics. This indicates the rotation difference operator.

[0016] Further, in step S3, the root horizontal plane drift is corrected on the modified initial robot motion sequence, expressed by the following formula: ; ; ; in, For the first The two-dimensional position coordinates of the human body in the horizontal plane. and For the coordinate data on the corresponding axis, For the first The low-frequency trend term of the trajectory is calculated by the frame through sliding window averaging filter. This is the length of the filtering window. Indicates to Perform floor function. For the first The two-dimensional position coordinates of the human body in the horizontal plane. For the first The two-dimensional trajectory position of the frame after drift correction. The trend value of the initial frame; The root height segmentation repair is performed on the robot motion sequence after root horizontal plane drift correction, expressed by the calculation formula as follows:

[0017] ; ; ; ; ; ; in, The average speed of movement on the horizontal plane of the foot. and For the left and right feet The coordinates of the frame on the horizontal plane. and For the left and right feet The coordinates of the frame on the horizontal plane. This represents the lowest point height of all body parts of the robot. For the robot's entire body parts, For the first The height of each body part This is the corrected root height. This represents the original root height before correction. The take-off height, The moment of takeoff The height of the root, For landing height, Landing time The height of the root, For the detected continuous clearing intervals, For flight time, It is the acceleration due to gravity. This is the height of the reconstructed root. This is an intermediate frame during the takeoff phase. This represents the time offset of the corresponding frame. The joint angles of the robot motion sequence after segmented repair of the root height are smoothed, and the calculation formula is as follows: ; in, After smoothing, the first The joint in the first The joint angle values ​​of the frame. This represents the Savitzky–Golay smoothing operator. For smoothing the first The joint in the first The angle of the frame; Leg crossover detection and stability compensation are performed on the robot motion sequence after joint angle smoothing, as expressed by the following formula: ; ; in, The horizontal distance between the robot's left and right feet. For the first Frame hip joint lateral tilt angle correction amount This is the proportionality coefficient. For the target foot spacing, For the left foot position Coordinate data, For the right foot position Coordinate data, It is a symbolic function.

[0018] Further, in step S4, the kinematic quantities are calculated using the optimized robot motion sequence. Specifically, based on the optimized robot motion sequence, the linear velocity of the robot root node in the world coordinate system is approximated by the central difference method. Then, the linear velocity is transformed from the world coordinate system to the robot root coordinate system using the root node rotation matrix to obtain the velocity of the root node in its own coordinate system. The joint angular velocity is calculated using the discrete difference method for the sequence of joint angles of the robot. The velocity of each body part of the robot in the world coordinate system is approximated by the central difference method. After coordinate system transformation, the relative motion velocity of each body part in its own coordinate system is obtained.

[0019] A monocular video-based humanoid robot full-body motion generation system is used to execute a monocular video-based humanoid robot full-body motion generation method, including: The video human pose recovery module is used to recover a three-dimensional human motion sequence with gravity view coordinate system constraints from monocular video. The motion redirection module is used to map a human three-dimensional motion sequence into an initial robot motion sequence that satisfies the robot's kinematic constraints. The robot motion post-processing and stability optimization module is used to restore the physical consistency and enhance the stability of the initial robot motion sequence; The Unified Data Format Export and Control Interface module is used to convert optimized robot motion sequences into a standardized data format and provides a control interface.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention effectively solves the common spatial drift problem in monocular vision human posture recovery. Considering that the human root trajectory recovered by traditional monocular vision algorithms often has continuous drift in the vertical direction, which is prone to suspension or ground penetration, this invention performs physical consistency correction on the human root position sequence through foot contact detection and root height constraint, so that the human body maintains a reasonable ground height relationship during the contact phase, thereby improving the physical rationality of the motion trajectory. 2. When the present invention detects an airborne phase in the action sequence, it reconstructs the motion model of the vertical trajectory at the root by parabolic reconstruction based on the law of gravity motion. This effectively avoids the common problems of air trajectory jitter or discontinuity in traditional vision reconstruction, thereby improving the stability and feasibility of the robot when performing high dynamic actions such as jumping. 3. This invention improves the mapping stability between the human skeletal structure and the robot kinematic structure by introducing a local non-uniform scaling strategy. By defining scaling factors independently for different parts of the human body, local scaling is performed on key parts of the human body, so that the human motion space structure can better match the robot structure, thereby reducing problems such as foot slippage, joint limit conflict and posture distortion. 4. This invention improves the stability of generated actions in simulation and real robot execution through robot motion post-processing and stability optimization mechanisms. By performing root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation on the redirected robot motion sequence, the generated robot actions can maintain the original human motion characteristics while satisfying robot kinematic and dynamic constraints, thus enabling more stable application in robot simulation verification and actual control systems. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0022] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0023] Figure 3This is a flowchart illustrating the robot motion post-processing and stability optimization module of the present invention.

[0024] Figure 4 This is a schematic diagram of motion redirection for the Nix2 humanoid robot of the present invention.

[0025] Figure 5 This is a schematic diagram of the uncorrected horizontal motion trajectory of the robot root according to the present invention.

[0026] Figure 6 This is a schematic diagram showing the verification results of the minimum safe distance between the two feet after the introduction of stability compensation in this invention.

[0027] Figure label annotations: Video human posture recovery module 10, motion redirection module 20, robot motion post-processing and stability optimization module 30, unified data format export and control interface module 40. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] Example 1 In this embodiment, as Figure 1 As shown, this invention provides a method for generating full-body motion of a humanoid robot based on monocular video, specifically including the following steps: Step S1: Preprocess the monocular video and extract multimodal features. By linear mapping and element-wise addition, the features of each modality are fused to form a unified temporal feature vector. Based on the gravity view coordinate system, the three-dimensional motion sequence of the human body in the world coordinate system is recovered. The three-dimensional motion sequence includes the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient. Step S2: Map the three-dimensional motion sequence to the kinematic model of the target robot, establish the joint mapping relationship between the human body and the target humanoid robot, perform posture pre-alignment of the human body joint rotation, use a local non-uniform scaling strategy to match the bone segment ratio of the human body and the robot, obtain the target three-dimensional position of the corresponding body part of the robot, and then take the consistency of the rotation of the corresponding joints of the human body and the robot and the spatial alignment of the end effector position as the optimization objectives, combine the physical limit constraints of the robot joint angle, construct the differential inverse kinematics constraint optimization problem and solve iteratively to obtain the initial robot motion sequence; Step S3: Perform systematic post-processing and stability optimization on the initial robot motion sequence, including root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation, and output the optimized robot motion sequence; Step S4: Calculate the kinematic quantities through the optimized robot motion sequence. The kinematic quantities include root velocity, joint velocity, and motion velocity of various parts of the body. Convert the kinematic quantities into a unified motion data matrix to generate standardized interface data for simulation verification, reinforcement learning training, and real machine control deployment.

[0030] Example 2 In this embodiment, as Figure 2 and Figure 3 As shown, the present invention also provides a humanoid robot full-body motion generation system based on monocular video, used to execute the humanoid robot full-body motion generation method based on monocular video shown in Embodiment 1 above, including: The video human pose recovery module 10 is used to recover the human three-dimensional motion sequence with gravity view coordinate system constraints from monocular video. As the entry module of the system from visual input to robot motion generation, this module realizes stable estimation of human pose, root node motion and overall trajectory through video preprocessing, multimodal feature extraction, feature fusion and global trajectory recovery. The specific implementation process corresponds to step S1. The motion redirection module 20 is used to map the human body's three-dimensional motion sequence into an initial robot motion sequence that satisfies the robot's kinematic constraints. Considering the differences between the human skeletal structure and the robot mechanism in terms of the number of joints, range of motion, and limb proportions, this module achieves a stable conversion from human actions to robot motion through steps such as joint mapping, pose alignment, bone segment proportion adjustment, and differential inverse kinematics optimization. Figure 4 As shown, motion redirection is performed using the Nix2 humanoid robot model as the target robot, and the specific implementation process corresponds to step S2; The robot motion post-processing and stability optimization module 30 is used to perform physical consistency repair and stability enhancement on the initial robot motion sequence, and to perform systematic post-processing and stability optimization on the redirected robot motion sequence. Through this module, problems such as trajectory drift, ground penetration, joint shaking and leg crossing can be effectively eliminated, thereby generating a stable motion sequence that is physically consistent, smooth in motion and meets the robot's kinematic constraints. The specific implementation process corresponds to step S3. The unified data format export and control interface module 40 is used to convert the optimized robot motion sequence into a standardized data format and provide a control interface between the simulation environment, reinforcement learning training framework and the actual robot, thereby realizing efficient transmission and reuse of motion data. Through this module, discrete kinematic information can be organized into a structured motion feature sequence, and the visualization verification and automated export functions of motion data are supported, providing a unified data foundation for subsequent robot control, motion imitation learning and simulation testing.

[0031] In this embodiment, step S1 is executed by the video human pose recovery module 10, specifically including: Step S11: Receive monocular video from a monocular camera in real-time or as input video files, forming a continuous video frame sequence. To accurately locate the human body region, a target detection algorithm is used to detect human bodies in each frame and generate corresponding human body bounding boxes. For example, real-time target detection methods such as YOLOv8 are used to identify human targets in the image and generate corresponding human body bounding boxes. ,in, Indicates the first The human body bounding box parameters in the frame image include, but are not limited to, the center coordinates, width, and height information of the bounding box. After obtaining the human body bounding box, the system performs uniform size processing on the cropped human body image. For example, the image is scaled to 224×224 pixels, and then the human body bounding box is normalized (such as adjusting brightness and contrast) and enhanced (such as random translation, scaling, and rotation) to obtain a preprocessed video frame sequence, so as to enhance the system's adaptability to different camera angles, lighting conditions, and scene background changes. Step S12: Extract multimodal features from the preprocessed video frame sequence. Multimodal features include, but are not limited to, two-dimensional keypoint features, image semantic features, camera motion features, and bounding box features. The two-dimensional keypoint feature extraction process is represented as follows: using pose estimation algorithms such as ViTPose, the two-dimensional coordinate information of key joints of the human body is extracted within the human body bounding box region, as shown in the figure. The first frame The two-dimensional representation of a joint is as follows ; The image semantic feature extraction process can be represented as follows: global semantic features of each frame of the image are extracted using the ViT encoder. Capture clothing, body shape, and background context information; The camera motion feature extraction process can be represented as follows: Based on visual odometry, estimate the relative camera rotation matrix between adjacent frames. This provides a reference for cross-frame motion alignment; The bounding box feature encoding process can be represented as follows: the position and size information of the bounding boxes detected in each frame are mapped into compact vectors using an MLP. It is used to assist in global pose recovery; Step S13: In order to make full use of video, key points and camera information, the present invention adopts an Early-Fusion fusion strategy, which uses an independent multilayer perceptron (MLP) to linearly map the features of each modality, transform them into a unified dimensional (512-dimensional) feature space, and uses element-wise addition to fuse the features of each modality into a unified temporal feature vector. The temporal feature vector is represented as: ,in, Indicates the first The fused temporal feature vector of the frame, + indicates element-wise addition, that is, the corresponding dimension of each vector is added together. Using addition fusion instead of splicing can avoid dimensionality explosion while preserving the information of each modality, and achieve a balance between computational efficiency and performance. The expression for transforming various features into a unified 512-dimensional feature space is as follows: ; ; ; ; in, Indicates the first The parameters of the 2D human bounding box in a frame image. This represents a multilayer perceptron network used to process bounding box features. Indicates the first The representation vector of frame bounding box features after mapping. Indicates the first The first frame A set of two-dimensional coordinates of key points Indexing key points of the human body This represents a multilayer perceptron network used for two-dimensional keypoint feature encoding. This represents the representation vector after feature mapping of two-dimensional key points. This represents the original image features extracted from the image coding network. This represents a multilayer perceptron network used for image semantic feature mapping. This represents the representation vector after image feature mapping. This represents the camera relative rotation matrix between adjacent video frames, used to describe dynamic camera motion information. This represents a multilayer perceptron network used for encoding camera motion features. Represents the representation vector after camera motion feature mapping; Step S14: Recover the global motion trajectory of the human body in the real world based on the gravity view coordinate system (GV coordinate system), and obtain and output a three-dimensional motion sequence containing the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient; To restore the global motion trajectory of the human body in the real world, this invention introduces a gravity view coordinate system into this module, using it as a unified reference system: Y-axis: Aligned with the direction of world gravity Determined by physical priors or IMU data; Z-axis: Aligns with camera view direction That is, the direction perpendicular to the image plane; X-axis: Calculated via cross product Preserve coordinate system orthogonality; For dynamic camera scenes, it is necessary to consider the motion changes of the camera between different time frames. Therefore, this invention compensates for and restores the global direction of human motion by estimating the relative rotation of the camera between adjacent frames. The specific process is as follows: Calculate the rotational transformation from the camera coordinate system to the GV coordinate system based on the human body's posture relationship in different coordinate systems: ,in, This represents the rotation of the human body in the GV coordinate system. This represents the rotation of the human body's root node in the camera coordinate system. This represents the rotation transformation matrix from the camera coordinate system to the GV coordinate system; After obtaining the rotation transformation matrix, the root node velocity vector estimated in the camera coordinate system is... Transform to GV coordinate system: ,in, This represents the estimated human root node velocity vector in the camera coordinate system. This represents the velocity vector of the human root node transformed into the GV coordinate system; To eliminate the influence of camera motion on velocity direction, a camera relative rotation matrix between adjacent frames is used. Speed ​​compensation: ,in, Indicates the first Frame and the The camera relative rotation matrix between frames; Project the vectors obtained above onto the ground plane (XZ plane) and calculate the angle between the view directions of adjacent frames. Construct a rotation matrix about the vertical axis (Y-axis) based on this included angle. , In the GV coordinate system, the first... Frame relative to the first Frame orientation change; The global orientation of the human body in the world coordinate system is obtained by accumulating the rotation changes across all frames: ,in, Indicates the first position of the human body in the world coordinate system. Global orientation of the frame This represents the rotation of the human body in the GV coordinate system. Indicates the first Incremental rotation of frame orientation Indicates from frame 1 to frame 2. The cumulative product of the frame rotation matrices is used to describe the overall orientation change of the human body in the time series, and satisfies... ; Through the above accumulation process, a global orientation sequence of the human body that changes over time can be obtained, thereby ensuring the continuity and stability of the direction of movement in the time dimension. After obtaining the global orientation, estimate the human body's translation trajectory in the world coordinate system based on the root node velocity: ,in, For the first The velocity of the frame root node in the local coordinate system Indicates the first position of the human body in the world coordinate system. The location of the root node of the frame. Indicates the first The global orientation of the frame is used to transform the local velocity to the world coordinate system. Through the above velocity integration process, the continuous motion trajectory of the human body in the world coordinate system is recovered. The output complete three-dimensional human motion sequence is represented as follows: ,in, Indicates the first The human motion state in the frame, The number of frames in the motion sequence represents the total number of frames. Each frame of human motion includes three-dimensional position, global orientation, rotational attitude parameters, and human shape coefficients. This represents the rotational attitude parameters of each joint of the human body in the local coordinate system. Represents the shape coefficients of the human body, in the three-dimensional motion sequence of the human body. This provides a complete 3D representation of human motion, which can be used in the subsequent motion redirection module 20 to map human movements to the robot's joint space, thereby enabling the generation of robot imitation learning data.

[0032] In this embodiment, step S2 is executed by the motion redirection module 20, specifically including: Step S21: Receive the three-dimensional motion sequence, establish the mapping relationship between the human joint set and the robot joint set, and perform pre-alignment processing on the human joint rotation to eliminate the systematic deviation caused by the difference between the default resting posture of the human and the robot. Receive the human body three-dimensional motion sequence output by the video human posture recovery module 10 ; To establish the correspondence between the human skeleton and the robot structure, a mapping relationship is established between the human joint set and the robot joint set: ,in, and They are assemblies of human and robot joints, This indicates the human joints With the target robot kinematic model The correspondence, This is a mapping relationship; Because human and robot models differ in their default resting pose definitions—for example, humans typically use a T-pose while robots usually start with a ZeroPose—pre-alignment processing of human joint rotations is necessary. This invention addresses this by applying a fixed rotation offset matrix to the local rotations of the human skeleton. This ensures that the orientation of key body parts in a resting posture is consistent with the orientation of corresponding parts of the robot in its zero-position posture, as shown below: ,in, This indicates the alignment and correction of human joints. The local rotation matrix, This represents the original local rotation matrix of human joint i. This step can eliminate the systematic bias caused by the difference in the default pose definition of different models, and provide a unified reference pose for subsequent bone segment scaling and motion optimization. Step S22: Adopt a local non-uniform scaling strategy, define scaling factors independently for different body parts of the human body, perform local scaling on key parts of the human body, and match the proportion of robot bone segments. Considering the differences in height and limb length ratios between the human body and the target robot, and to avoid motion artifacts such as foot slippage, overstretching, or compression of limbs caused by simple global scaling, this invention employs a local non-uniform scaling strategy. This involves independently defining scaling factors for different body parts (such as the thigh, calf, torso, upper arm, and forearm). To match the proportions of the robot's bone segments, this invention uses a local non-uniform scaling strategy, specifically expressed as follows: Define scaling factors independently for key parts of the human body, and construct a locally non-uniform scaling matrix, which is represented as follows: ; in, Indicates application to key parts of the human body The three-dimensional scaling matrix is ​​the locally non-uniform scaling matrix. , , These represent key parts of the human body. Scaling factors in the X, Y, and Z axes, and , , All are real numbers greater than zero; Based on key parts of the human body Given the original position and the root node position of the human body, calculate the target space position of the key parts of the human body after scaling, corresponding to the robot. The calculation formula is: ; in, Key parts of the human body The target's 3D coordinates after scaling and coordinate system transformation. This is an optional global scaling factor used to adjust the relative magnitude of the robot's motion to the original human motion. Key parts of the human body The original three-dimensional position in the original human body coordinate system This represents the three-dimensional position of the human root node (usually the pelvis or hip) in the original human coordinate system. The scale transformation matrix of the root node is used to adjust the proportion of the overall translation of the human body, thereby ensuring the consistency between the overall motion trajectory of the robot and the motion of the human body in terms of spatial scale. like The calculation formula is: Ensure that the position scaling logic of the root node (usually corresponding to the robot's pelvis or torso base) is consistent with that of the limb parts; Step S23: Construct a differential inverse kinematics constraint optimization problem and solve iteratively. In each iteration, calculate the position and rotation errors at the current joint angle. Use the robot Jacobian matrix to establish a linear approximate relationship between the joint angle change and the end position / rotation change. Calculate the joint angle increment that reduces the error and update the current joint angle. Perform constraint projection on the updated joint angle to obtain the robot joint angle configuration that satisfies the kinematic constraints. After completing pose alignment and scale transformation, the module solves for the robot joint configuration that satisfies joint constraint. The problem is to construct a differential inverse kinematics constrained optimization problem. The optimization process involves the following constraints: firstly, the rotational consistency of corresponding joints between the human and robot, which is determined by a mapping relationship between the two. The corresponding joints should rotate as uniformly as possible; secondly, the spatial alignment of the end effector positions (such as hands and feet) should be ensured, and the target position of the end effector should be determined. The positive kinematic position should correspond to the corresponding part of the robot. Align as much as possible, and thirdly, the robot joint angles. It does not violate physical limit constraints; Considering the above constraints, the differential inverse kinematics constraint optimization problem in step S23 can be expressed as follows:

[0033] in, For robot joint angles, For human joints With the target robot kinematic model The correspondence, This represents the mapping relationship between the set of human joints and the set of robot joints. These are the weighting coefficients for the rotation error term. Indicates the rotation of human joints. This indicates the rotation of the robot's joints. These are the weighting coefficients for the position error term. The target position of the end effector. The operator is used to calculate the end-effector position of the robot in the forward kinematics. This represents the rotation difference operator, which allows different weights to be set for different joints during the optimization process to ensure the pose consistency of key joints (such as the pelvis and spine). The optimization process employs iterative solutions using differential inverse kinematics. In each iteration, the current joint angle is calculated. To address the errors in position and rotation, a linear approximation relationship is established between the changes in joint angles and the changes in end-effector position / rotation using the robot's Jacobian matrix. This allows for the calculation of the joint angle increments that reduce the errors. and update The process is represented as: In actual implementation, the range of values ​​for each joint angle is usually predetermined by the joint parameters in the robot's structural model. For example, in the robot dynamics model file, the allowable angle range can be defined through the joint's range parameter. When the joint angles are updated, the updated joint angles are... Perform a constraint projection to satisfy: , This is the minimum joint angle. This represents the maximum joint angle. In practical applications, the range of joint angles can be adjusted according to different robot structure models. For example, in one specific implementation, the angle range of some robot joints, such as the pitch degree of freedom of the hip joint, can be set to... , For example, the ankle joint's flexion-extension degree of freedom can be set to... , ; Step S24: Perform motion redirection processing on each frame of the motion sequence, using the optimized solution of the previous frame as the initial value for the current frame optimization solution. After all frames are redirected, perform ground consistency correction to eliminate overall floating or ground penetration phenomena, and obtain the initial robot motion sequence. Considering that the above steps mainly target motion retargeting for a single frame, this module adopts a frame-by-frame processing strategy for the complete motion sequence. (in (total number of frames), and the previous frame Optimization solution As the current frame Optimize the initial values ​​for the solution to accelerate convergence and ensure the quality of the generated robot joint angle sequence. It is smooth and continuous in time, avoiding inter-frame jitter; For the entire complete motion sequence, this module applies the above single-frame attitude motion retargeting process frame by frame, using the solution from the previous frame as the initial value for optimization in the current frame to ensure temporal continuity. After completing the retargeting of all frames, the module performs ground consistency correction, which is represented as follows: Using robot forward kinematics, calculate the sequence of minimum heights for all robot body parts (especially the feet) throughout the entire motion sequence. The calculation ensures that the robot's lowest point just touches the ground. The root node position of the entire motion sequence is uniformly corrected: , This step, used to correct the values, eliminates potential overall floating or ground penetration in the motion sequence, generating a physically reliable reference motion sequence that can be directly used for robot control. .

[0034] In this embodiment, step S3 is executed by the robot motion post-processing and stability optimization module 30, specifically including: Step S31: Perform root horizontal plane drift correction on the corrected initial robot motion sequence. Decompose the robot root horizontal plane trajectory into low-frequency trend term and high-frequency motion term. Extract the low-frequency trend term of the robot root horizontal plane trajectory through sliding window averaging filter. Then correct the original robot root horizontal plane trajectory to remove low-frequency drift. In the process of human motion recovery in monocular vision, due to the influence of depth estimation error, scale drift and complex environmental factors, the reconstructed root motion trajectory often exhibits low-frequency cumulative drift on the horizontal plane (XY plane), which manifests as the robot as a whole slowly sliding in the horizontal direction, thus affecting the realism of the motion. Considering that the robot's root displacement exhibits an abnormal range expansion (such as irregular drift of 2m×3m) in the original reconstructed trajectory, which manifests as the robot as a whole producing slow and unexpected "gliding" or "translation" in the horizontal direction, seriously violating the support phase constraints and motion authenticity of legged robots, this invention adopts a trajectory correction method based on low-frequency trend decomposition to decompose the root horizontal trajectory into low-frequency trend terms and high-frequency motion terms, thereby eliminating unrealistic drift. The above method can effectively preserve the high-frequency displacement changes generated by the real human body movement and remove the low-frequency drift caused by visual estimation errors. The root horizontal plane drift is corrected for the modified initial robot motion sequence, specifically as follows: A trajectory correction method based on low-frequency trend decomposition is adopted: ; The low-frequency trend term of the trajectory is obtained by calculating the average filter using a sliding window: ; Correct the original trajectory: ; in, For the first The two-dimensional position coordinates of the human body in the horizontal plane. and For the coordinate data on the corresponding axis, For the first The low-frequency trend term of the trajectory, calculated by the frame through sliding window averaging filtering, is used to describe the overall slow change trend of the trajectory over a period of time. The filter window length corresponds to a motion sequence within a certain time range. Indicates to Perform a floor operation to determine the current frame. The symmetrical window range centered on the center For the first The two-dimensional position coordinates of the human body in the horizontal plane. For the first The two-dimensional trajectory position of the frame after drift correction. The trend value of the initial frame is used to maintain the overall continuity of the trajectory; Step S32: Perform root height segment repair on the robot motion sequence after root horizontal plane drift correction. Based on the robot's left and right foot horizontal plane movement speed and preset speed threshold, divide the robot motion sequence into ground contact phase and airborne phase. In the ground contact phase, calculate the lowest point height of the robot's body parts and adjust the root height by overall translation so that the lowest point contacts the ground. In the airborne phase, reconstruct the robot's root height trajectory into a parabola according to the law of gravity field motion so that the airborne trajectory satisfies the free fall motion formula. Considering that the root node height trajectory (Z-axis) in the visual reconstruction process usually has two types of defects that violate physical consistency, one is the failure of support phase constraints. In the steady-state stage when the human foot is in contact with the ground (ground contact phase), due to the disturbance of depth estimation and scale uncertainty, there is an unreal spatial suspension or physical penetration phenomenon between the robot's foot and the virtual ground. The other is the distortion of dynamics in the air phase. In the jumping or high-dynamic action stage (air phase), the reconstructed trajectory often cannot accurately restore the evolution logic of the center of gravity under the action of the gravitational field, resulting in the motion trajectory not conforming to the laws of classical mechanics (such as the laws of free fall or oblique projectile motion). Therefore, this invention adopts a segmented repair strategy. This method can restore the air motion trajectory that conforms to the laws of physics, making the jumping action more realistic and stable. This strategy divides the trajectory into the ground contact phase and the air phase according to the motion state. The ground contact phase focuses on applying zero-velocity constraints of the foot and ground height compensation to eliminate suspension and ground penetration interference. The air phase focuses on applying gravitational field constraints and physical trajectory reconstruction. The root height segmentation repair is performed on the robot motion sequence after root horizontal plane drift correction, specifically as follows: The frame is determined to be in the ground-touching phase based on the foot's movement speed on the horizontal plane. Assume the coordinates of the left and right feet on the horizontal plane are... and Set speed threshold The average speed of the foot can be expressed as:

[0035] To ensure the robot's lowest point is in contact with the ground and to prevent its feet from hovering or penetrating the ground, the lowest point height of all body parts of the robot is calculated for each ground contact frame: ; Adjust the root height by overall translation: ; For the detected continuous clearing intervals Based on the laws of motion of an object in a gravitational field, the trajectory of the root height is reconstructed as a parabola. , , ; For intermediate frames during the takeoff phase The height can be reconstructed as follows: ; in, The average speed of foot movement on the horizontal plane, when If the ground contact occurs, it is considered the ground contact phase; otherwise, it is considered the airborne phase. and For the left and right feet The coordinates of the frame on the horizontal plane. and For the left and right feet The coordinates of the frame on the horizontal plane. This represents the lowest point height of all body parts of the robot. For the robot's entire body parts, For the first The height of each body part This is the corrected root height. This represents the original root height before correction. The take-off height, The moment of takeoff The height of the root, For landing height, Landing time The height of the root, For the detected continuous clearing intervals, For flight time, It is the acceleration due to gravity. This is the height of the reconstructed root. This is an intermediate frame during the takeoff phase. This represents the time offset of the corresponding frame. Step S33: Perform joint angle smoothing on the robot motion sequence after segmented repair of the root height. Use the Savitzky-Golay filtering method to smooth each joint angle sequence. Perform polynomial fitting in the local window through the Savitzky-Golay smoothing operator to achieve the data filtering effect. Considering that during motion retargeting, the robot joint angle sequence may exhibit high-frequency jitter due to frame-by-frame calculation and numerical optimization errors, thereby affecting control stability, this invention employs the Savitzky-Golay filtering method to smooth each joint angle sequence. This method uses local polynomial fitting to remove high-frequency noise while preserving the peak features of the motion, making the joint trajectory more continuous and smooth in time. The robot motion sequence after segmented repair of the root height is subjected to joint angle smoothing, specifically as follows: ,in, After smoothing, the first The joint in the first The joint angle values ​​of the frame. This represents the Savitzky-Golay smoothing operator, used for smoothing time series data. This operator performs polynomial fitting within a local window. For smoothing the first The joint in the first The angle of the frame; Step S34: Perform leg crossing detection and stability compensation on the robot motion sequence after joint angle smoothing. Calculate the horizontal distance between the robot's left and right feet. When the horizontal distance is less than the preset minimum safe distance threshold, it is determined that there is a risk of leg crossing. The leg crossing problem is corrected by adjusting the side swing angle of the left and right hip joints through a proportional coefficient. Considering that the left and right legs may cross or get too close in space during motion redirection, which may lead to a decrease in robot motion stability, this module detects the distance between the feet and compensates and corrects the hip joint posture. This compensation mechanism can effectively avoid leg crossing and improve gait stability. All joint corrections are performed within the robot joint limit range to ensure that the generated motion meets the robot hardware constraints. Leg crossover detection and stability compensation are performed on the robot motion sequence after joint angle smoothing, specifically as follows: Calculate the distance between the left and right feet on the horizontal plane: ; When a risk of leg crossing is detected, compensation is made by adjusting the lateral roll angle of the left and right hip joints: ; in, Let the horizontal distance between the robot's left and right feet be , if the following conditions are met. If so, it is considered that there is a risk of leg crossing. The minimum safe distance threshold, For the first Frame hip joint lateral tilt angle correction amount This is the proportionality coefficient. For the target foot spacing, For the left foot position Coordinate data, For the right foot position Coordinate data, It is a symbolic function; The corresponding hip joint correction is as follows: , , and The figures show the left hip joint lateral displacement angle before and after correction. and The right hip joint lateral displacement angles before and after correction are shown respectively. Step S35: After root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg crossing detection and stability compensation, the optimized robot motion sequence is output. The optimized robot motion sequence satisfies physical consistency, motion smoothness and kinematic constraints. During the ground contact phase, the feet stably contact the ground. During the airborne phase, it conforms to the law of gravity motion. The joint trajectories are continuous, without obvious high-frequency jitter, and there is no leg crossing phenomenon. The joint angles are all within the allowable range of the robot. Therefore, the optimized motion sequence can be directly used for robot simulation verification, imitation learning training and control system input.

[0036] In this embodiment, step S4 is executed through the unified data format export and control interface module 40. After obtaining the robot's pose data for each frame, it is necessary to further calculate complete kinematic information, including root velocity, joint velocity, and the motion velocity of various parts of the body, to form a complete dynamic feature description. In step S4, the kinematic quantities are calculated through the optimized robot motion sequence, specifically as follows: Based on the optimized robot motion sequence, the linear velocity of the robot root node in the world coordinate system is approximated using the central difference method. Then, the linear velocity is transformed from the world coordinate system to the robot root coordinate system using the root node rotation matrix, yielding the velocity of the root node in its own coordinate system. Let the position of the robot root node in the world coordinate system be denoted as . The linear velocity of the root node in the world coordinate system is then approximated by the central difference as follows: To obtain the motion velocity with reference to the robot itself, the velocity is transformed from the world coordinate system to the robot's root coordinate system. Let the root node rotation matrix be... Then the velocity of the root in its own coordinate system is: ; The discrete difference method is used to calculate the joint angular velocity of the robot's joints for each joint angle sequence. There are 1 joint, and its angle sequence is as follows: Joint angular velocities can be calculated using the discrete difference method: , Joint angular velocity; The velocities of each body part of the robot in the world coordinate system are approximated using the central difference method. After coordinate transformation, the relative velocities of each body part in its own coordinate system are obtained. Let the position of a certain body part of the robot in the world coordinate system be... Then, its velocity in the world coordinate system can be approximated using the central difference method, and we can obtain: To obtain the motion velocity relative to the robot body, it is necessary to eliminate the influence of the overall motion of the root node and perform a coordinate system transformation: Through the above processing, the relative motion velocity of each body part in the robot's own coordinate system can be obtained, thereby more accurately describing the robot's motion state. After obtaining the complete kinematic data, the root node position, root node pose, linear and angular velocities of the root node in the world and its own coordinate systems, joint angles and angular velocities, and the positions and velocities of body parts in the world and its own coordinate systems are concatenated and arranged in a frame sequence along the time dimension to form a unified motion data matrix, represented as follows: , For motion data matrix, The root node position is... For the root node pose, Let be the linear velocity of the root node in the world coordinate system. Let be the angular velocity of the root node in the world coordinate system. Let be the linear velocity of the root node in its own coordinate system. Let be the angular velocity of the root node in its own coordinate system. For the angles of each joint, For the angular velocities of each joint, This represents the position of various body parts in the world coordinate system. Let be the velocity of each part of the body in the world coordinate system. This represents the positions of various body parts in the robot's coordinate system. The velocity of each part of the body in the robot's coordinate system.

[0037] To further demonstrate the practical effects of the humanoid robot full-body motion generation method and system based on monocular video in this invention, a visual schematic diagram of the humanoid robot full-body motion generation technology solution based on monocular video is provided, such as... Figures 4-6 As shown; Figure 4 This is a schematic diagram of motion redirection for the Nix2 humanoid robot, showing the mapping relationship from human motion to robot motion. It is a visualization of step S2. The human model on the left is labeled with complete human skeleton nodes, representing the human 3D motion sequence recovered from monocular video. Each node corresponds to the key joint position and posture of the human body. The Nix2 robot model on the right is labeled with the robot's links and joint names. The joints correspond one-to-one with the human nodes, establishing a joint mapping relationship between the human body and the robot. Figure 5This is a schematic diagram of the uncorrected horizontal motion trajectory of the robot's root, and a visualization of the horizontal drift correction of the root in step S3. The X-axis represents the position of the root node in the X direction of the world coordinate system, and the Y-axis represents the position of the root node in the Y direction of the world coordinate system. The curve represents the XY plane motion trajectory of the root node over time. According to this figure, it can be seen that in the original reconstructed trajectory, the displacement of the robot's root exhibits an abnormal range expansion (such as an irregular drift of 2m×3m), which manifests as the robot as a whole producing slow and unexpected "gliding" or "translation" in the horizontal direction, which seriously violates the support phase constraints and motion realism of the legged robot. Figure 6 The diagram illustrates the verification results of the minimum safe distance between the two feet after introducing stability compensation. It shows the comparison of the changes in the minimum safe distance between the two feet before and after introducing this compensation mechanism under complex motion sequences. The horizontal axis represents the frame number of the motion sequence, and the vertical axis represents the three-dimensional spatial distance (m) between the two feet. The dashed line in the figure represents the set minimum safe distance threshold of 0.15m. According to the figure, it can be seen that after introducing the hip joint roll compensation mechanism, the distance between the two feet is generally kept above the minimum safe distance threshold, which can effectively avoid the problem of foot collision that may occur during robot movement.

[0038] In summary, this invention proposes a method and system for generating full-body motion of humanoid robots based on monocular video. It can automatically generate physically consistent, stable, and kinematically consistent full-body motion sequences from ordinary monocular video, thereby improving the quality of robot motion generation and reducing the cost of motion data acquisition. It has good application value in fields such as humanoid robot imitation learning, robot motion data generation, and human-computer interaction.

[0039] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating full-body motion of a humanoid robot based on monocular video, characterized in that, Specifically, the following steps are included: Step S1: Preprocess the monocular video and extract multimodal features. By linear mapping and element-wise addition, the features of each modality are fused to form a unified temporal feature vector. Based on the gravity view coordinate system, the three-dimensional motion sequence of the human body in the world coordinate system is recovered. The three-dimensional motion sequence includes the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient. Step S2: Map the three-dimensional motion sequence to the kinematic model of the target robot, establish the joint mapping relationship between the human body and the target humanoid robot, perform posture pre-alignment of the human body joint rotation, use a local non-uniform scaling strategy to match the bone segment ratio of the human body and the robot, obtain the target three-dimensional position of the corresponding body part of the robot, and then take the consistency of the rotation of the corresponding joints of the human body and the robot and the spatial alignment of the end effector position as the optimization objectives, combine the physical limit constraints of the robot joint angle, construct the differential inverse kinematics constraint optimization problem and solve iteratively to obtain the initial robot motion sequence; Step S3: Perform systematic post-processing and stability optimization on the initial robot motion sequence, including root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation, and output the optimized robot motion sequence; Step S4: Calculate the kinematic quantities through the optimized robot motion sequence. The kinematic quantities include root velocity, joint velocity, and motion velocity of various parts of the body. Convert the kinematic quantities into a unified motion data matrix to generate standardized interface data for simulation verification, reinforcement learning training, and real machine control deployment. Specifically, step S2 includes: Step S21: Receive the three-dimensional motion sequence, establish the mapping relationship between the human joint set and the robot joint set, perform pre-alignment processing on the human joint rotation, and eliminate the systematic deviation caused by the difference between the default resting posture of the human body and the robot. Step S22: Adopt a local non-uniform scaling strategy, define scaling factors independently for different body parts of the human body, perform local scaling on key parts of the human body, and match the proportion of robot bone segments. Step S23: Construct a differential inverse kinematics constraint optimization problem and solve iteratively. In each iteration, calculate the position and rotation errors at the current joint angle. Use the robot Jacobian matrix to establish a linear approximate relationship between the joint angle change and the end position / rotation change. Calculate the joint angle increment that reduces the error and update the current joint angle. Perform constraint projection on the updated joint angle to obtain the robot joint angle configuration that satisfies the kinematic constraints. Step S24: Perform motion redirection processing on each frame of the motion sequence, using the optimized solution of the previous frame as the initial value for the current frame optimization solution. After all frames are redirected, perform ground consistency correction to eliminate overall floating or ground penetration phenomena, and obtain the initial robot motion sequence. Step S3 specifically includes: Step S31: Perform root horizontal plane drift correction on the corrected initial robot motion sequence. Decompose the robot root horizontal plane trajectory into low-frequency trend term and high-frequency motion term. Extract the low-frequency trend term of the robot root horizontal plane trajectory through sliding window averaging filter. Then correct the original robot root horizontal plane trajectory to remove low-frequency drift. Step S32: Perform root height segment repair on the robot motion sequence after root horizontal plane drift correction. Based on the robot's left and right foot horizontal plane movement speed and preset speed threshold, divide the robot motion sequence into ground contact phase and airborne phase. In the ground contact phase, calculate the lowest point height of the robot's body parts and adjust the root height by overall translation so that the lowest point contacts the ground. In the airborne phase, reconstruct the robot's root height trajectory into a parabola according to the law of gravity field motion so that the airborne trajectory satisfies the free fall motion formula. Step S33: Perform joint angle smoothing on the robot motion sequence after segmented repair of the root height. Use the Savitzky-Golay filtering method to smooth each joint angle sequence. Perform polynomial fitting in the local window through the Savitzky-Golay smoothing operator to achieve the data filtering effect. Step S34: Perform leg crossing detection and stability compensation on the robot motion sequence after joint angle smoothing. Calculate the horizontal distance between the robot's left and right feet. When the horizontal distance is less than the preset minimum safe distance threshold, it is determined that there is a risk of leg crossing. The leg crossing problem is corrected by adjusting the side swing angle of the left and right hip joints through a proportional coefficient. Step S35: After root horizontal plane drift correction, root height segment repair, joint angle smoothing, leg cross detection and stability compensation, the optimized robot motion sequence is output. The optimized robot motion sequence satisfies physical consistency, motion smoothness and kinematic constraints.

2. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Receive monocular video from a monocular camera in real time or input video file to form a continuous video frame sequence. Use a target detection algorithm to perform human detection on each frame and generate the corresponding human bounding box. Normalize and enhance the human bounding box to obtain the preprocessed video frame sequence. Step S12: Extract multimodal features from the preprocessed video frame sequence. The multimodal features include two-dimensional keypoint features, image semantic features, camera motion features, and bounding box features. Step S13: Linearly map each modal feature using an independent multilayer perceptron to transform it into a unified dimensional feature space, and fuse each modal feature into a unified temporal feature vector by adding elements one by one. Step S14: Recover the global motion trajectory of the human body in the real world based on the gravity view coordinate system, and obtain and output a three-dimensional motion sequence containing the root node position of the human body, global orientation, rotational posture parameters and human body shape coefficient.

3. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 2, characterized in that, Step S14, which involves recovering the global motion trajectory of the human body in the real world based on the gravity view coordinate system, specifically involves: constructing a gravity view coordinate system with the Y-axis aligned with the world gravity direction, the Z-axis aligned with the camera view direction, and the X-axis orthogonally preserved through cross product; calculating the rotation transformation matrix from the camera coordinate system to the gravity view coordinate system based on the posture relationship of the human body in different coordinate systems; transforming the velocity vector of the human body root node in the camera coordinate system to the gravity view coordinate system and compensating for the influence of camera motion; accumulating the rotation changes of each frame to obtain the global orientation of the human body; calculating the translation trajectory of the human body in the world coordinate system based on the velocity of the human body root node in the local coordinate system to obtain the position of the human body root node; and recovering the continuous motion trajectory of the human body in the world coordinate system through velocity integration.

4. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 3, characterized in that, The local non-uniform scaling strategy adopted in step S22 is specifically expressed as follows: Scaling factors are independently defined for key parts of the human body, and a local non-uniform scaling matrix is ​​constructed, which is expressed as: ; in, Indicates application to key parts of the human body The three-dimensional scaling matrix is ​​the locally non-uniform scaling matrix. , , These represent key parts of the human body. Scaling factors in the X, Y, and Z axes, and , , All are real numbers greater than zero; Based on the original positions of key human body parts and the position of the human body root node, the target space position of the key human body parts after scaling is calculated as follows: ； in, Key parts of the human body The target's 3D coordinates after scaling and coordinate system transformation. This is the global scaling factor, used to adjust the relative magnitude of the robot's motion to the original human motion. Key parts of the human body The original three-dimensional position in the original human body coordinate system This represents the three-dimensional position of the human root node in the original human coordinate system. The scaling transformation matrix for the root node; like The calculation formula is: .

5. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 3, characterized in that, The differential inverse kinematics constraint optimization problem in step S23 is expressed as follows: in, For robot joint angles, For human joints With the target robot kinematic model The correspondence, This represents the mapping relationship between the set of human joints and the set of robot joints. These are the weighting coefficients for the rotation error term. Indicates the rotation of human joints. This indicates the rotation of the robot's joints. These are the weighting coefficients for the position error term. The target position of the end effector. The operator is used to calculate the end-effector position of the robot in the forward kinematics. This indicates the rotation difference operator.

6. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 4, characterized in that, In step S3, the root horizontal plane drift is corrected on the modified initial robot motion sequence, which is expressed by the following formula: ； ； ； in, For the first The two-dimensional position coordinates of the human body in the horizontal plane. and For the coordinate data on the corresponding axis, For the first The low-frequency trend term of the trajectory is calculated by the frame through sliding window averaging filter. This is the length of the filtering window. Indicates to Perform floor function. For the first The two-dimensional position coordinates of the human body in the horizontal plane. For the first The two-dimensional trajectory position of the frame after drift correction. The trend value of the initial frame; The root height segmentation repair is performed on the robot motion sequence after root horizontal plane drift correction, expressed by the calculation formula as follows: ； ； ； ； ； ； in, The average speed of movement on the horizontal plane of the foot. and For the left and right feet The coordinates of the frame on the horizontal plane. and For the left and right feet The coordinates of the frame on the horizontal plane. This represents the lowest point height of all body parts of the robot. For the robot's entire body parts, For the first The height of each body part This is the corrected root height. This represents the original root height before correction. The take-off height, The moment of takeoff The height of the root, For landing height, Landing time The height of the root, For the detected continuous clearing intervals, For flight time, It is the acceleration due to gravity. This is the height of the reconstructed root. This is an intermediate frame during the takeoff phase. This represents the time offset of the corresponding frame. The joint angles of the robot motion sequence after segmented repair of the root height are smoothed, and the calculation formula is as follows: ; in, After smoothing, the first The joint in the first The joint angle values ​​of the frame. This represents the Savitzky–Golay smoothing operator. For smoothing the first The joint in the first The angle of the frame; Leg crossover detection and stability compensation are performed on the robot motion sequence after joint angle smoothing, as expressed by the following formula: ； ； in, The horizontal distance between the robot's left and right feet. For the first Frame hip joint lateral tilt angle correction amount This is the proportionality coefficient. For the target foot spacing, For the left foot position Coordinate data, For the right foot position Coordinate data, It is a symbolic function.

7. The method for generating full-body motion of a humanoid robot based on monocular video according to claim 1, characterized in that, In step S4, kinematic quantities are calculated using the optimized robot motion sequence. Specifically, based on the optimized robot motion sequence, the linear velocity of the robot root node in the world coordinate system is approximated by the central difference method. Then, the linear velocity is transformed from the world coordinate system to the robot root coordinate system using the root node rotation matrix to obtain the velocity of the root node in its own coordinate system. The joint angular velocity is calculated using the discrete difference method for the sequence of robot joint angles. The velocity of each body part of the robot in the world coordinate system is approximated by the central difference method. After coordinate system transformation, the relative motion velocity of each body part in its own coordinate system is obtained.

8. A humanoid robot full-body motion generation system based on monocular video, used to execute the humanoid robot full-body motion generation method based on monocular video according to any one of claims 1-7, characterized in that, include: The video human pose recovery module is used to recover a three-dimensional human motion sequence with gravity view coordinate system constraints from monocular video. The motion redirection module is used to map a human three-dimensional motion sequence into an initial robot motion sequence that satisfies the robot's kinematic constraints. The robot motion post-processing and stability optimization module is used to restore the physical consistency and enhance the stability of the initial robot motion sequence; The Unified Data Format Export and Control Interface module is used to convert optimized robot motion sequences into a standardized data format and provides a control interface.

Images