Humanoid robot motion strategy training method and system

Abstract

The application discloses a kind of humanoid robot motion strategy training method and system, it is related to robot motion strategy training technical field, solve the technical problems that humanoid robot action migration is difficult and non-task related body part movement constraint is insufficient.The method includes obtaining the motion video of reference motion subject executing target task of human form structure, and determining the local visible body part trajectory of reference motion subject according to motion video;Local visible body part trajectory is mapped to target body part set, and the motion trajectory corresponding to the execution target task of humanoid robot is determined;According to motion trajectory, basic action priori library, generate the corresponding auxiliary posture constraint of body part not covered by motion trajectory;According to motion trajectory and auxiliary posture constraint, the motion strategy of humanoid robot is trained.The application can improve the migration ability of humanoid robot to demonstration action, realize the effective training of motion strategy under complex task.

Description

Technical Field

[0001] This invention relates to the field of robot motion strategy training technology, and in particular to a method and system for training humanoid robot motion strategies. Background Technology

[0002] Humanoid locomotion systems, with their near-human body structure, can adapt to human-designed workspaces, tool systems, and objects, making them an important development direction in service industries, industrial manufacturing, warehousing and logistics, home assistance, rehabilitation and nursing, and specialized operations. In recent years, humanoid locomotion systems have gradually moved from laboratory research to prototype verification, scenario testing, and small-scale demonstration applications. However, their stable walking, whole-body coordination control, dexterity, and task generalization capabilities in complex real-world environments still need improvement. Therefore, it is necessary to use demonstration data of human or other humanoid locomotion systems performing tasks to train stable, reliable, and generalizable locomotion strategies for humanoid locomotion systems, enabling them to gradually complete complex tasks such as grasping, carrying, walking, assembling, opening and closing doors, operating tools, and organizing objects.

[0003] In existing research and technical solutions, various technical approaches can be adopted to train the motion strategies of humanoid structures using demonstration data. For example, some solutions acquire human joint trajectories based on motion capture data or human motion datasets and redirect them to the target humanoid structure's motion subject; others estimate human posture, human mesh, SMPL humanoid structure motion subject parameters, hand posture, task object, and scene environment based on monocular video, RGB-D humanoid structure motion subject video, or multi-view video, and further generate robot joint trajectories, end effector traject ... When the video capture range is limited, the human body is partially occluded, or the key actions of the task are mainly concentrated in local body parts such as the hands, arms, and upper body, the body motion information that can be used for direct reconstruction and redirection will become incomplete, thus affecting the reliability of complete body trajectory recovery, posture generation of unobserved body parts, and overall coordinated control of the robot.

[0004] In real-world task scenarios, cameras cannot always capture the entire moving human or humanoid subject. For example, in tasks such as tabletop grasping, parts assembly, bottle cap unscrewing, tool operation, button pressing, door opening, and wiping, cameras often primarily capture the upper body, arms, wrists, hands, or fingers directly related to the task, while the lower body, pelvis, feet, and even the upper part of the arms may be obstructed by the tabletop, limited field of view, shooting angle, or environmental objects. In such cases, requiring the recovery of the complete human body trajectory from the video not only increases the difficulty of pose estimation but may also introduce unreliable body trajectory information, thereby affecting the stability of subsequent robot repositioning and policy training. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for training motion strategies for humanoid robots, addressing the problems of difficulty in selecting reliable trajectories, difficulty in redirecting visible trajectories, and lack of executable constraints for unobserved body parts when demonstration videos are only partially visible or when there are structural differences between the reference moving subject and the target humanoid robot. The numerous technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0006] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for training a humanoid robot's motion strategy, comprising: Acquire motion video of a humanoid reference moving subject performing a target task, and determine the trajectory of locally visible body parts of the reference moving subject based on the motion video; Based on the target task and the structural information of the humanoid robot, a set of target body parts is determined, and the trajectory of the locally visible body parts is mapped to the set of target body parts to determine the motion trajectory of the humanoid robot performing the target task. A basic action prior library corresponding to the humanoid robot is obtained. The basic action prior library is obtained by controlling the humanoid robot to perform basic action tasks and performing strategy training. Based on the motion trajectory, the target task, the state of the humanoid robot, and the basic action prior library, auxiliary posture constraints are generated for body parts not covered by the motion trajectory. The motion strategy of the humanoid robot is trained based on the motion trajectory and the auxiliary posture constraints.

[0007] In a second aspect, the present invention also provides a humanoid motion subject motion strategy training system, comprising: a processor and a memory; The memory stores a computer program, which, when executed by the processor, implements the humanoid robot motion strategy training method as described in the first aspect.

[0008] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects: 1. The present invention acquires motion video of a humanoid reference motion subject performing a target task, and determines the trajectory of locally visible body parts of the reference motion subject based on the motion video. It can extract motion information related to the target task from the demonstration movements of the reference motion subject, avoiding direct reliance on complete demonstration movement data. 2. The present invention determines the target body part set based on the target task and the structural information of the humanoid robot, and maps the trajectory of the locally visible body part to the target body part set. It can adapt and convert the demonstration motion information according to the structural differences between different moving subjects, so as to address the problem that the demonstration action is difficult to transfer directly due to the different structures of the moving subject and the humanoid robot. 3. This invention obtains a priori library of basic movements corresponding to a humanoid robot, and generates auxiliary posture constraints for body parts not covered by the movement trajectory based on the movement trajectory, target task, humanoid robot state, and the priori library of basic movements. It utilizes the humanoid robot's own basic movement capabilities to supplement the movement information not covered by the demonstration trajectory, thus avoiding the problem of insufficient movement integrity caused by the humanoid robot only learning local demonstration movements. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a schematic diagram of a humanoid robot motion strategy training method according to an embodiment of the present invention; Figure 2 This is an overall flowchart of an embodiment of the present invention; Figure 3 This is a schematic diagram of the basic action pre-training stage and the basic action prior library construction process in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of locally visible motion video framing, joint sensing, and body part visibility mask generation according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the process for extracting locally visible body trajectories, hand trajectories, or tool end-effector trajectories based on body part visibility masks according to an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the redirection process from a partially visible trajectory to a humanoid robot according to an embodiment of the present invention. Figure 7 This is a schematic diagram of the process for generating auxiliary posture constraints for unshot body parts based on a basic action prior library, according to an embodiment of the present invention. Figure 8 This is a flowchart illustrating the process of training the motion strategy of the target moving subject based on task trajectory, basic action priors, and task environment information, according to an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating the process of learning tasks such as grasping when only a partial video of the hand is captured in an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0011] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0012] Example 1: like Figure 1 As shown, the present invention provides a method for training a humanoid robot's motion strategy, comprising: S1. Obtain a motion video of a humanoid reference moving subject performing a target task, and determine the trajectory of a locally visible body part of the reference moving subject based on the motion video; S2. Based on the target task and the structural information of the humanoid robot, determine the target body part set, and map the trajectory of the locally visible body part to the target body part set to determine the motion trajectory of the humanoid robot performing the target task. S3. Obtain the basic action prior library corresponding to the humanoid robot. The basic action prior library is obtained by controlling the humanoid robot to perform basic action tasks and performing strategy training. S4. Based on the motion trajectory, the target task, the state of the humanoid robot, and the basic action prior library, generate auxiliary posture constraints corresponding to body parts not covered by the motion trajectory. S5. Train the motion strategy of the humanoid robot according to the motion trajectory and the auxiliary posture constraints.

[0013] This invention acquires motion videos of a reference moving subject performing a target task, determines the trajectories of locally visible body parts based on the motion videos, and maps the trajectories of the locally visible body parts to the humanoid robot based on the target task and the structural information of the humanoid robot, thereby determining the motion trajectory corresponding to the humanoid robot performing the target task. It can utilize the demonstration motion information of the reference moving subject to realize motion trajectory transfer for humanoid robots, overcoming the problem in the prior art that demonstration actions are difficult to apply directly due to structural differences between different moving subjects.

[0014] In the description of the embodiments of this application, the reference motion subject of the humanoid structure can be a human, or a robotic device that simulates the upper body or whole body shape and movement ability of a human. The composition of the robotic device includes, but is not limited to, a full-body humanoid robot and a half-body humanoid robot without a complete lower limb structure. For example, the specific form of the half-body humanoid robot may include, but is not limited to, an upper-body robot, a dual-arm robot, a mobile chassis plus a half-body robot, or other humanoid robots without a complete lower limb structure. Regardless of whether the motion subject adopts a combination of full-body bionic joint drive, half-body fixed base installation, or mobile chassis plus upper body, and regardless of whether it is used for service interaction, industrial operation, scientific research verification, or special operation scenarios, it falls within the scope of the reference motion subject of the humanoid structure defined in this application. However, it does not necessarily mean that the motion subject must have the same number of joint degrees of freedom or range of motion as a human, nor does it specifically mean that it must maintain an upright posture in all application scenarios.

[0015] As an optional implementation, step S1, acquiring a motion video of a humanoid reference moving subject performing a target task, and determining the trajectory of locally visible body parts of the reference moving subject based on the motion video, includes: S11. Obtain motion video of a humanoid reference moving subject performing the target task; S12. Obtain the body posture information, task object information, and working environment information of the reference moving subject based on continuous image frames in the motion video. The task object information is used to determine the interactive object and task constraints corresponding to the target task, such as the position and state of the target object, tool, door, drawer, button, part, needle, thread, cloth, embroidery frame, or other manipulated object. The working environment information is used to determine the environmental constraints in the trajectory generation process, such as the environmental structure of the ground, desktop, chair, operating table, support surface, obstacles, safety boundary, and local working area. The body posture information of the reference moving subject is used to characterize the spatial state of the human body or humanoid robot body parts. like Figure 4 As shown, the motion video is processed by frame segmentation, and joint perception is performed on consecutive image frames to obtain information on the reference moving subject, the task object, and the working environment. Specifically, for the body posture information of the reference moving subject, it can include body part positions, body keypoint confidence, hand keypoints, tool end pose, task object pose, task object state, environmental semantic labels, support surface parameters, obstacle positions, and local working area boundaries; for the task object, its position, posture, motion trajectory, operable area, and contact area can be obtained through target detection, keypoint detection, depth estimation, pose solving, or point cloud registration; for the working environment, desktop, ground, workbench, chair, obstacles, support surface, reachable space, and collision boundaries can be identified through semantic segmentation, depth estimation, plane fitting, or 3D reconstruction. S13. Determine the visibility status of each body part based on the image features, pose estimation results, and occlusion status of each body part in consecutive image frames. S14. Generate a body part visibility mask based on the visibility state, and filter body parts that meet the visibility conditions from the body posture information based on the body part visibility mask. S15. Based on the positional changes of the selected body parts in consecutive image frames, generate the corresponding locally visible body part trajectories.

[0016] As an optional implementation, step S14, generating a body part visibility mask based on the visibility state, and filtering body parts that meet the visibility conditions based on the body part visibility mask, includes: S141. Read the keypoint probability or heatmap peak value output by the pose estimation model as the keypoint confidence, calculate the ratio of the visible area of ​​the body part segmentation to the area of ​​the predicted region of the body part as the visible area ratio, determine the temporal continuity based on the continuity of position and pose changes in adjacent frames, and determine the degree of occlusion based on the ratio of the occluded region area to the predicted region area; calculate the visibility score of each body part in each image frame based on the keypoint confidence, visible area ratio, temporal continuity, and degree of occlusion. S142. Compare the visibility score with a preset visibility threshold to generate a body part visibility mask; S143. Body parts whose visibility score is greater than or equal to the preset visibility threshold and whose consecutive frame count meets the condition reach a preset frame count are identified as body parts that meet the visibility condition.

[0017] like Figure 5 As shown, based on the body part visibility mask, body parts that meet the visibility conditions are selected from the body pose information, and the trajectory of locally visible body parts is generated based on the position change relationship of the selected body parts in consecutive image frames.

[0018] The visibility score is used to quantify whether parts such as the head, torso, arms, hands, fingers, pelvis, legs, feet, and tool ends are visible, occluded, have reliable pose estimation, and are suitable for retargeting in different time frames. The calculation expression for the visibility score is as follows: in, Let represent the visibility score of the i-th body part in the t-th frame. This represents the average confidence level of the keypoint or pose estimation results corresponding to the i-th body part of the reference moving subject. This represents the proportion of the visible area of ​​the i-th body part of the reference moving subject in the image. This indicates the temporal continuity of the i-th body part of the reference moving subject in adjacent frames. The degree to which the i-th body part of the reference moving subject is obscured by other body parts, task objects, tools, or environmental objects is represented by α, β, γ, and δ, which are the weighting coefficients of the reference moving subject, and α+β+γ+δ=1.

[0019] when When the visibility threshold is greater than or equal to a preset visibility threshold, the body part is marked as visible and usable for trajectory extraction; when... If the visibility threshold is less than the preset threshold, the body part will be marked as invisible, low confidence, or unsuitable for direct redirection.

[0020] For example, taking α=0.35, β=0.25, γ=0.25, δ=0.15, and setting the visibility threshold η=0.60; when If the condition is met for ≥η and 5 consecutive frames, the body part is marked as visible and used for trajectory extraction; otherwise, it is marked as invisible or low confidence. The above weights, thresholds, and number of consecutive frames can be adjusted based on the camera viewpoint and the validation set results of the pose estimation model.

[0021] For body parts that meet the visibility conditions for several consecutive frames, they are considered as candidate locally visible body parts. For body parts that are occasionally missing or temporarily occluded, interpolation or smoothing can be performed by combining the trajectories of the preceding and following frames. For body parts that are not visible for a long time or have a consistently low confidence level, they are not used as direct redirection objects, but are instead handed over to the basic action prior library to generate auxiliary poses or task condition action constraints.

[0022] As an optional implementation, for S15, based on the positional changes of the selected body parts in consecutive image frames, a corresponding locally visible body part trajectory is generated. In one specific implementation, the locally visible body part trajectory can be represented as: in, Let i represent the locally visible trajectory of the i-th body part of the moving subject of the humanoid structure. The moving subject of the humanoid structure represents the spatial position of that body part within the moving subject frame t of the humanoid structure. This represents the pose or rotation matrix of the body part. The moving body of a humanoid structure represents the joint angle corresponding to that body part. The moving subject of a humanoid structure indicates the speed or direction of movement of that body part. This indicates the confidence level of the trajectory. For hand or tool end-effector trajectories, It can indicate the position of the wrist, key points of the fingers, end effector, or tool tip. The moving subject of the humanoid structure can represent wrist posture, tool posture, or end effector posture.

[0023] Extraction of locally visible body part trajectories can include frame-by-frame keypoint detection, cross-frame identity association, 2D-to-3D pose recovery, joint angle estimation, tool end-effector pose estimation, and temporal smoothing. Continuous available frames can be filtered based on body part visibility masks, short-term missing frames can be interpolated for completion, and trajectory points with significant jumps or that do not meet kinematic constraints can be removed or corrected.

[0024] In particular, the extracted local trajectory can be validated based on reprojection error, trajectory continuity, joint length consistency, degree of velocity abrupt change, and attitude estimation confidence. When the reprojection error of the trajectory is less than a preset threshold, the velocity change between adjacent frames is less than a preset threshold, the change in body link length is within the allowable range, and the average confidence level meets the requirements, the system determines that the local trajectory can be used as a candidate trajectory for the core task trajectory; otherwise, the system reduces the weight of the trajectory or does not use it as a direct redirection object.

[0025] In the description of the embodiments of the present invention, the target body part set refers to a combination of at least one body part that needs to be perceived, tracked or controlled in order to perform the task in a specific task scenario, and the combination of body parts is dynamically determined according to the task type.

[0026] For example, in a tool handling task, the target body parts set may include hands, arms, head, and tool ends; in an embroidery task, the target body parts set may include hands, fingers, wrists, tool ends, and necessary arm joints; in a carrying task, the target body parts set may include arms, hands, torso, and necessary lower body moving parts.

[0027] like Figure 6 As shown, firstly, a body part mapping relationship is established between the reference moving subject and the humanoid robot, and the trajectory of locally visible body parts is mapped to the target body part set according to the body part mapping relationship; furthermore, the motion constraints of the mapped trajectory are adjusted according to the differences in skeletal structure, joint degrees of freedom and range of motion between the reference moving subject and the humanoid robot.

[0028] The topological mapping relationship of body parts can be represented as a directed graph G=(V,E): Node V records the semantic name, coordinate system, and degrees of freedom of body parts, joints, or end effectors, and edges E record the kinematic chain connections between adjacent parts. The source and target graphs are first matched with end effector nodes according to task function, and then upstream nodes are matched along the kinematic chain towards the root; for example, the human right hand—right wrist—right elbow—right shoulder are mapped sequentially to the target robot's right end effector—wrist joint group—elbow joint—shoulder joint group. Source nodes are not directly copied if they do not have corresponding nodes in the target graph; when target nodes do not have corresponding nodes in the source graph, auxiliary posture constraints are generated from prior knowledge of the basic actions.

[0029] As an optional implementation, in step S2, mapping the trajectory of the locally visible body part to the target body part set to determine the motion trajectory corresponding to the humanoid robot performing the target task includes: S21. Establish a body part mapping relationship between the reference motion subject and the humanoid robot; S22. Based on the body part mapping relationship, map the trajectory of the locally visible body part to the target body part set; S23. Based on the differences in skeletal structure, joint degrees of freedom, and range of motion between the reference moving body and the humanoid robot, the motion constraints of the mapped trajectory are adjusted to determine the motion trajectory corresponding to the humanoid robot performing the target task.

[0030] As an optional implementation, step S23, based on the differences in skeletal structure, joint degrees of freedom, and range of motion between the reference moving body and the humanoid robot, performs coordinate transformation and motion constraint adjustment on the mapped trajectory to determine the motion trajectory corresponding to the humanoid robot performing the target task, including: S231. Establish trajectory coordinate transformation relationship based on the positional and orientational relationships between the reference moving body and the corresponding coordinate systems of the humanoid robot; S232. Based on the size ratio between the reference moving body and the corresponding body part of the humanoid robot, the trajectory of the locally visible body part is scaled to obtain the scale adjustment result. S233. Based on the trajectory coordinate transformation relationship and the scale adjustment result, the desired trajectory of the corresponding body part of the humanoid robot is obtained; S234. Based on the desired trajectory, the joint trajectory is solved using the kinematic model of the humanoid robot to determine the motion trajectory corresponding to the humanoid robot performing the target task.

[0031] Specifically, for S232, the moving body of the humanoid structure at the position point in the source trajectory. The desired position in the target task space can be obtained based on the link length ratio and coordinate transformation: in, This represents the desired trajectory point in the target humanoid robot. This represents the trajectory point of a reference body part of the moving subject. This represents the trajectory reference point of the moving subject. Let R represent the reference point of the target humanoid robot, R represent the trajectory coordinate transformation relationship from the reference motion subject coordinate system to the target humanoid robot coordinate system, and λ represent the scale factor determined by the link length between the corresponding body part of the reference motion subject and the corresponding body part of the target humanoid robot.

[0032] For S233, the desired trajectory of the corresponding body part of the humanoid robot can be obtained through the following optimization problem: in, This represents the joint angle of the humanoid robot at time t. This represents the positive kinematics result of the humanoid robot. and These represent the desired end-effector position and desired end-effector attitude, respectively. Indicates the cost of a collision. This indicates the cost of joint limitation. This represents the cost of stability. , , , , and For the corresponding weights.

[0033] As an optional implementation, it also includes the following: when there is a difference in degrees of freedom between the reference motion subject and the humanoid robot: The task stages are determined based on the expected state changes of the task object, and the end effector, tool, or body part that directly causes the state changes is selected in each task stage; its target position, target posture, contact state, and allowable error are determined as the key motion targets in the task space. For example, in a button pressing task, the end effector displacement, end effector posture, and contact holding time in the button pressing direction are taken as the key motion targets; For the missing degrees of freedom of the humanoid robot, with the goal of minimizing the weighted sum of the position error, posture error and contact error of the key motion target, the trajectory of the locally visible body part is projected into the reachable task space of the humanoid robot. For the redundant degrees of freedom of the humanoid robot, a cost function is established that includes the joint changes, collision distance, joint limit margin, and deviation of the center of mass relative to the support area at adjacent time moments. The goal is to minimize the cost function under the condition of satisfying the key motion target of the end effector. The joint trajectory is then obtained, and the redundant degrees of freedom are optimized and allocated based on the joint trajectory.

[0034] For the redundant degrees of freedom of the humanoid robot, under the condition that the error of the key motion target does not exceed the preset threshold, the target joint angle at each moment is obtained by minimizing the changes in joint velocity and acceleration, the collision cost caused by insufficient safe distance from obstacles, the limit cost caused by approaching the joint limit, and the stability cost caused by the center of mass exceeding the support area.

[0035] In one specific implementation, for the redundant degrees of freedom of the humanoid robot, the joint trajectory of length T is... To optimize, the cost function can be expressed as: in, This represents the joint angle vector of the humanoid robot at time t; the first term is used to constrain the changes in joint angles between adjacent time steps, so that the joint trajectory remains continuous and smooth. This represents the minimum distance between the k-th robot link and environmental obstacles or other robot links. This indicates a preset safe distance; a collision penalty occurs when the minimum distance is less than the safe distance. The margin of the j-th joint relative to the joint limit can be expressed as: in, and Let these represent the minimum and maximum allowed joint angles of the j-th joint, respectively. This indicates the preset joint limit safety margin; This indicates the projection position of the humanoid robot's center of mass onto the supporting surface. This indicates a pre-defined stable support area defined by the foot, fixed base, mobile chassis, or other support structures. This indicates the deviation of the centroid projection position relative to the stable support region; , , and These are the weighting coefficients for trajectory smoothing, collision avoidance, joint limiting, and stability, respectively.

[0036] Under the condition of satisfying the key motion objectives, the above cost function is minimized, that is: And satisfy: as well as: in, This represents the optimized joint trajectory. This represents the function for calculating the attitude error of the end effector. and These represent the end effector position calculated using forward kinematics at the robot's current joint state and the end effector pose at the robot's current joint state, respectively. and These represent the end effector target position and target orientation, determined by the trajectory of locally visible body parts, respectively. and These represent the allowable position error threshold and attitude error threshold, respectively. When the target task involves grasping, pressing, plugging, or tooling operations, contact state constraints between the end effector and the task object can be further set.

[0037] Through the above optimization, while maintaining the position, posture or contact relationship of the end effector to meet the key motion objectives, the joint states of the shoulder, wrist, torso, pelvis or other body parts are adjusted by utilizing the joint degrees of freedom added to the target humanoid robot relative to the reference moving body, thereby obtaining a joint trajectory that meets the requirements of trajectory smoothness, collision avoidance, joint limitation and stability, and the joint trajectory is used as the result of optimized allocation of redundant degrees of freedom.

[0038] Furthermore, when the degrees of freedom of the source humanoid structure and the target humanoid robot differ, the system does not require a one-to-one correspondence between the source joints and the target joints. Instead, it prioritizes maintaining consistency in key trajectories within the task space, including the end effector position, wrist posture, tool end effector trajectory, or hand-object contact relationship. For degrees of freedom lacking in the target robot, the system projects the source trajectory into the task space accessible to the target robot; for degrees of freedom exceeding those of the target robot, the system optimizes their allocation based on trajectory smoothing, obstacle avoidance, joint constraints, and stability requirements.

[0039] As an optional implementation, step S3, obtaining the basic action prior library corresponding to the humanoid robot, includes: S31. Control the humanoid robot to perform basic motion tasks; S32. Collect the state information and action information of the humanoid robot during the execution of the basic action task; S33. Train a basic action strategy based on the state information and action information, store the trained basic action strategy in the basic action prior library, and establish a basic action index based on the state information and action information corresponding to different basic action tasks.

[0040] like Figure 7 As shown, the auxiliary posture constraint generation process includes task and environment analysis, structural capability and reachability analysis, auxiliary body part determination, and auxiliary posture constraint generation. Specifically, based on the core task trajectory, task object information, working environment information, and the target moving body state, it is determined whether the core task trajectory is located within the target moving body's current reachable space; for body parts not covered by the core task trajectory or requiring supplementary constraints, matching basic action strategies are selected from the basic action prior library, and corresponding auxiliary posture or task condition action constraints are generated.

[0041] As an optional implementation, step S4, generating auxiliary posture constraints for body parts not covered by the motion trajectory based on the motion trajectory, the target task, the state of the humanoid robot, and the basic action prior library, includes: S41. Determine the auxiliary body parts of the humanoid robot that are not covered by the motion trajectory based on the motion trajectory; S42. Based on the target task and the humanoid robot's state, select a matching basic action strategy from the basic action prior library; S43. Generate auxiliary posture constraints for the body parts not covered by the motion trajectory based on the matched basic motion strategy; Specifically, the target robot's current joint state, the pose of the task object, the support surface, obstacles, and the core task trajectory are input into the matching basic motion strategy. The reference joint trajectory, the allowed joint angle range, the target torso or pelvis pose, the foot contact state, and the center of mass stability range output by the strategy are read, and auxiliary posture constraints are generated according to the body parts corresponding to each output.

[0042] In some specific implementations, the auxiliary posture constraints (or task-conditional action constraints) can be generated through a combination of constraint template invocation, basic action strategy output, and optimization solution. The system first determines the body parts of the target humanoid robot that are not directly constrained by the core task trajectory based on the body part visibility mask and the target body part set. Then, based on the task object position, working environment information, and the target robot's current state, it selects corresponding basic action entries from the basic action prior library. Subsequently, the task object position, support surface information, reachable space, obstacle position, and the robot's current joint state are input into the selected basic action entries to generate the corresponding auxiliary posture or task-conditional action constraints.

[0043] The auxiliary posture constraints or task condition action constraints may include at least one of the following: trunk posture, pelvic posture, lower limb joint angle range, foot contact state, center of mass stability range, support surface constraint, arm obstacle avoidance constraint, end effector posture constraint, or tool contact constraint. This constraint may be expressed as a target posture, joint angle range, trajectory reference, contact condition, cost function term, or reward function term, rather than being directly expressed as a low-level motor control command.

[0044] For example, when only the hand trajectory is captured and the task object is on a table, basic action items such as standing stability, trunk posture adjustment, and arm approach to the target can be invoked to generate foot contact constraints, center of mass stability constraints, trunk posture constraints, and arm approach constraints. When the task object is on the ground, basic action items related to squatting, bending over, or center of gravity shift can be invoked to generate pelvic height constraints, knee flexion constraints, trunk forward tilt constraints, and foot support constraints. When the task object exceeds the currently reachable space, basic action items such as walking, turning, or repositioning can be invoked to generate moving target, turning target, or repositioning constraints. Further optimization can be used to ensure that the core task trajectory and auxiliary posture constraints simultaneously meet the robot's reachability, stability, joint constraints, and collision avoidance requirements. When the core task trajectory is within the target robot's currently reachable space, strategies such as arm approach, arm obstacle avoidance, torso posture adjustment, maintaining balance, or end effector fine tracking can be invoked. When the core task trajectory exceeds the target robot's currently reachable space, strategies such as walking, turning, center of gravity shifting, or repositioning can be invoked. When the target task is a desktop grasping or embroidery task, strategies such as sitting stabilization or standing stabilization can be invoked to keep the robot stable while performing fine hand movements.

[0045] Taking desktop grabbing as an example, if the target point is within the currently accessible space and the desktop height is 0.75m, standing stability and arm approach can be selected as the target action items. The standing stability item outputs constraints such as the contact state of both feet, the center of mass being within the double-foot support polygon and the distance from the boundary being no less than 0.03m. The arm approach target item outputs a reference trajectory with an end-position error of no more than 0.02m. If the distance from the target point to the shoulder exceeds the maximum reachable distance of the arm, then walking repositioning and standing stability are selected instead. The base or foot repositioning target is generated first, and then the core hand trajectory is executed.

[0046] like Figure 8 As shown, training states, constraint information, and training samples are constructed based on task trajectory, prior knowledge of basic actions, and task environment information. The target moving subject's motion strategy is then trained according to the configured training method. Specifically, reinforcement learning, imitation learning, behavior cloning, or hierarchical control can be used to train the target moving subject's motion strategy. During training, strategy parameters are adjusted based on task trajectory constraints, auxiliary posture constraints, and task completion evaluation information until a motion strategy that meets the task execution requirements is obtained.

[0047] As an optional implementation, step S5, training the motion strategy of the humanoid robot based on the motion trajectory and the auxiliary posture constraints, includes: S51. Determine the task target constraints based on the motion trajectory; S52. Determine the motion constraints of non-core task body parts based on the auxiliary posture constraints; S53. During the strategy training process, the motion strategy parameters are jointly optimized based on the task objective constraints and the motion constraints.

[0048] During the training process of the humanoid robot, a training objective function can be constructed by combining task object information and working environment information. The training objective function can be expressed as: in, This indicates the loss in core mission trajectory tracking. This indicates the loss due to auxiliary posture or task-conditional action constraints. This indicates loss due to hand-object contact or tool-object contact. This indicates a loss of robot stability. This indicates a loss of smoothness in motion. The values ​​represent collision avoidance losses, with λ1 to λ6 being the corresponding weight coefficients. This training data organization method allows for the simultaneous constraint of the target robot to complete the core task actions corresponding to the locally visible trajectory, while ensuring that un-photographed body parts meet stability, accessibility, and safety requirements.

[0049] The embodiment is merely a specific example and does not indicate that this is the only way to implement the present invention.

[0050] Example 2: like Figure 2 As shown, the difference between this second embodiment and the first embodiment is that this second embodiment divides the humanoid robot motion strategy training method into a basic action pre-training stage and a local video task learning stage.

[0051] like Figure 3 As shown, in the basic movement pre-training stage, the target humanoid robot learns several basic movement strategies in advance through methods such as teaching learning, simulation training, reinforcement learning, imitation learning, behavior cloning, manually designed controllers, or real robot data acquisition. These basic movement strategies may include standing, sitting, walking, turning, reaching out, grasping, placing, maintaining balance, arm approaching a target, arm obstacle avoidance, torso posture adjustment, lower limb support, center of gravity transfer, end effector posture maintenance, and end effector fine tracking. These basic movement strategies can be stored in a basic movement prior library in the form of action primitives, skill models, neural network strategies, motion control strategies, trajectory generation models, reward function templates, loss function templates, constraint templates, or controller parameters.

[0052] The basic motion pre-training phase can be conducted in a simulation environment, on a real robot, or by combining simulation with real data.

[0053] In one implementation, basic motion strategies from a priori library of basic motions are selected based on the core task trajectory, the location of the task object, the working environment information, and the current state of the target humanoid robot. Specifically, a motion matching score can be calculated for each candidate basic motion strategy. S(a)=w1R(a)+w2T(a)+w3E(a)+w4P(a)-w5C(a)-w6L(a) Where 'a' represents the candidate basic motion strategy, R(a) represents the degree to which the motion strategy satisfies the reachability of the core task trajectory, T(a) represents the degree to which the motion strategy matches the target task type, E(a) represents the degree to which the motion strategy matches the working environment, P(a) represents the degree to which the motion strategy matches the robot's current posture or motion state, C(a) represents the collision risk cost, L(a) represents the joint limit, stability, or motion capability constraint cost, and w1-w6 are all weight coefficients. The basic motion strategy with the highest score or the lowest total cost is selected as the auxiliary motion prior for the current task.

[0054] After training, the system stores these basic motion strategies in the form of motion primitives, skill models, neural network strategies, controller parameters, trajectory generation models, reward function templates, or constraint templates, forming a basic motion prior library. This basic motion prior library is not simply a collection of motion names, but a structured data set used to describe, store, retrieve, and call up the existing motion capabilities of the target humanoid robot.

[0055] Specifically, the basic action prior library stores each basic action as a searchable, callable, and combinable basic action entry. Each basic action entry can include information such as action identifier, action name, action category, applicable scenario, input information, output information, execution model, constraints, success criteria, and combinable actions. Among them, the action identifier is used to uniquely identify the basic action, such as stand_balance, arm_reach_target, trunk_adjust, walk_relocate, or end_effector_tracking; the action name is used to indicate the specific meaning of the basic action, such as standing stability, arm approaching a target, trunk posture adjustment, walking approaching a target, or end-effector fine tracking; the action category is used to indicate whether the basic action belongs to the support stability, movement approach, upper limb operation, posture adjustment, obstacle avoidance, or end-effector tracking category; the applicable scenario is used to describe the task environment in which the basic action is suitable for being invoked, such as desktop grabbing, low-level picking, long-distance target approach, fine tool operation, bi-arm carrying, writing, embroidery, or upper body partial operation tasks.

[0056] The input information may include at least one of the following: the current joint state of the target robot, the position of the task object, the posture of the task object, the core task trajectory, the target end effector pose, the obstacle position, the support surface information, the body part visibility mask, or the task type. The output information may include at least one of the following: the target joint trajectory, the end effector trajectory, the torso posture, the pelvic posture, the foot contact state, the center of mass stability constraint, the support constraint, the obstacle avoidance constraint, or the controller parameters. The execution model may be an action primitive, a neural network strategy, a motion control strategy, a trajectory generation model, an inverse kinematics model, a trajectory optimization model, or the controller parameters. The constraints may include joint limits, velocity limits, collision avoidance, foot contact stability, the center of mass being within the support area, the end effector pose error being less than a preset threshold, or the hand-object contact relationship meeting the task requirements. The success determination conditions may include the end effector reaching the target area, stable grasping, the target object undergoing the expected state change, the robot remaining stable, no collision occurring, or the trajectory tracking error being less than a preset threshold. The combinable actions are used to record which other basic actions the basic action can be used in conjunction with. For example, arm approaching a target can be combined with standing stability, torso posture adjustment, arm obstacle avoidance, end effector fine tracking, and grasping actions.

[0057] For example, the basic motion prior library can store a basic motion entry called "arm approaching target". This entry is identified as `arm_reach_target`, categorized as upper limb manipulation, and applicable scenarios include grasping, placing, pressing, plugging / unplugging, or tool approach tasks. Applicable body parts include the shoulder, upper arm, forearm, wrist, hand, or end effector. Input information for this entry can include the target object's position, target object posture, core task trajectory, robot's current joint state, reachable space information, and obstacle information. Output information can include shoulder joint trajectory, elbow joint trajectory, wrist joint trajectory, end effector approach trajectory, and arm posture constraints. Constraints can include joint angles not exceeding joint limits, the end effector gradually approaching the target area, the arm links maintaining a safe distance from obstacles, and trajectory changes meeting smoothness requirements. Success criteria can include the end effector reaching the target grasping area, the end effector pose error being less than a preset threshold, and no collision occurring. Combined actions can include standing stability, torso posture adjustment, arm obstacle avoidance, fine end effector tracking, and grasping.

[0058] For example, the basic motion prior library can store a basic motion entry called "Standing Stability". This entry's motion category is support stability, and its applicable scenarios include desktop grasping, assembly, delivery, tool operation, or upper body partial operation tasks. The input information for this entry can include the robot's current joint state, foot contact state, ground or support surface information, target object position, and core task trajectory; the output information can include lower limb support constraints, foot contact constraints, center of mass stability constraints, and trunk posture constraints; constraint conditions can include no foot slippage, the center of mass being within the support area, the trunk tilt angle being less than a preset threshold, and the joint angle not exceeding the joint limit; success determination conditions can include the robot not falling while executing the upper body operation trajectory, stable foot contact, and the center of mass offset being less than a preset threshold.

[0059] When learning locally visible video tasks, one or more basic action entries are retrieved from the basic action prior library based on the core task trajectory, the location of the task object, the working environment information, and the current state of the target robot. If the core task trajectory is within the target robot's currently reachable space, basic actions such as standing stability, torso posture adjustment, arm approaching the target, arm obstacle avoidance, and fine tracking by the end effector can be invoked. If the core task trajectory exceeds the currently reachable space, basic actions such as walking, turning, center of gravity transfer, or repositioning can be invoked to move or adjust the target robot to a position suitable for performing the local operation trajectory.

[0060] Taking a grasping task that only captures a portion of the hand in video as an example, the system extracts the hand trajectory, wrist trajectory, and hand-object contact relationship from the video and uses these as the core task trajectory. Since the video does not capture the torso, lower limbs, and feet of the demonstration subject, the system does not forcibly reconstruct the complete full-body motion. Instead, based on the fact that the target object is located on a table, it calls upon standing stability motion entries, torso posture adjustment motion entries, arm approaching the target motion entries, and end effector fine tracking motion entries from the basic motion prior library. The system uses the partial hand trajectory to constrain the movement of the target robot's hand or end effector, and uses basic motion entries such as standing stability, torso posture adjustment, and arm approaching the target to generate auxiliary posture or task-conditional motion constraints for the uncaptured body parts, ultimately forming training data that can be used to train the target humanoid robot's motion strategy.

[0061] Example 3: This embodiment, based on the aforementioned embodiments, provides a specific example of upper body partial trajectory redirection in a desktop capture task.

[0062] like Figure 4 and Figure 5As shown in this embodiment, the camera captures the process of a human grabbing a cup on a table. Due to the camera's installation angle, tabletop obstruction, and limited field of view, only the person's head, torso, right arm, and right hand are visible in the video; the pelvis, legs, and feet are not visible.

[0063] First, the system acquires the motion video of the grasping task and breaks it down into consecutive image frames arranged in chronological order. For each frame, the system identifies the human body, the cup, and the tabletop. The human body is the moving human-shaped structure, the cup is the task object, and the tabletop is the supporting surface in the work environment.

[0064] Then, body pose estimation is performed on each frame of the image to obtain two-dimensional key points, three-dimensional key points, SMPL parameters, or human body mesh representation.

[0065] Since the video does not capture the entire human body, the estimation results for the legs and feet may be missing or have low confidence. In this embodiment, a body part visibility mask is generated based on the visibility status and confidence of each body part. This mask indicates that the head, torso, right arm, and right hand are visible, while the pelvis, legs, and feet are invisible or have low confidence.

[0066] In one parameter example, we take α=0.35, β=0.25, γ=0.25, δ=0.15, and η=0.60. The right hand's K, A, T, and O values ​​are 0.92, 0.88, 0.90, and 0.05, respectively, corresponding to V=0.87, which is above the threshold for five consecutive frames. The right foot's K, A, T, and O values ​​are 0.28, 0.10, 0.35, and 0.80, respectively, corresponding to V=0.10. Therefore, the right hand enters the local trajectory set, while the right foot does not enter the direct redirection set.

[0067] Based on body part visibility masks, only locally visible body part trajectories for the head, torso, right arm, and right hand are extracted from the human pose representation. For legs and feet, complete trajectories are not forcibly recovered from the video, and low-confidence results are not used for robot retargeting.

[0068] Next, based on the position of the cup, the height of the table, and the type of grasping task, the body parts that the target humanoid robot needs to track are determined to be the right arm, right hand, and part of the torso.

[0069] The local motion trajectories of the human right arm and right hand are redirected to the target humanoid robot's right arm and right hand. During the redirection process, the proportional relationship and joint range between the human trajectory and the robot trajectory are adjusted according to the structure of the target robot's shoulder joint, elbow joint, wrist joint, and hand, so that the target trajectory meets the robot's executable requirements.

[0070] For example, if the length of the human shoulder-to-wrist kinematic chain is 0.62m and the corresponding kinematic chain length of the target robot is 0.50m, then the scale factor λ = 0.50 / 0.62 is adopted. Using the human shoulder point and the center of the robot shoulder joint as the source and target reference points respectively, coordinate transformation is performed using the extrinsic parameter R and translation from the camera coordinate system to the robot base coordinate system. Inverse kinematics is then performed on the transformed wrist trajectory, and the joint angles are constrained within the limits of the target robot's shoulder, elbow, and wrist joints.

[0071] For the lower body that is not redirected, standing posture constraints are generated based on the characteristics of the desktop grasping task and the standing stability strategy in the basic motion prior library. These standing posture constraints include foot contact constraints, center of mass stability constraints, pelvic height constraints, trunk posture constraints, and knee joint angle range constraints. Thus, the target robot maintains a stable standing posture while learning the grasping trajectories of its right arm and right hand.

[0072] Finally, training data is generated. This training data includes the right arm joint trajectory, the right hand end effector trajectory, the cup position and orientation, table plane information, lower body standing posture constraints, trajectory tracking reward parameters, and task completion reward parameters. This training data can be used to train the humanoid robot's motion strategy for completing a table grasping task.

[0073] Example 4: This embodiment, based on the aforementioned embodiments, provides a specific example of local trajectory redirection and lower body pose completion in a low-ground pickup task.

[0074] like Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown in the image, in this embodiment, the camera captures the process of a human picking up an object from the ground. Due to the limitations of the shooting angle, only the upper body, arms, and hands of the person are clearly shown in the video, while the legs are partially obscured.

[0075] The video is broken down into consecutive image frames, and the human body, ground, target objects, and surrounding obstacles are identified. The system estimates the human pose and generates a visibility mask based on the visibility of body parts. The masking results show that the hands, arms, and torso are visible, while the legs and feet are either invisible or have insufficient confidence.

[0076] The system extracts visible hand, arm, and torso trajectories and identifies the target object as being near the ground. Based on the target object's height and the task type, it determines the task to be a low-level pickup task. Unlike desktop grasping tasks, low-level pickup tasks require the robot to lower its body height so that its hands can reach the target object at a lower location.

[0077] Therefore, the hand and arm trajectories are redirected to the target humanoid robot's arms and hands. Simultaneously, basic motion strategies related to squatting, bending, or center of gravity stability are selected from a priori library of basic motions to generate auxiliary posture constraints for the lower body, which have not been reliably extracted. These posture constraints include pelvic descent constraints, knee flexion constraints, foot contact constraints, center of gravity stability constraints, and torso forward tilt angle constraints.

[0078] In this way, even if the complete leg trajectory cannot be obtained from the video, the target robot can still generate reasonable lower body movements based on the task object, environmental information, and existing basic motor abilities, thereby completing the low-level picking task.

[0079] Example 5: This embodiment, based on the foregoing embodiments, provides an example of redirection from a full-body humanoid robot to a full-body humanoid robot.

[0080] like Figure 4 , Figure 5 , Figure 6 and Figure 8 As shown, in this embodiment, the source full-body humanoid robot performs tasks such as handling, opening doors, grasping desktops, assembling, or tooling, while the target robot is another full-body humanoid robot with a different structure. Both the source robot and the target robot have a full-body structure including upper limbs, torso, lower limbs, and feet, but they may differ in terms of link length, joint degrees of freedom, joint limits, end effector structure, or motion capabilities.

[0081] First, the source full-body humanoid robot is used as the main moving humanoid structure, and the source motion video is processed frame by frame to identify the source robot's body parts, task object, and working environment information. Based on the source robot skeleton, target robot model, task object position, and environmental structure, the correspondence between body parts, joint mapping, and task space mapping between the source full-body robot and the target full-body robot are established.

[0082] For example, when the source robot's right arm has 7 degrees of freedom and the target robot's right arm has 6 degrees of freedom, the position, posture, and hand-object contact relationship of the source robot's right end are set as key motion targets. The 6-degree-of-freedom joint trajectory of the target robot is solved through the aforementioned optimization problem. The extra forearm rotational degrees of freedom of the source robot are not copied one by one, but are compensated by the available degrees of freedom of the target's wrist and shoulder under joint constraints and obstacle avoidance constraints.

[0083] For body parts visible in the video and whose pose estimation confidence meets the requirements, the trajectories of the source robot's upper body, arms, hands, torso, lower limbs, feet, or end effectors can be extracted and redirected based on the structural parameters of the target whole-body robot. When the target task requires full-body coordination, the upper limb manipulation trajectory, torso posture, lower limb support state, or foot movement trajectory can be redirected simultaneously. When the task focuses on manipulation actions, the system can also select only the upper body, arms, hands, or end effectors trajectories relevant to the task as the core task trajectory.

[0084] For body parts that are not visible in the source video, have low confidence, or are not suitable for direct retargeting due to structural differences, the basic action prior library of the target full-body humanoid robot is called. Combined with the position of the task object, the constraints of the working environment, and the current state of the target robot, auxiliary postures or task condition action constraints such as standing, walking, turning, center of gravity transfer, balance maintenance, torso posture adjustment, or foot contact are generated.

[0085] Finally, joint trajector trajector trajector trajectories, auxiliary posture constraints, task object state, and environment state are generated for training the target full-body humanoid robot.

[0086] This embodiment illustrates that the present invention enables structurally adapted skill transfer between full-body humanoid robots, allowing the target full-body robot to learn the task skills of the source full-body robot within the limits of its own structure and dynamic constraints.

[0087] Example 6: Based on the foregoing embodiments, this embodiment provides an example of redirection from a half-humanoid robot to a half-humanoid robot.

[0088] like Figure 4 , Figure 5 , Figure 6 and Figure 8 As shown, in this embodiment, the source robot is a half-body humanoid robot, and the target robot is another half-body humanoid robot with a different structure. The half-body humanoid robot can be an upper-body robot, a dual-arm robot, a half-body robot with a fixed base, a half-body robot with a mobile chassis, or other humanoid robots without a complete lower limb structure. The source robot performs tasks such as grasping, assembling, writing, plugging and unplugging, twisting, or tooling, and the goal is to transfer its upper-body operational skills to the target half-body robot.

[0089] Acquire video footage of a source humanoid half-body robot performing a task, and identify the robot's torso, shoulders, arms, wrists, hands, end effectors, task object, and local working environment. Based on the target half-body robot's structural model, joint degrees of freedom, end effector type, and reachable space, determine the set of transferable body parts.

[0090] During the redirection process, the half-body robot is not required to possess or learn lower limb motion trajectories. Instead, only the trajectories of the upper body, torso, arms, wrists, hands, or end effectors in the source video that correspond to the structure of the target half-body robot and are relevant to the task are redirected. For body parts that the target half-body robot does not possess, the system does not use them as direct learning targets. For local trajectories that cannot be directly copied due to differences in the target robot's structure, trajectory adjustments are made based on joint constraints, reachable space, and end effector constraints.

[0091] For example, the source half-body robot is a fixed-base, dual-arm structure, while the target half-body robot is a mobile chassis with dual arms. The common "torso-shoulder-elbow-wrist-end effector" nodes of both are matched and redirected in topological order. When the target robot's newly added mobile chassis node does not have a source trajectory, a fictitious source chassis trajectory is not generated. However, when the end-body target exceeds the reachable space of the current arm, chassis pose constraints are generated from the "chassis repositioning" basic action item.

[0092] When the target half-robot has a mobile chassis, constraints for chassis movement, turning, or repositioning can be generated by combining the position of the task object and the working environment information. When the target half-robot has a fixed base structure, auxiliary constraints such as torso posture adjustment, arm approach to the target, arm obstacle avoidance, fine end-effector tracking, or maintaining stability can be generated. These auxiliary constraints can be provided by the target half-robot's basic motion prior library.

[0093] Finally, the upper body joint trajector ...ies trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories

[0094] This embodiment illustrates that the present invention enables structurally adaptable skill transfer between humanoid robots, allowing different humanoid robots to learn each other's operational skills within the limits of their own executable body parts and degrees of freedom.

[0095] Example 7: This embodiment, based on the foregoing embodiments, provides an embodiment of redirection from a full-body humanoid robot to a half-body humanoid robot.

[0096] like Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8As shown, in this embodiment, the source robot is a full-body humanoid robot, and the target robot is a half-body humanoid robot. When the source full-body humanoid robot performs tasks such as desktop grasping, parts assembly, tool operation, writing, plugging, unplugging, or twisting, the video may capture the complete body movements of the source robot, including the upper body, arms, hands, torso, pelvis, lower limbs, and feet. However, since the target robot is a half-body humanoid robot, it does not have a complete lower limb structure or does not need to learn lower limb movements. Therefore, the entire body movement trajectory in the source video is not used as the direct learning target for the target robot.

[0097] First, the body structure, task object, and working environment information of the source full-body humanoid robot are identified. Then, based on the structural model, joint degrees of freedom, end effector type, and reachable space of the target half-body humanoid robot, the set of target body parts that the target half-body robot can execute is determined. The set of target body parts may include at least one of the following: upper body, torso, shoulder, arm, wrist, hand, or end effector.

[0098] When the source video contains complete full-body motion, based on the body part visibility mask and the structural capabilities of the target half-robot, only the trajectories of the upper body, arms, wrists, hands, or end effectors that correspond to the target half-robot's structure and are relevant to task execution are extracted and redirected. The lower limbs, feet, or walking trajectories of the source full-body robot are not used as direct learning targets for the half-robot. Lower-body motion information in the source video can be used as task state, support state, or environmental reference information, but the half-robot is not required to replicate this lower-body motion.

[0099] Specifically, the lower limbs and foot nodes of the source full-body robot are marked as "no corresponding node in the target structure" and removed from the core task trajectory. If the source robot reaches a distant object with its hand by walking, the position of the task object relative to the torso at the end of the walking is extracted as the task space target. For mobile chassis-type half-body robots, this target is converted into a chassis repositioning constraint. For fixed-base half-body robots, if the target exceeds the reachable space, the demonstration segment is determined to be unexecutable and direct migration is stopped.

[0100] During the repositioning process, the upper body maneuver trajectory of the source full-body robot is structurally adapted based on the joint limits, link lengths, reachable space, and end effector structure of the target half-body robot. When the target half-body robot has a mobile chassis, constraints for chassis movement, turning, or repositioning can be generated based on the position of the task object and the working environment information. When the target half-body robot has a fixed base structure, auxiliary posture constraints such as torso posture adjustment, arm approach to the target, arm obstacle avoidance, and fine tracking or stabilization of the end effector can be generated.

[0101] Finally, the upper body joint trajector ...ies trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories trajectories

[0102] This embodiment illustrates that, for videos capturing full-body movements, a half-body humanoid robot can learn only the executable upper-body operation skills based on its own structural capabilities, thereby achieving structural adaptation-based skill transfer from a full-body humanoid robot to a half-body humanoid robot.

[0103] Example 8: Based on the foregoing embodiments, this embodiment provides an example of redirection from a half-body humanoid robot to a full-body humanoid robot.

[0104] like Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown, in this embodiment, the source robot is a half-body humanoid robot, and the target robot is a full-body humanoid robot. When the source half-body humanoid robot performs tasks such as grasping, assembling, writing, embroidering, plugging and unplugging, twisting, or tool operation, the video mainly includes information about the source robot's upper body, arms, wrists, hands, end effectors, task objects, and local working environment, but does not include information about the movement of the complete lower limbs or feet.

[0105] First, the source half-body humanoid robot is used as the main humanoid structure for movement. The source motion video is processed frame by frame to identify the trajectory of the source robot's torso, shoulders, arms, wrists, hands, or end effectors. At the same time, the task object and working environment information are identified. Based on the body structure, joint degrees of freedom, end effector type, and motion capabilities of the target full-body humanoid robot, the correspondence between the upper body parts of the source half-body robot and the target full-body robot and the task space mapping relationship are established.

[0106] During the redirection process, the upper body movement trajectory of the source half-body robot is redirected to the corresponding body part of the target full-body humanoid robot, such as the torso, shoulder, arm, wrist, hand, or end effector. Since the source video does not contain complete lower limb motion information, the system does not forcibly recover full-body motion from the source half-body robot video. Instead, it calls the basic motion prior library of the target full-body humanoid robot to generate auxiliary postures or task-conditional motion constraints for the lower body, pelvis, feet, and overall balance of the target full-body robot.

[0107] Specifically, common upper body nodes are redirected according to topological relationships; newly added pelvic, hip, knee, ankle, and foot nodes of the target full-body robot are all marked as passive trajectories. When the core hand trajectory is within the standing reachable space, the standing stability entry is invoked to generate bipedal contact, pelvic height, and center of mass stability range; when the trajectory exceeds the standing reachable space, the walking repositioning entry is invoked to generate foot and pelvic trajectories, and the upper body core trajectory is executed after the robot enters the reachable area.

[0108] For example, when the task object is on a desktop or operating table, standing stability constraints or sitting stability constraints can be generated for the target full-body humanoid robot; when the task object exceeds the currently reachable space, basic action priors such as walking, turning, center of gravity transfer or repositioning can be invoked to enable the target full-body robot to move to a suitable position and execute the upper body operation trajectory; when the task requires continuous fine operation, auxiliary constraints such as torso posture adjustment, foot contact, center of mass stability and collision avoidance can be generated.

[0109] Ultimately, the system generates upper body operation trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajectories ...

[0110] This embodiment illustrates that the present invention can transfer the upper body operation skills of a half-body humanoid robot to a full-body humanoid robot, and use the prior basic movements of the target full-body robot to complete the lower body support, movement and balance-related movements that are not present in the source video.

[0111] Example 9: Based on the foregoing embodiments, this embodiment provides an example of end-point trajectory redirection in a video where only the hand is visible.

[0112] like Figure 9 As shown, in this embodiment, the video footage only includes the process of a person's hand operating the tool, such as tightening screws, pressing buttons, or inserting or removing parts, and the person's complete arm, torso, and lower body cannot be seen.

[0113] First, identify the hand, tool, target object, and operation area, and estimate the key points of the hand, wrist pose, or tool end-effector trajectory. Based on the visibility mask of the body parts, only the hand or tool end-effector trajectory meets the visibility condition.

[0114] When the video is monocular, the known tool size, target object size, desktop plane, or camera calibration parameters can be used to recover the 2D hand keypoints into the 3D end effector trajectory in the robot's base coordinate system through PnP pose solving, planar homography, or scale calibration. If the absolute scale cannot be obtained, it is first normalized to the tool length or target object feature size, and then recovered according to the working scale of the target robot end effector. Taking a 0.20m long screwdriver as an example, its image endpoints are used to determine the scale and tool axis direction, and the tool tip position, tool axis attitude, and rotation angle are used as key motion targets.

[0115] The trajectory of the hand or tool end effector is redirected to the hand or end effector of the target humanoid robot. For the arm, torso, and lower body that are not captured in the video, corresponding posture constraints are generated based on the target object's location, the robot's reachable space, and the basic motion prior library. For example, when the operating area is at the height of the table in front of the robot's chest, the arm approach strategy, arm obstacle avoidance strategy, and standing stability strategy are invoked to generate upper body forward extension constraints and lower body standing constraints; when the operating area is at a farther location, the torso turning, walking, or repositioning strategy is invoked to allow the robot to move or adjust its body and continue performing the hand task.

[0116] In solving the problem, the tool tip trajectory error and tool axis attitude error are taken as the core objectives, and the shoulder, elbow, and wrist joints are taken as inverse kinematic variables. The constraints are joint limits, collision distances with the table and tool, and joint changes in adjacent frames. For the torso and lower limbs that were not photographed, if the end-effector error can be reduced to below 0.02m within the standing reachable space, the standing stability constraint is maintained; otherwise, repositioning is performed until the end-effector enters the reachable space.

[0117] Thus, the robot can learn end effector operation skills with only hand operation videos, and the system can automatically complete the posture constraints of other body parts by using basic action priors.

[0118] Example 10: This embodiment, based on the foregoing embodiments, provides a method for updating a basic action prior library.

[0119] In this embodiment, after the target humanoid robot completes new task strategy training based on locally visible video, the system stores the trained strategy, trajectory segments, constraint parameters or reward function parameters into the basic action prior library.

[0120] For example, if a robot acquires a stable needle-tip tracking strategy and wrist fine-tuning strategy through embroidery tasks, this strategy can be stored as a new basic fine-end-operation action in the basic action prior library. When the robot subsequently learns tasks such as writing, drawing, or carving, it can call upon this basic fine-end-operation action to provide auxiliary posture and control priors for the new task.

[0121] In this way, the basic action prior library can be continuously expanded as the number of robot learning tasks increases, enabling the robot to have stronger pose completion and action generation capabilities in subsequent partially visible video tasks.

[0122] Example 11: This embodiment, based on the foregoing embodiments, provides a method for a robot that already possesses basic skills to learn new skills.

[0123] In this embodiment, the target humanoid robot already possesses basic skills such as standing, walking, turning, reaching out, grasping, and maintaining balance, but has not yet learned new tasks such as embroidery, writing, or fine motor skills. The system uses the robot's existing basic skills as part of a priori library of basic movements and acquires locally visible motion videos of humans performing new tasks.

[0124] When the video primarily captures the hand, wrist, tools, and localized work areas of the task object, the system extracts visible and reliable local motion trajectories from the video and redirects them as the core task trajectory to the target humanoid robot's hand, wrist, end effector, or related arm joints. For body parts not captured in the video or unsuitable for direct redirection, the system utilizes the robot's existing basic skills such as standing, sitting stability, arm approach to the target, torso posture adjustment, and maintaining balance to generate auxiliary postures or task-conditional motion constraints.

[0125] After training is completed, the motion strategy, end-effector trajectory tracking capability, hand-object contact control capability, or tool operation constraint parameters corresponding to the new task can be updated to the basic action prior library, so that the target humanoid robot can reuse the skill when learning similar tasks in the future.

[0126] Example 12: This embodiment provides a humanoid motion subject motion strategy training system, including: a processor and a memory; The memory stores a computer program, which, when executed by the processor, is used to implement the humanoid robot motion strategy training method described in the foregoing embodiments.

[0127] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for training a humanoid robot's motion strategy, characterized in that, include: Acquire motion video of a humanoid reference moving subject performing a target task, and determine the trajectory of locally visible body parts of the reference moving subject based on the motion video; Based on the target task and the structural information of the humanoid robot, a set of target body parts is determined, and the trajectory of the locally visible body parts is mapped to the set of target body parts to determine the motion trajectory of the humanoid robot performing the target task. A basic action prior library corresponding to the humanoid robot is obtained. The basic action prior library is obtained by controlling the humanoid robot to perform basic action tasks and performing strategy training. Based on the motion trajectory, the target task, the state of the humanoid robot, and the basic action prior library, auxiliary posture constraints are generated for body parts not covered by the motion trajectory. The motion strategy of the humanoid robot is trained based on the motion trajectory and the auxiliary posture constraints.

2. The humanoid robot motion strategy training method according to claim 1, characterized in that, Determining the trajectory of a locally visible body part of the reference moving subject based on the motion video includes: The body posture information, task object information, and working environment information of the reference moving subject are obtained from the continuous image frames in the motion video. The task object information is used to determine the interaction object and task constraints corresponding to the target task. The working environment information is used to determine the environmental constraints in the trajectory generation process. Based on the image features, pose estimation results, and occlusion status of each body part in consecutive image frames, determine the visibility status of each body part. A body part visibility mask is generated based on the visibility state, and body parts that meet the visibility conditions are filtered from the body posture information based on the body part visibility mask. Based on the positional changes of the selected body parts in consecutive image frames, the corresponding locally visible body part trajectories are generated.

3. The humanoid robot motion strategy training method according to claim 2, characterized in that, The step of generating a body part visibility mask based on the visibility state, and filtering body parts that meet the visibility conditions from the body pose information based on the body part visibility mask, includes: Based on the confidence level of key points corresponding to each body part, the proportion of visible area of ​​the body part in the motion video image frame, temporal continuity, and degree of occlusion, the visibility score of each body part in each image frame is determined. Based on the relationship between the visibility score and the preset visibility threshold, a body part visibility mask is generated; The body parts whose visibility score is greater than or equal to the preset visibility threshold are considered as body parts that meet the visibility conditions.

4. The humanoid robot motion strategy training method according to claim 1, characterized in that, The step of mapping the trajectory of the locally visible body parts to the set of target body parts to determine the motion trajectory corresponding to the humanoid robot performing the target task includes: Establish a body part mapping relationship between the reference motion subject and the humanoid robot; Based on the body part mapping relationship, the trajectory of the locally visible body part is mapped to the target body part set; Based on the differences in skeletal structure, joint degrees of freedom, and range of motion between the reference moving body and the humanoid robot, the motion constraints of the mapped trajectory are adjusted to determine the motion trajectory corresponding to the humanoid robot performing the target task.

5. The humanoid robot motion strategy training method according to claim 4, characterized in that, The step of performing coordinate transformation and motion constraint adjustment on the mapped trajectory based on the differences in skeletal structure, joint degrees of freedom, and range of motion between the reference moving body and the humanoid robot, to determine the motion trajectory corresponding to the humanoid robot performing the target task, includes: Based on the positional and orientational relationships between the reference moving body and the corresponding coordinate systems of the humanoid robot, a trajectory coordinate transformation relationship is established; Based on the size ratio between the reference moving body and the corresponding body part of the humanoid robot, the trajectory of the locally visible body part is scaled to obtain the scale adjustment result. Based on the trajectory coordinate transformation relationship and the scale adjustment result, the expected trajectory of the corresponding body part of the humanoid robot is obtained; Based on the desired trajectory, the joint trajectories are solved using the kinematic model of the humanoid robot to determine the motion trajectory corresponding to the humanoid robot performing the target task.

6. The humanoid robot motion strategy training method according to claim 4 or 5, characterized in that, This also includes situations where there is a difference in degrees of freedom between the reference moving body and the humanoid robot: Identify key motion targets in the task space based on task execution requirements; For the missing degrees of freedom of the humanoid robot, with the goal of minimizing the weighted sum of the position error, posture error and contact error of the key motion target, the trajectory of the locally visible body part is projected into the reachable task space of the humanoid robot. For the redundant degrees of freedom of the humanoid robot, a cost function is established that includes the joint changes, collision distance, joint limit margin, and deviation of the center of mass relative to the support area at adjacent time moments. The goal is to minimize the cost function under the condition of satisfying the key motion target of the end effector. The joint trajectory is then obtained, and the redundant degrees of freedom are optimized and allocated based on the joint trajectory.

7. The humanoid robot motion strategy training method according to claim 1, characterized in that, The process of obtaining the basic action prior library corresponding to the humanoid robot includes: Control the humanoid robot to perform basic motion tasks; Collect state and motion information of the humanoid robot during the execution of the basic motion task; The basic action strategy is trained based on the state information and action information, and the trained basic action strategy is stored in the basic action prior library. A basic action index is established based on the state information and action information corresponding to different basic action tasks.

8. The humanoid robot motion strategy training method according to claim 1, characterized in that, The step of generating auxiliary posture constraints for body parts not covered by the motion trajectory based on the motion trajectory, the target task, the state of the humanoid robot, and the basic action prior library includes: Based on the motion trajectory, determine the auxiliary body parts of the humanoid robot that are not covered by the motion trajectory; Based on the target task and the humanoid robot's state, a matching basic action strategy is selected from the basic action prior library; Based on the matched basic motion strategy, auxiliary posture constraints are generated for the body parts not covered by the motion trajectory.

9. The humanoid robot motion strategy training method according to claim 1, characterized in that, The step of training the motion strategy of the humanoid robot based on the motion trajectory and the auxiliary posture constraints includes: Determine the task objective constraints based on the described motion trajectory; The motion constraints of non-core task body parts are determined based on the auxiliary posture constraints. During strategy training, the motion strategy parameters are optimized based on the task objective constraints and the motion constraints.

10. A training system for movement strategies of a humanoid subject, characterized in that, include: Processor and memory; The memory stores a computer program, which, when executed by the processor, is used to implement the humanoid robot motion strategy training method according to any one of claims 1-9.

Images