A method and system for motion trajectory transfer

Abstract

This disclosure provides a method and system for motion trajectory transfer. The method includes: semantically parsing acquired motion segments representing human body movement behavior, and determining the motion trajectory in the motion segments based on the parsed semantic information; determining trajectory constraint information based on the motion trajectory; constructing a target motion trajectory corresponding to the target task based on the semantic information of the target task in the target task scene, using the trajectory constraint information as constraints; and mapping the motion parameters representing the human body in the target motion trajectory to the robot's motion parameters using a preset trajectory optimization algorithm to obtain the robot's motion trajectory. In this disclosure, the motion segments can come from human videos, eliminating the need for complex drag-and-drop teaching and improving efficiency; the separation of the motion trajectory from the original scene enables flexible trajectory transfer; and the inherent constraints of the robot are considered, ensuring that the robot's motion trajectory can be physically used for robot control, thus guaranteeing execution stability.

Description

Technical Field

[0001] This disclosure relates to the field of embodied intelligence, and more particularly to a method and system for transferring motion trajectories. Background Technology

[0002] Currently, the mainstream technical solutions for transferring human actions to robots are mainly divided into two categories: physical teaching and visual motion capture.

[0003] Data acquisition schemes based on physical teaching record joint angle sequences by dragging the robot or using a teach pendant. The drawback of this method is that the teaching process is severely disconnected from natural human movements. Its operation does not conform to the human's habitual movements when performing daily tasks, resulting in a non-intuitive, inefficient teaching process that is difficult to reproduce the fine manipulation and dynamic coordination in human movements.

[0004] Vision-based motion capture solutions utilize cameras to capture human motion videos and estimate robot joint angle sequences based on human posture. While this approach lowers the teaching threshold, it suffers from the following significant drawbacks in practical applications: First, it ignores the inherent kinematic constraints of the robot (such as joint limits, velocity / acceleration limits, and link interference), rendering the generated trajectories physically unusable for robot control. Second, this approach is sensitive to camera perspective and typically relies on structured scenes with multiple cameras, resulting in strong site dependence, high deployment costs, and insufficient flexibility. Third, the generated trajectories are strongly bound to the environment at the time of acquisition; when the positions of objects or the task scenario change, the trajectories cannot adaptively adjust, lacking the ability to generalize trajectory shape and position, making it difficult to support the transfer of robot skills to new scenarios.

[0005] In summary, existing motion transfer solutions struggle to simultaneously achieve ease of data acquisition, physical feasibility of trajectories, low environmental dependence, and scene generalization capability. Therefore, there is an urgent need for a method to extract executable robot trajectories from unstructured human videos to overcome these shortcomings and achieve effective transfer from natural human actions to generalizable robot skills. Summary of the Invention

[0006] This disclosure provides a motion trajectory transfer method and system for efficiently transferring natural human movements into robot trajectories in specific scenarios.

[0007] In view of the above problems, firstly, the present disclosure provides a motion trajectory migration method, including: Semantic parsing is performed on the obtained motion segments representing the movement behavior of human body parts, and the reference nodes in the motion segments and the corresponding action trajectories of each reference node are determined based on the semantic information obtained from the parsing. The trajectory constraint information is determined based on each reference node and the corresponding motion trajectory of each reference node. Based on the semantic information of the target task in the target task scenario, and with the trajectory constraint information as constraints, the target action trajectory corresponding to the target task is constructed. By using a preset trajectory optimization algorithm, the motion parameters representing the movement behavior of human body parts in the target motion trajectory are mapped to the motion parameters of the robot's skeletal structure, thus obtaining the robot's motion trajectory.

[0008] In conjunction with the first aspect, in one possible implementation, semantic parsing is performed on the acquired motion segments representing the movement behavior of human body parts, and the reference nodes in the motion segments and the corresponding action trajectories of each reference node are determined based on the parsed semantic information, including: The motion segment is parsed using a preset semantic parsing model to obtain the semantic information contained in the motion segment describing the movement behavior of human body parts; The spatial coordinates of key locations representing the movement behavior of human body parts in the semantic information are determined as reference nodes in the motion segment; The key action segments representing the movement behavior of human body parts in the semantic information are determined as the action trajectories corresponding to each reference node; the key action segments include the key positions.

[0009] In conjunction with the first aspect, in one possible implementation, trajectory constraint information is determined based on each reference node and the corresponding motion trajectory of each reference node, including: Based on each reference node and the corresponding motion trajectory, the trajectory offset features under the corresponding motion progress of each reference node are determined; wherein, the trajectory offset features under each motion progress are used to characterize the shape features of the motion trajectory. Based on the semantic information of the target task in the target task scenario, and using the trajectory constraint information as constraints, the target action trajectory corresponding to the target task is constructed, including: Based on the semantic information of the target scene, determine the key path locations of the target action to be performed in the target scene; For each critical path location, a reference node corresponding to the movement progress of that critical path location is determined; the movement progress corresponding to that reference node is determined as the movement progress of that critical path location during the movement within the target scene; Using the trajectory offset features as constraints, the target motion information of each sampling point during the execution of the target action is determined based on the motion progress corresponding to each key path position; Based on the target motion information, a target trajectory representing the human body parts performing target actions in the target scene is constructed.

[0010] In conjunction with the first aspect, in one possible implementation, a preset trajectory optimization algorithm is used to map the motion parameters representing the movement behavior of human body parts in the target motion trajectory to the motion parameters of the robot's skeletal structure, thereby obtaining the robot's motion trajectory, including: Based on the mapping model, human body parts are mapped to execution nodes in the robot's skeletal structure; For key execution nodes that perform the target action, under the first constraint condition, determine the trajectory tracking error between the motion parameters of the key execution nodes and the motion parameters of the corresponding human body parts; For each execution node in the robot's skeletal structure, under the second constraint, determine the motion similarity error between the motion parameters of each execution node of the robot and the motion parameters of the corresponding human body parts; Based on the trajectory tracking error and the motion similarity error, the robot motion trajectory, which guides the robot to perform actions according to the target motion trajectory, is obtained by using a kinematic solver.

[0011] In conjunction with the first aspect, in one possible implementation, it further includes: Acquire video data containing human movements and determine the type of acquisition viewpoint; The video data is processed frame by frame, and the three-dimensional spatial information of each part of the human body contained in each frame image is extracted by a preset pose estimation model to obtain three-dimensional pose data representing the human body's motion state. The motion state of the video data acquisition device is determined based on the acquisition viewpoint type corresponding to the video data. Based on the device's motion state, the three-dimensional posture data is unified into the target coordinate system to obtain motion segments that characterize the motion behavior of various parts of the human body in the target coordinate system.

[0012] In conjunction with the first aspect, in one possible implementation, determining the device motion state of the video data acquisition device based on the acquisition viewpoint type corresponding to the video data includes: When the acquisition viewpoint type is a moving viewpoint, the pixel information contained in each frame of the video data is aligned with the inertial information contained in the video data according to the timestamp. Between the timestamps of every two frames, the aligned inertial information is pre-integrated to obtain inertial observation data characterizing the device motion constraints between the two frames. Based on inertial observation data and corresponding pixel information, the spatial coordinates of the device in each frame in the target coordinate system are calculated using the visual inertial odometry method, thereby obtaining the device motion state that characterizes the device's activity trajectory in the video data. Based on the device's motion state, the three-dimensional posture data is unified to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system, including: The 3D pose data in each frame, which is in the coordinate system of the acquisition device, is converted into 3D pose data in the target coordinate system based on the spatial coordinates of the device in that frame. Based on the 3D pose data in the target coordinate system corresponding to each frame, motion segments in the target coordinate system corresponding to the video data are obtained.

[0013] In conjunction with the first aspect, in one possible implementation, determining the device motion state of the video data acquisition device based on the acquisition viewpoint type corresponding to the video data includes: When the acquisition viewpoint type is fixed viewpoint, it is determined that the device is in a fixed state; Based on the device's motion state, the three-dimensional posture data is unified to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system, including: With the device in a fixed state, the three-dimensional attitude data in the coordinate system of the acquisition device is converted into three-dimensional attitude data in the target coordinate system, with the device as the origin. Based on the 3D pose data in the target coordinate system corresponding to each frame, motion segments in the target coordinate system corresponding to the video data are obtained.

[0014] In conjunction with the first aspect, in one possible implementation, the mapping model is constructed using the following method: Extract the structural parameters that define the skeletal structure of the target robot from the robot description file; First semantic features representing the movement function of human body parts and second semantic features representing the movement function of robot skeletal structure are obtained respectively. Based on the matching results of the first semantic features and the second semantic features, a mapping relationship is established between the human body parts and the robot skeletal structure. Based on the structural parameters and the mapping relationship, a mapping model is constructed to characterize the kinematic correspondence between the robot's skeletal structure and human body parts, so as to convert the motion trajectory of human body parts into the motion trajectory of the target robot's skeletal structure through the mapping model.

[0015] In conjunction with the first aspect, in one possible implementation, structural parameters defining the skeletal structure of the target robot are extracted from the robot description file of the target robot, including: Extract the motion attributes and motion constraint parameters of each joint node in the defined target robot skeleton structure; Extract the physical properties and geometric parameters of the three-dimensional shape of each bone link in the target robot's skeletal structure; Extract the bone connection parameters that define the linkage relationship between bone links in the bone structure.

[0016] Secondly, this disclosure provides a motion trajectory transfer system, including: The trajectory processing module is used to perform semantic parsing on the acquired motion segments representing the movement behavior of human body parts, and determine each reference node and the corresponding action trajectory in the motion segment based on the parsed semantic information; determine trajectory constraint information based on each reference node and the corresponding action trajectory; and construct the target action trajectory corresponding to the target task based on the semantic information of the target task in the target task scene and the trajectory constraint information. The motion redirection module is used to map the motion parameters representing the movement behavior of human body parts in the target motion trajectory to the motion parameters of the robot's skeletal structure through a preset trajectory optimization algorithm, thereby obtaining the robot's motion trajectory.

[0017] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a method and system for motion trajectory transfer. The method includes: semantically parsing an acquired motion segment representing the movement behavior of a human body part, and determining each reference node and its corresponding motion trajectory based on the parsed semantic information; determining trajectory constraint information based on each reference node and its corresponding motion trajectory; constructing a target motion trajectory corresponding to the target task based on the semantic information of the target task in the target task scene, using the trajectory constraint information as constraints; and mapping the motion parameters representing the movement behavior of the human body part in the target motion trajectory to the motion parameters of the robot's skeletal structure using a preset trajectory optimization algorithm to obtain the robot's motion trajectory. In this disclosure, the motion segment can be derived from human video, eliminating the need for complex drag-and-drop teaching and improving efficiency; the obtained robot motion trajectory directly originates from human movements, conforming to natural human operating habits; based on the trajectory constraint information, the motion trajectory is separated from the original scene, allowing the motion trajectory to be generalized to different target tasks, achieving flexible trajectory transfer; and the motion parameter mapping considers the inherent kinematic and dynamic constraints of the robot, ensuring that the transferred robot motion trajectory can be physically used for robot control, guaranteeing execution stability. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a motion trajectory migration method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a video capture scenario provided in an embodiment of this disclosure; Figure 3 A schematic diagram of three-dimensional attitude data provided in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of a motion trajectory transfer system provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the motion trajectory migration system architecture provided in the embodiments of this disclosure; Figure 6 This is a schematic diagram of trajectory generalization operation provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram illustrating the robot's execution effect as provided in an embodiment of this disclosure. Detailed Implementation

[0019] This disclosure provides a method and system for motion trajectory migration. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0020] This disclosure provides a motion trajectory migration method, such as... Figure 1 As shown, it can be implemented as follows: S101. Semantic analysis is performed on the obtained motion segments representing the movement behavior of human body parts, and the reference nodes in the motion segments and the corresponding action trajectories of each reference node are determined based on the semantic information obtained from the analysis. S102. Determine trajectory constraint information based on each reference node and the corresponding motion trajectory of each reference node; S103. Based on the semantic information of the target task in the target task scenario, and with the trajectory constraint information as a constraint, construct the target action trajectory corresponding to the target task; S104. Using a preset trajectory optimization algorithm, the motion parameters representing the movement behavior of human body parts in the target motion trajectory are mapped to the motion parameters of the robot's skeletal structure to obtain the robot's motion trajectory.

[0021] In this embodiment, the motion segment can be a continuous sequence of video streams representing the movement behavior of human body parts. Each frame in the sequence can contain the three-dimensional coordinates of the human body part and the corresponding timestamp information. The human body part in the motion segment can be a normal human body part with a complete physiological shape; or it can be an abstract skeletal entity constructed according to the human physiological structure by multiple sets of joint nodes and corresponding connected bone segments. This entity can map the movement state and posture characteristics of the limbs in the human body based on the three-dimensional coordinates of the joint nodes and bone segments. Specifically, the motion segment here can be extracted from unstructured human operation videos; it can also be constructed and generated by 3D modeling software based on a human body model; or it can be extracted from digital humans or videos with humanoid biological characteristics generated based on video generation models. There are no limitations here.

[0022] These motion fragments can be input into a pre-built semantic parsing model. Based on the contextual understanding capabilities of the semantic parsing model, the semantics contained in the motion fragments are analyzed, and actions related to human body parts are extracted based on the semantics. This content can include actions performed by human body parts, such as approaching, grasping, manipulating, and placing. It can also include the human body parts performing these actions. For example, for grasping operations, it can include the contact point between the human body part and the object to be grasped and the corresponding three-dimensional coordinates; for placing operations, it can include the position where the human body part leaves contact with the object to be grasped (i.e., the placement point) and the three-dimensional coordinates of that position. Based on this extracted semantic information, the motion trajectory of each part of the human body when performing different actions can be obtained. Specifically, the motion trajectory refers to the trajectory of the human body part during the execution of a specific action. The positions of the human body parts performing key actions in these trajectories are reference nodes, such as the placement point and contact point mentioned above. For actions that do not come into contact with specific objects in the scene, the reference nodes can also include the start and end points of the human body part's movement in these trajectories, such as approaching or waving.

[0023] Based on the above, trajectory constraint information can be obtained. Trajectory constraint information refers to the features and conditions extracted from the action segment that guide the subsequent trajectory migration process, ensuring that the migrated trajectory maintains the action logic of the original trajectory and has physical smoothness. Reference nodes (i.e., the placement points and contact points mentioned above) are used to characterize the state that the trajectory needs to approximate at a specific time point. The action trajectory is used to preserve the overall motion trend corresponding to the reference nodes.

[0024] Furthermore, in order to obtain the trajectory when performing the same action in a new task scenario (i.e., the target task scenario), the semantic information of the target task scenario can be obtained first. This semantic information can be obtained from a textual description of the target task scenario (e.g., describing the location of the target task and obstacles in the task scenario); or it can be obtained by parsing the image of the target task scenario (e.g., RGB image or depth image) through a multimodal large model.

[0025] Based on the location of the target task as represented in the target task scenario, a trajectory for performing the action on the target task is recalculated using a trajectory adjustment algorithm based on a second-order spring-damped system. For example, for a grasping operation, the trajectory for grasping target objects of different positions and sizes in the target task scenario is calculated based on the trajectory of the human arm and hand represented in the action trajectory. During the calculation process, the spatial coordinates, acceleration, velocity, and other motion parameters of the human body parts when performing the action in the new target task scenario can be calculated in each specific cycle, based on the action trajectory and semantic information in the target task scenario (e.g., the position of the target object and the starting position of the action). Furthermore, based on the motion parameters of each cycle during the action execution, the action trajectory can be updated into a continuous target action trajectory based on the shape features of the original trajectory (i.e., the action trajectory).

[0026] This target motion trajectory is based on the human body structure. To enable this trajectory to guide the target robot, it's necessary to map different parts of the human body to the robot's skeletal structure. Based on this mapping, the motion parameters of the human body parts in each cycle of the target motion trajectory are converted into motion parameters of the robot's skeletal structure for that frame (e.g., joint angles, joint velocities). Then, based on the motion parameters contained in these frames, a complete robot motion trajectory can be obtained to guide the target robot in performing the corresponding actions, thus realizing the transfer of human motion trajectories to the target robot's motion trajectory.

[0027] In summary, the motion segments in this disclosure can be derived from various types of human videos, eliminating the need for complex drag-and-drop teaching or motion capture, thus improving efficiency; furthermore, the resulting robot motion trajectories are directly derived from human movements, conforming to natural human operating habits.

[0028] By acquiring trajectory constraint information, the action trajectory can be separated from the original scene, allowing the action trajectory to be generalized to different target tasks and to be adjusted to generate new trajectories, thus achieving flexible trajectory transfer.

[0029] When mapping motion parameters, the inherent kinematic and dynamic constraints of the robot are taken into account to ensure that the robot's motion trajectory after migration can be physically used for robot control, thus guaranteeing the stability of execution.

[0030] In another embodiment provided in this disclosure, the step S101 above, "performing semantic parsing on the obtained motion segments representing the movement behavior of human body parts, and determining each reference node in the motion segments and the corresponding action trajectory of each reference node based on the parsed semantic information," can be implemented as follows: S1011. The motion segment is parsed using a preset semantic parsing model to obtain the semantic information containing the motion segment describing the movement behavior of human body parts; S1012. Determine the spatial coordinates of the key locations representing the movement behavior of human body parts in the semantic information as reference nodes in the motion segment; S1013. The key action segments representing the movement behavior of human body parts in the semantic information are determined as the action trajectories corresponding to each reference node; the key action segments include the key positions.

[0031] In this embodiment of the disclosure, the preset semantic parsing model can be composed of a combination of a visual language model (e.g., LLaVA or Qwen-VL) and an object recognition algorithm (e.g., YOLO or Mask R-CNN).

[0032] For motion segments in the form of continuous video streams, object recognition algorithms can be used to identify and label the human body parts and motion-related manipulable objects (i.e., target objects) contained in each frame of the video stream. Then, the visual language model can receive the labeled video stream, combine the labeling information in each frame, perform cross-modal reasoning, and infer the entire motion process according to the temporal sequence of each frame, thereby outputting structured semantic information describing the motion process.

[0033] The above process will be illustrated with an example below. Suppose the motion segment is a video of "using the right hand to hold a cup on a table." An object recognition algorithm can identify and label the manipulable objects and human body parts within the motion segment. For example, labels such as "cup," "tabletop," "right arm," "right hand," and "torso" can be added to each frame. The visual language model receives each frame in the temporal sequence of the video stream and infers from temporal changes in the images (e.g., the coordinates of the part labeled "right hand" and the part labeled "cup" shrink), thus enabling the visual language model to deduce the essential content of the motion segment (i.e., the semantic information within the motion segment).

[0034] The output semantic information can be constrained into specific structured text through prompt word engineering. This text can include: time slices, representing the start and end frames of the movement of human body parts or operable objects in an action segment; action segment type, representing the operation stage of the action to which the action segment belongs, such as approach stage, operation / movement stage, release stage, etc.; contact state, representing the moving human body part (e.g., left and right hands) and the object in contact with that part; dominant part, representing the human body part that dominates the movement or exerts the main force in the current action segment (e.g., arm); operable object, representing the object that is directly interacted with in the current action segment (e.g., the cup mentioned above); scene object, representing the reference object used as a coordinate reference (e.g., the table mentioned above), used to define the relative displacement of the target position in the movement process; and position data, representing the spatial offset of the operable object or human body part relative to the reference object in different operation stages. This spatial offset can be output in the form of three-dimensional coordinates.

[0035] Furthermore, key position nodes (i.e., reference nodes) in the motion process can be extracted based on the semantic information mentioned above. For example, during the approach phase, when the hand moves towards the target task until the hand speed is zero, the spatial coordinates of the hand's position are the position of the grasping point; during the operation / movement phase, if there is a certain distance of relative displacement between the hand and the operable object, the position of the operable object is the position of the release point.

[0036] After obtaining the positions of these reference nodes, the changes in the spatial coordinates of all reference nodes in the entire motion segment can be correlated in time sequence to obtain the motion trajectory representing the movement behavior of human body parts in the motion segment.

[0037] In another embodiment provided in this disclosure, the step S102 above, "determining trajectory constraint information based on each reference node and the corresponding motion trajectory of each reference node," can be implemented as follows: Based on each reference node and the corresponding motion trajectory, the trajectory offset features under the corresponding motion progress of each reference node are determined; wherein, the trajectory offset features under each motion progress are used to characterize the shape features of the motion trajectory. In step S103 above, "constructing the target action trajectory corresponding to the target task based on the semantic information of the target task in the target task scenario and using the trajectory constraint information as constraints" can be implemented as follows: S1031. Based on the semantic information of the target scene, determine the key path locations of the target action to be performed in the target scene; S1032. For each key path location, determine the reference node corresponding to the movement progress of the key path location; determine the movement progress corresponding to the reference node as the movement progress corresponding to the key path location during the movement in the target scene; S1033. Using the trajectory offset features as constraints, determine the target motion information of each sampling point during the execution of the target action based on the motion progress corresponding to each key path position; S1034. Based on the target motion information, construct a target trajectory that represents the human body parts performing target actions in the target scene.

[0038] In this embodiment of the disclosure, motion progress refers to the degree of advancement of the movement of a human body part over time. Trajectory offset features refer to the actual motion state of the motion trajectory at a specific motion progress. By using the trajectory offset features at each motion progress, the geometric and kinematic morphology of the original motion trajectory in space can be outlined, and these morphologies can then be used to characterize the shape features of the motion trajectory.

[0039] The semantic information of the target scene can be obtained by parsing the new scene photograph using the aforementioned semantic parsing model, or by organizing the natural language description of the target scene using a large language model, ultimately yielding structured semantic information of the target scene. This information may include the positional and structural information of new operable objects in the target scene, as well as relevant information about scene objects used as references in the target scene.

[0040] Based on the semantic information mentioned above, the key path locations in the target scene related to the target task can be determined. Here, key path locations refer to the locations in the target scene where a part of the human body must achieve a specific spatial relationship (e.g., reaching or approaching) when performing the target action.

[0041] This allows mapping of reference nodes in the original motion trajectory to key path locations in the target scene. For example, the contact point for grabbing the cup in the original motion trajectory can be mapped to the location of a new operable object in the target scene. Furthermore, the motion progress corresponding to this reference node can be bound to the key path location, assigning it a specific execution sequence.

[0042] The key path locations, corresponding motion progress, and trajectory offset features can then be used as constraints within the kinematic solution framework. The key path locations and corresponding motion progress serve as hard constraints, while the trajectory offset features act as soft constraints. During the solution process, based on these constraints, the motion behavior in the target scene is guided to pass through the corresponding key path locations at specific motion progresses, and the target motion information, such as target position, target velocity, and target acceleration, is calculated for each time step (i.e., sampling point) during action execution. After calculating the target motion information for each time step, the target motion information can be stitched together according to the time sequence and smoothed to ultimately construct a complete target trajectory.

[0043] The kinematic solution framework here can be based on a second-order spring-damped system, or it can be a fractional-order spring-damped system or a nonlinear adaptive impedance system, or any other framework capable of performing kinematic solutions. There are no restrictions on this.

[0044] The above process will be explained below based on a second-order spring-damped system. Within the framework of a second-order spring-damped system, it can be explained by the dynamic equations. The solution is then performed. Among them, , and This refers to the position, acceleration, and velocity of human body parts in a specific dimension (e.g., the X, Y, or Z axis) when performing target motion in the target scene. and They are The first and second derivatives with respect to time. This is the target position, representing the critical path position of the target action, and the convergence position corresponding to each motion stage; under different motion stages... The positions can be sequentially aligned and switched to the key path positions, thereby guiding the dynamic equations to converge piecewise.

[0045] As the phase variable (i.e., the progress of motion), through the equation Calculated in During the monotonically decaying process from 1 to 0, the motion state of each sampling point is recorded, and the position is updated by integrating the velocity, thus obtaining a new trajectory converging from the starting point to the target position. For phase variables The first derivative with respect to time represents the rate at which the motion progresses; Used to control the convergence speed of the phase variable (i.e., how fast it decays from 1 to 0). The bigger The faster the decay, the more rapidly the movement progresses. For example, The value can range from 0.5 to 5.0.

[0046] The time scaling factor is used to control the rate of motion progression. Adjusting the time scaling factor can proportionally enlarge or reduce the execution time of the target action without changing the spatial shape characteristics of the trajectory. and The gain of the proportional-derivative (PD) controller determines the ratio of stiffness to damping in the spring system, thus controlling the current state. Towards The intensity of the convergence response. An example setting is... By maintaining this critical damping, it can be ensured that no oscillation occurs when the action reaches the target position, and the trajectory can monotonically approach the target.

[0047] Its function is to force the target trajectory to maintain the shape characteristics of the original motion trajectory, that is, the mathematical carrier of the aforementioned trajectory offset characteristics. In one possible implementation, It can be implemented as a forced function, and its formula can be: .in, The initial state, i.e., the initial position in the target scene when the target action is performed; through It can provide a spatial amplitude reference for the target motion, so that the trajectory offset will be scaled proportionally without shape distortion. The weighting parameters can be obtained by minimizing the local weighting error through locally weighted regression (LWR) on the original motion trajectory. Specifically, for each Gaussian function... Corresponding weight parameters Based on the actual motion information (e.g., position, velocity, acceleration, etc.) of the original motion trajectory at each motion progress x, the target coercive force at that moment can be calculated, and then the target coercive force can be applied to each Gaussian function. Construct the corresponding squared loss function, and solve for the one that minimizes the squared loss function. This allows us to determine the weight parameters corresponding to the basis function.

[0048] Furthermore, the normalized Gaussian function can be defined as follows: .in, The center point determines the activation center of the i-th basis function within the interval from motion progress 1 to 0. Distributing the phase variables at equal intervals within the range of motion progress can compensate for the nonlinear decay characteristics of the phase variables. For width parameter, , The activation interval width of the Gaussian function can be controlled; The spacing between different activation centers determines the density of the basis functions; It can be used to control the degree of overlap between adjacent basis functions to ensure that a sufficient number of basis functions are activated at any motion progress.

[0049] As can be seen from the above formula, in the early stage of the movement's progress, due to... Larger values ​​indicate higher trajectory offset characteristics. This can force the target trajectory to replicate its original motion trajectory. Towards the later stages of the movement, As the target trajectory approaches zero, it is guided to the target position under the control of the PD controller gain.

[0050] In another embodiment provided in this disclosure, step S104 above, "by using a preset trajectory optimization algorithm, mapping the motion parameters representing the movement behavior of human body parts in the target motion trajectory to the motion parameters of the robot's skeletal structure to obtain the robot's motion trajectory," can be implemented as follows: S1041. Based on the mapping model, map human body parts to execution nodes in the robot's skeletal structure; S1042. For the key execution nodes that perform the target action, under the first constraint condition, determine the trajectory tracking error between the motion parameters of the key execution nodes and the motion parameters of the corresponding human body parts; S1043. For each execution node in the robot's skeletal structure, under the second constraint, determine the motion similarity error between the motion parameters of each execution node of the robot and the motion parameters of the corresponding human body parts. S1044. Based on the trajectory tracking error and the action similarity error, the robot action trajectory is obtained by using a kinematic solver to guide the robot to perform actions according to the target action trajectory.

[0051] In this embodiment of the disclosure, the mapping model can refer to a pre-built mapping relationship between human body parts and robot skeletal structures. Since the degrees of freedom of movement and skeletal proportions of the human body and the robot are usually different, this mapping model can align them at various scales.

[0052] Based on this mapping model, key execution nodes (i.e., critical actuators) in the robot's skeletal structure that play a decisive role in completing the target action can be mapped to corresponding human body parts (e.g., in a grasping action, the key execution nodes are the robotic hand and elbow joints that grasp the object, mapped to the human hand and elbow). The first constraint condition includes hard constraints on the robot's key execution nodes (e.g., joint limit constraints, joint velocity constraints, and self-collision constraints) to ensure that the movement of the key execution nodes remains within the robot's mechanical safety boundaries. The first constraint condition can also include soft constraints, such as weighting coefficients used to constrain the tolerance of specific parts to trajectory tracking errors.

[0053] Tracking error The calculation can be performed based on the following formula (1).

[0054] (1); In the above formula (1), It refers to the set of correspondences between the execution nodes of the target robot and the human body parts. It refers to the set of key execution nodes of the robot required to execute the target motion trajectory. , These are the positions of the robot's current key execution nodes and the Cartesian coordinates of the corresponding human body parts in the target motion trajectory. , These are the rotation matrices for the robot's current key execution nodes and the corresponding human body parts, respectively.

[0055] and These are weighting coefficients that constrain the degree to which key execution nodes adhere to the target motion trajectory. For example, if you want the robot's manipulator to maintain consistency with the position and posture of the human skeleton, you can increase the weight at that point. If there are no special requirements for the position and posture of the robot's head, you can decrease the weight at that point, allowing for larger deviations. This prevents constraint failure caused by excessively high weights for all parts of the body. As an example, the weighting coefficient of the part you want to constrain can be about 10 times that of the part you don't want to constrain.

[0056] It should be noted that the rotation matrix of the human body part can be calculated based on the target motion information in the target motion trajectory. For example, other posture calculation methods such as the local coordinate system construction method based on adjacent key points can be used, and there are no restrictions here.

[0057] To address motion similarity errors, all skeletal structures of the robot can be mapped to human body parts. The second constraint includes hard constraints on all execution nodes of the robot, as well as weighting coefficients characterizing the tolerance for trajectory tracking errors at each part.

[0058] Action similarity error The calculation can be performed based on the following formula (2).

[0059] (2); in, and These are weighting coefficients different from those used for trajectory tracking errors. Using this motion similarity error formula, the set of mapped parts can be expanded to include all parts of the body, making the robot's overall motion more closely resemble human movements within the motion segment.

[0060] The joint velocities of the robot can then be solved using a kinematic solver, as shown in formula (3).

[0061] (3); in, It is the Jacobian matrix output by the kinematics solver, used to characterize the gradient of the loss function with respect to the generalized coordinates; This is the current error vector. ; This is the regularization coefficient used to penalize the robot's joint velocities; an example value is 1.0. For generalized speed.

[0062] In the calculation process of the above formula, by minimizing the trajectory tracking error and motion similarity error under the corresponding constraints, the optimal joint velocity that minimizes the objective function can be solved at each sampling point, and finally the result is obtained with joint angles... and joint velocity The robot's motion trajectory is represented.

[0063] It should be noted that if the joint angle obtained from the solution exceeds the constraint, it can be truncated to the maximum or minimum value allowed by the constraint. For example, in joint limit constraints. It can be extracted into or Joint velocity constraints It can be extracted into or Furthermore, for joint angles that may experience self-collision, a truncation operation can also be performed. The process of solving for joint angles and joint velocities under constraints described above can also employ solution frameworks such as Sequential Quadratic Programming (SQP) and Model Predictive Control (MPC), which are not limited here.

[0064] In yet another embodiment provided in this disclosure, the motion trajectory transfer method further includes the following steps: Step 1: Acquire video data containing human movements and determine the type of acquisition viewpoint; Step 2: Process the video data frame by frame, and extract the three-dimensional spatial information of each part of the human body contained in each frame image through a preset pose estimation model to obtain three-dimensional pose data representing the human body's motion state. Step 3: Determine the motion state of the video data acquisition device based on the acquisition viewpoint type corresponding to the video data; Step 4: Based on the motion state of the device, unify the three-dimensional posture data into the target coordinate system to obtain motion segments that characterize the motion behavior of various parts of the human body in the target coordinate system.

[0065] In this embodiment of the disclosure, the video data can be an unstructured video of human operation. Such videos can have different acquisition perspective types (e.g., a first-person moving perspective from a head-mounted camera or a third-person fixed perspective from a fixed camera), wherein the acquisition scenario of the video data can be as follows: Figure 2 As shown, Figure 2 The image on the right is a first-person moving view from a head-mounted camera; Figure 2 The left side shows a fixed camera view.

[0066] First, the acquired video data can be decoded and extracted frame by frame, and each extracted frame image can be standardized (e.g., cropping, scaling, and normalization). Then, the preprocessed frames can be input into a pre-defined pose estimation model. As an example, this pre-defined pose estimation model can be a pre-trained human pose estimation network (e.g., a pose estimation method based on a diffusion model, a single-frame regression-based HMR (Human Mesh Recovery) network, a VIBE (Video Inference for Human Body Pose and Shape Estimation) network incorporating temporal video features, or a METRO (MEsh Transformer) network based on a Transformer architecture). These networks can extract the spatial features of the images, extracting the spatial coordinates and pose of the major joints and bones of the human body from each frame image (e.g., the rotation angle of each joint and the position of each bone).

[0067] Furthermore, the extracted spatial coordinates and poses can be mapped to parameters in a specific format (e.g., a human skeleton representation in SMPL (Skinned Multi-Person Linear) format) as 3D pose data characterizing the human motion state. Depending on the different acquisition viewpoints and the movement of the video acquisition equipment during the acquisition process, the human pose can be transformed into a unified world coordinate system based on the aforementioned 3D pose data to eliminate the influence of the acquisition equipment's movement on the human actions in the motion segment.

[0068] like Figure 3 The image shown is a schematic diagram of 3D pose data and corresponding images. The left side shows the image extracted from the video data, and the right side shows the extracted 3D pose data.

[0069] In another embodiment provided in this disclosure, step 3 above, "determining the device motion state of the video data acquisition device according to the acquisition viewpoint type corresponding to the video data," can be implemented as follows: Step 1: When the acquisition viewpoint type is a moving viewpoint, align the pixel information contained in each frame of the video data with the inertial information contained in the video data according to the timestamp. Step 2: Between the timestamps corresponding to every two frames, perform pre-integration processing on the aligned inertial information to obtain inertial observation data characterizing the device motion constraints between the two frames; Step 3: Based on inertial observation data and corresponding pixel information, the spatial coordinates of the device in each frame in the target coordinate system are calculated using the visual inertial odometry method, thereby obtaining the device motion state that characterizes the device's activity trajectory in the video data. Step 4 above, "based on the motion state of the device, unify the three-dimensional posture data to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system," can be implemented as follows: Step 1: Convert the 3D pose data in the acquisition device coordinate system of each frame into 3D pose data in the target coordinate system based on the spatial coordinates of the device in that frame. Step 2: Based on the 3D pose data in the target coordinate system corresponding to each frame, obtain the motion segment in the target coordinate system corresponding to the video data.

[0070] In this embodiment of the disclosure, for moving perspectives where the spatial position of the video data acquisition device changes during video acquisition (e.g., shooting with a handheld smartphone, drone aerial photography, or head-mounted camera), pixel information and inertial information in the video data can be acquired. Pixel information refers to visual features such as feature points and edges contained in the image data of a video frame; inertial information refers to kinematic data (e.g., three-axis acceleration and angular velocity) acquired by the inertial measurement unit built into the video acquisition device.

[0071] When there is a difference in the sampling frequency between pixel information and inertial information, they can be aligned in time to eliminate errors caused by time asynchrony. Then, pre-integration processing can be performed based on the inertial information between two frames to calculate the motion increment of the video acquisition device between two frames (e.g., displacement increment, velocity increment, etc.). These motion increments constitute the physical constraints on the device motion between two frames.

[0072] Based on previously obtained pixel information and inertial observation data, visual inertial odometry (VIO) methods (such as Monocular Visual-Inertial Navigation System (VINS-Mono), Open Keyframe-based Visual-Inertial SLAM (OKVIS), and Multi-State Constraint Kalman Filter (MSCKF)) can be used. In this process, the relative pose of the acquisition device in each frame can first be calculated through triangulation based on the positions of edges and feature points in the pixel information, and then an initial device motion trajectory with the device's starting position as the origin can be stitched together. This initial trajectory can be re-estimated into a more continuous trajectory using angular velocity and acceleration data from the inertial observation data. VIO calculation can maintain a sliding window containing multiple consecutive frames, optimizing the entire trajectory within the window to ultimately obtain a continuous device activity trajectory.

[0073] The coordinate system of the acquisition device refers to a coordinate system based on the video data acquisition device. Its origin and coordinate axis directions change in real time as the device moves and rotates. Three-dimensional pose data describes the human body's motion state within this local coordinate system. Therefore, a coordinate system transformation matrix can be constructed based on the spatial coordinates of the device in that frame within the device's activity trajectory (e.g., using a quaternion transformation algorithm). This algorithm can transform the three-dimensional pose data in each frame, originally within the acquisition device's coordinate system, to a unified target coordinate system (i.e., the coordinate system of the device's activity trajectory).

[0074] After completing the coordinate system transformation of the single-frame attitude data, the corresponding 3D attitude data of each frame can be transformed to the target coordinate system to obtain continuous motion segments. During this process, smoothing filtering or interpolation can be performed to improve the continuity and smoothness of the motion segments.

[0075] In another embodiment provided in this disclosure, step 3 above, "determining the device motion state of the video data acquisition device according to the acquisition viewpoint type corresponding to the video data," can be implemented as follows: When the acquisition viewpoint type is fixed viewpoint, it is determined that the device is in a fixed state; Step 4 above, "based on the motion state of the device, unify the three-dimensional posture data to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system," can be implemented as follows: Step 1: With the device in a fixed state, using the device as the origin, convert the three-dimensional attitude data in the coordinate system of the acquisition device into three-dimensional attitude data in the target coordinate system. Step 2: Based on the 3D pose data in the target coordinate system corresponding to each frame, obtain the motion segment in the target coordinate system corresponding to the video data.

[0076] In this embodiment of the disclosure, for video data in which the spatial position of the video data acquisition device does not change during the video shooting process, a target coordinate system can be established with the device as the origin. The three-dimensional posture data can be mapped and transformed to the target coordinate system through a method similar to the coordinate system transformation described above, thereby obtaining the motion segment in the target coordinate system.

[0077] In another embodiment provided in this disclosure, the mapping model is constructed using the following steps: Step 1: Extract the structural parameters that define the skeletal structure of the target robot from the robot description file of the target robot; Step 2: Obtain the first semantic feature representing the movement function of human body parts and the second semantic feature representing the movement function of robot skeletal structure, and establish a mapping relationship between the human body parts and the robot skeletal structure based on the matching results of the first semantic feature and the second semantic feature. Step 3: Based on the structural parameters and the mapping relationship, construct a mapping model that represents the kinematic correspondence between the robot's skeletal structure and human body parts, so as to convert the motion trajectory of human body parts into the motion trajectory of the target robot's skeletal structure through the mapping model.

[0078] In this embodiment of the disclosure, structural parameters defining the robot's skeletal structure can be obtained by extracting the contents of the obtained unified robot description format file of the target robot.

[0079] Furthermore, the first semantic features contained in the description of the motion functions of human body parts can be extracted. For example, functions such as grasping and manipulating correspond to the human hand; the function of rotating the hand corresponds to the human wrist. By calculating feature similarity, the second semantic features of different robot parts can be matched. For example, the human hand is matched with a robotic hand that also has functions such as grasping and manipulating; the human wrist is matched with the wrist motor in the robot that drives the rotation of the robotic hand. Based on these mapping relationships, the human body parts with matching motion functions are bound to the robot's structure to obtain the corresponding mapping model.

[0080] Furthermore, since the limb structures of humans and robots are different, kinematic reversal can be performed (for example, by using inverse kinematics algorithms) to obtain a mapping model that includes the correspondence between human and robot structural functions and constraints.

[0081] In another embodiment provided in this disclosure, step 1 above, "extracting structural parameters defining the skeletal structure of the target robot from the robot description file of the target robot," can be implemented as follows: Step 1: Extract the motion attributes of each joint node and the motion constraint parameters of the activity boundary in the defined target robot skeleton structure; Step 2: Extract the physical properties and geometric parameters of the three-dimensional shape of each bone link in the defined target robot skeleton structure; Step 3: Extract the bone connection parameters that define the linkage relationship between bone links in the bone structure.

[0082] In this embodiment of the disclosure, motion constraint parameters may include the number of joint degrees of freedom of the robot, joint type, range of motion of each joint, maximum speed and acceleration limit, etc.; geometric parameters may include the mass and geometric dimensions of the robot's links, etc.; skeletal connection parameters may include the transformation matrix between the robot's links.

[0083] This disclosure has at least the following beneficial effects: Firstly, it offers flexible deployment and low cost: This invention supports single / multiple camera input as well as video file input, eliminating the need for expensive and complex multi-camera array systems and significantly reducing the requirements for acquisition sites and equipment costs. Videos captured by both fixed laboratory cameras and handheld devices can be used as input, greatly enhancing the flexibility and ease of use of the solution.

[0084] Secondly, the generated trajectory has high physical feasibility: Compared with existing mapping schemes, this disclosure forcibly introduces the robot's joint limits and velocity physical constraints during the trajectory generation process, which solves problems such as command over-limit and motion jamming. The generated trajectory can be directly deployed on real robots without secondary correction.

[0085] Third, it possesses trajectory generalization capabilities to support embodied intelligence applications: This disclosure achieves decoupling of trajectory shape and position and adaptive adjustment based on the target scene. The generated trajectory can be generalized to new target task scenarios such as changes in object position and adjustments in scene layout, while retaining the shape features of human action trajectories in the video, rather than simply reproducing a single action. This provides high-quality, transferable skill data support for robot skill learning and embodied intelligence applications.

[0086] Fourth, it has a high degree of automation: from video input to skill trajectory output, the whole process runs automatically without human intervention, which significantly improves the processing efficiency from human demonstration to robot skills.

[0087] This disclosure also provides a motion trajectory transfer system, such as Figure 4 As shown, it includes: The trajectory processing module 401 is used to perform semantic parsing on the acquired motion segments representing the movement behavior of human body parts, and determine each reference node and the corresponding action trajectory of each reference node in the motion segment based on the parsed semantic information; determine trajectory constraint information based on each reference node and the corresponding action trajectory of each reference node; and construct the target action trajectory corresponding to the target task based on the semantic information of the target task in the target task scene and the trajectory constraint information. The motion redirection module 402 is used to map the motion parameters representing the movement behavior of human body parts in the target motion trajectory to the motion parameters of the robot's skeletal structure through a preset trajectory optimization algorithm, thereby obtaining the robot's motion trajectory.

[0088] In another embodiment provided in this disclosure, the trajectory processing module 401 is used to parse the motion segment through a preset semantic parsing model to obtain semantic information containing the motion segment describing the movement behavior of human body parts; determine the spatial coordinates of key positions representing the movement behavior of human body parts in the semantic information as reference nodes in the motion segment; determine key action segments representing the movement behavior of human body parts in the semantic information as action trajectories corresponding to each reference node; the key action segments include the key positions.

[0089] In another embodiment provided in this disclosure, the trajectory processing module 401 is used to determine the trajectory offset features under the motion progress corresponding to each reference node based on each reference node and the motion trajectory corresponding to each reference node; wherein, the trajectory offset features under each motion progress are used to characterize the shape features of the motion trajectory. The trajectory processing module 401 is used to determine the key path positions of the target action to be performed in the target scene based on the semantic information of the target scene; for each key path position, determine the reference node corresponding to the movement progress of the key path position; determine the movement progress corresponding to the reference node as the movement progress corresponding to the key path position during the movement in the target scene; determine the target motion information of each sampling point during the execution of the target action based on the movement progress corresponding to each key path position, using the trajectory offset feature as a constraint; and construct the target trajectory representing the human body part performing the target action in the target scene based on the target motion information.

[0090] In another embodiment provided in this disclosure, the motion redirection module 402 is used to map human body parts to execution nodes in the robot's skeletal structure based on the mapping model; for key execution nodes performing the target action, under a first constraint, determine the trajectory tracking error between the motion parameters of the key execution nodes and the motion parameters of the corresponding human body parts; for each execution node in the robot's skeletal structure, under a second constraint, determine the motion similarity error between the motion parameters of each execution node of the robot and the motion parameters of the corresponding human body parts; based on the trajectory tracking error and the motion similarity error, a kinematic solver is used to solve for the robot motion trajectory used to guide the robot to perform actions according to the target motion trajectory.

[0091] In another embodiment provided in this disclosure, the motion trajectory migration system further includes: a video parsing module 403, used to acquire video data containing human motion and determine the acquisition viewpoint type; to process the video data frame by frame, and to extract the three-dimensional spatial information of each part of the human body contained in each frame image through a preset pose estimation model to obtain three-dimensional pose data representing the human motion state; to determine the device motion state of the video data acquisition device according to the acquisition viewpoint type corresponding to the video data; and to unify the three-dimensional pose data to the target coordinate system based on the device motion state to obtain motion segments representing the motion behavior of each part of the human body in the target coordinate system.

[0092] In another embodiment provided in this disclosure, the video parsing module 403 is used to align the pixel information contained in each frame of the video data with the inertial information contained in the video data according to the timestamp when the acquisition viewpoint type is a moving viewpoint; between the timestamps corresponding to every two frames, the aligned inertial information is pre-integrated to obtain inertial observation data characterizing the device motion constraint between two frames; based on the inertial observation data and the corresponding pixel information, the spatial coordinates of the device in each frame in the target coordinate system are calculated by the visual inertial odometry method to obtain the device motion state characterizing the device activity trajectory in the video data; The video parsing module 403 is used to convert the three-dimensional pose data in the acquisition device coordinate system of each frame into three-dimensional pose data in the target coordinate system based on the spatial coordinates of the device in that frame; and to obtain the motion segment in the target coordinate system corresponding to the video data based on the three-dimensional pose data in the target coordinate system corresponding to each frame.

[0093] In another embodiment provided in this disclosure, the video parsing module 403 is used to determine that the device is in a fixed state when the acquisition viewpoint type is a fixed viewpoint. The video parsing module 403 is used to convert the three-dimensional posture data in the acquisition device coordinate system into three-dimensional posture data in the target coordinate system, with the device as the origin, when the device is in a fixed state; and to obtain the motion segment in the target coordinate system corresponding to the video data based on the three-dimensional posture data in the target coordinate system corresponding to each frame.

[0094] In another embodiment provided in this disclosure, the motion trajectory transfer system further includes: a kinematic mapping module 404, used to extract structural parameters defining the skeletal structure of the target robot from the robot description file of the target robot; obtain a first semantic feature representing the motion function of a human body part and a second semantic feature representing the motion function of the robot skeletal structure, and establish a mapping relationship between the human body part and the robot skeletal structure based on the matching result of the first semantic feature and the second semantic feature; and construct a mapping model representing the kinematic correspondence between the robot skeletal structure and the human body part according to the structural parameters and the mapping relationship, so as to convert the motion trajectory of the human body part into the motion trajectory of the skeletal structure of the target robot through the mapping model.

[0095] In another embodiment provided in this disclosure, the kinematic mapping module 404 is used to extract the kinematic attributes and motion constraint parameters of each joint node in the target robot skeletal structure; extract the physical attributes and geometric parameters of each bone link in the target robot skeletal structure; and extract the bone connection parameters that define the linkage relationship between the bone links in the skeletal structure.

[0096] The following section describes a possible motion trajectory transfer system architecture, such as... Figure 5 As shown, this system can be integrated into platforms including robot-embedded computing processors, workstations, or personal computers; there are no limitations. The human operation video data to be processed can come from pre-prepared video files or data captured by real-time cameras. This data is input into the video parsing module to obtain motion segments containing human body movement behaviors. The trajectory processing module receives these motion segments and performs semantic parsing and scene transfer generalization operations. Specifically, it can be as follows... Figure 6 The process includes operations such as trajectory rotation and scaling. During these operations, constraints can be applied based on trajectory constraint information to ensure that the target motion trajectory does not deviate from the shape characteristics of the original trajectory. After these operations, a target motion trajectory adapted to the new target task scenario can be generated. The motion retargeting module then receives the target motion trajectory and solves for a robot motion trajectory that satisfies all robot physical constraints based on a mapping model. This mapping model comes from the kinematic mapping module, which configures the mapping relationship between the human body and the robot based on a unified robot description format file. Finally, the system outputs the optimized robot motion trajectory, which the user can directly send to the robot controller for execution or verify in a simulation environment. The robot execution effect can be as follows: Figure 7 As shown, Figure 7 The image on the left is a frame from a video showing the object being placed in a bag, and the image on the right is the execution of the robot's motion trajectory based on that video, corresponding to that frame.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0098] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0099] Those skilled in the art will understand that the modules in the system of the embodiments can be distributed in the system of the embodiments as described in the embodiments, or they can be located in one or more systems different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0100] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for transferring motion trajectories, characterized in that, include: Semantic parsing is performed on the obtained motion segments representing the movement behavior of human body parts, and the reference nodes in the motion segments and the corresponding action trajectories of each reference node are determined based on the semantic information obtained from the parsing. The trajectory constraint information is determined based on each reference node and the corresponding motion trajectory of each reference node. Based on the semantic information of the target task in the target task scenario, and with the trajectory constraint information as constraints, the target action trajectory corresponding to the target task is constructed. By using a preset trajectory optimization algorithm, the motion parameters representing the movement behavior of human body parts in the target motion trajectory are mapped to the motion parameters of the robot's skeletal structure, thus obtaining the robot's motion trajectory.

2. The method as described in claim 1, characterized in that, Semantic parsing is performed on the acquired motion segments representing the movement behavior of human body parts, and the reference nodes and corresponding action trajectories in the motion segments are determined based on the parsed semantic information, including: The motion segment is parsed using a preset semantic parsing model to obtain the semantic information contained in the motion segment describing the movement behavior of human body parts; The spatial coordinates of key locations representing the movement behavior of human body parts in the semantic information are determined as reference nodes in the motion segment; The key action segments representing the movement behavior of human body parts in the semantic information are determined as the action trajectories corresponding to each reference node; the key action segments include the key positions.

3. The method as described in claim 1, characterized in that, Trajectory constraint information is determined based on each reference node and the corresponding motion trajectory, including: Based on each reference node and the corresponding motion trajectory, the trajectory offset features under the corresponding motion progress of each reference node are determined; wherein, the trajectory offset features under each motion progress are used to characterize the shape features of the motion trajectory. Based on the semantic information of the target task in the target task scenario, and using the trajectory constraint information as constraints, the target action trajectory corresponding to the target task is constructed, including: Based on the semantic information of the target scene, determine the key path locations of the target action to be performed in the target scene; For each critical path location, a reference node corresponding to the movement progress of that critical path location is determined; the movement progress corresponding to the reference node is determined as the movement progress of that critical path location during the movement within the target scene. Using the trajectory offset features as constraints, the target motion information of each sampling point during the execution of the target action is determined based on the motion progress corresponding to each key path position; Based on the target motion information, a target trajectory representing the human body parts performing target actions in the target scene is constructed.

4. The method as described in claim 1, characterized in that, Using a preset trajectory optimization algorithm, the motion parameters representing the movement behavior of human body parts in the target motion trajectory are mapped to the motion parameters of the robot's skeletal structure, resulting in the robot's motion trajectory, including: Based on the mapping model, human body parts are mapped to execution nodes in the robot's skeletal structure; For key execution nodes that perform the target action, under the first constraint condition, determine the trajectory tracking error between the motion parameters of the key execution nodes and the motion parameters of the corresponding human body parts; For each execution node in the robot's skeletal structure, under the second constraint, determine the motion similarity error between the motion parameters of each execution node of the robot and the motion parameters of the corresponding human body parts; Based on the trajectory tracking error and the motion similarity error, the robot motion trajectory, which guides the robot to perform actions according to the target motion trajectory, is obtained by using a kinematic solver.

5. The method as described in claim 1, characterized in that, Also includes: Acquire video data containing human movements and determine the type of acquisition viewpoint; The video data is processed frame by frame, and the three-dimensional spatial information of each part of the human body contained in each frame image is extracted by a preset pose estimation model to obtain three-dimensional pose data representing the human body's motion state. The motion state of the video data acquisition device is determined based on the acquisition viewpoint type corresponding to the video data. Based on the device's motion state, the three-dimensional posture data is unified into the target coordinate system to obtain motion segments that characterize the motion behavior of various parts of the human body in the target coordinate system.

6. The method as described in claim 5, characterized in that, Based on the acquisition viewpoint type corresponding to the video data, determine the device motion state of the video data acquisition device, including: When the acquisition viewpoint type is a moving viewpoint, the pixel information contained in each frame of the video data is aligned with the inertial information contained in the video data according to the timestamp. Between the timestamps of every two frames, the aligned inertial information is pre-integrated to obtain inertial observation data characterizing the device motion constraints between the two frames. Based on inertial observation data and corresponding pixel information, the spatial coordinates of the device in each frame in the target coordinate system are calculated using the visual inertial odometry method, thereby obtaining the device motion state that characterizes the device's activity trajectory in the video data. Based on the device's motion state, the three-dimensional posture data is unified to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system, including: The 3D pose data in each frame, which is in the coordinate system of the acquisition device, is converted into 3D pose data in the target coordinate system based on the spatial coordinates of the device in that frame. Based on the 3D pose data in the target coordinate system corresponding to each frame, motion segments in the target coordinate system corresponding to the video data are obtained.

7. The method as described in claim 5, characterized in that, Based on the acquisition viewpoint type corresponding to the video data, determine the device motion state of the video data acquisition device, including: When the acquisition viewpoint type is fixed viewpoint, it is determined that the device is in a fixed state; Based on the device's motion state, the three-dimensional posture data is unified to the target coordinate system to obtain motion segments representing the motion behavior of various parts of the human body in the target coordinate system, including: With the device in a fixed state, the three-dimensional attitude data in the coordinate system of the acquisition device is converted into three-dimensional attitude data in the target coordinate system, with the device as the origin. Based on the 3D pose data in the target coordinate system corresponding to each frame, motion segments in the target coordinate system corresponding to the video data are obtained.

8. The method as described in claim 4, characterized in that, The mapping model is constructed using the following method: Extract the structural parameters that define the skeletal structure of the target robot from the robot description file; First semantic features representing the movement function of human body parts and second semantic features representing the movement function of robot skeletal structure are obtained respectively. Based on the matching results of the first semantic features and the second semantic features, a mapping relationship is established between the human body parts and the robot skeletal structure. Based on the structural parameters and the mapping relationship, a mapping model is constructed to characterize the kinematic correspondence between the robot's skeletal structure and human body parts, so as to convert the motion trajectory of human body parts into the motion trajectory of the target robot's skeletal structure through the mapping model.

9. The method as described in claim 8, characterized in that, Extract the structural parameters that define the skeletal structure of the target robot from the robot description file, including: Extract the motion attributes and motion constraint parameters of each joint node in the target robot's skeletal structure; Extract the physical properties and geometric parameters of the three-dimensional shape of each bone link in the target robot's skeletal structure; Extract the bone connection parameters that define the linkage relationship between bone links in the bone structure.

10. A motion trajectory transfer system, characterized in that, include: The trajectory processing module is used to perform semantic parsing on the acquired motion segments representing the movement behavior of human body parts, and to determine each reference node and the corresponding action trajectory in the motion segment based on the parsed semantic information; and to determine trajectory constraint information based on each reference node and the corresponding action trajectory. Based on the semantic information of the target task in the target task scenario, and with the trajectory constraint information as constraints, the target action trajectory corresponding to the target task is constructed. The motion redirection module is used to map the motion parameters representing the movement behavior of human body parts in the target motion trajectory to the motion parameters of the robot's skeletal structure through a preset trajectory optimization algorithm, thereby obtaining the robot's motion trajectory.

Images