Track diffusion imitation learning method and system based on single human video demonstration

Abstract

The invention relates to the technical field of intelligent equipment. A trajectory diffusion imitation learning method based on single human video demonstration comprises the steps that a demonstration video is analyzed to obtain an operation body posture motion trajectory and an object pose motion trajectory, and the demonstration video is a video of a human or machine operation object; respectively carrying out interpolation calculation on the positions and postures of the operation body posture motion trail and the object pose motion trail to obtain a training data set; using the training data set to train a diffusion strategy model to obtain a target diffusion strategy model; and during deployment, the real-time pose of an end effector of the robot serves as observation input of the target diffusion type strategy model, an operation body pose movement track and an object pose movement track in the deployment stage are formed, and a joint control instruction of a target operation body is obtained. And the task success rate is kept, and the practicability and the expandability of the imitation learning method are improved at the same time.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and more specifically, to a trajectory diffusion imitation learning method and system based on a single human video demonstration. Background Technology

[0002] In the field of embodied intelligence, imitation learning is an important method. It trains policy models by recording and learning the motion trajectories demonstrated by humans or robots, enabling intelligent agents to autonomously complete similar tasks. However, imitation learning requires a large amount of data for training. Currently, imitation learning generally relies on teleoperation systems to collect demonstration data, i.e., humans operate robots to complete tasks and record the motion processes. Although this method is precise and controllable, it has obvious drawbacks: First, data collection is costly, each demonstration requires manual operation, resulting in low efficiency and difficulty in scaling to large-scale tasks or diverse scenarios; second, it depends on specific hardware platforms and lacks generalization capabilities across robots and scenarios; finally, because teleoperation is usually carried out in ideal environments, the learned policies are less robust to complex visual changes in the real world and struggle to cope with changes in perspective, background, or lighting conditions.

[0003] Therefore, how to lower the threshold for data collection and improve the generalization ability of learning strategies has become the core issue that imitation learning urgently needs to address. Summary of the Invention

[0004] The main objective of this application is to provide a trajectory diffusion imitation learning method and system based on a single human video demonstration, in order to solve the aforementioned technical problems.

[0005] To achieve the above objectives, the first aspect of this application proposes a trajectory diffusion imitation learning method based on a single human video demonstration, comprising: The demonstration video is analyzed to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position. The demonstration video is a video of a human or machine operating an object. The position and orientation of the manipulator's posture motion trajectory and the object's pose motion trajectory are interpolated to obtain a training dataset. Using the training dataset, the diffusion policy model is trained to obtain the target diffusion policy model; During deployment, the real-time pose of the robot's end effector is used as the observation input to the target diffusion strategy model to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase, thereby obtaining the joint control commands of the target manipulator.

[0006] In some feasible methods, the step of analyzing the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position includes: Identify the task types in the demonstration video, wherein the task types include operator tasks and object tasks; When the task type is the manipulator task, the manipulator posture features are extracted in the demonstration video using a 3D reconstruction model to obtain the manipulator posture motion trajectory, wherein the manipulator posture motion trajectory represents the remapped posture calculated based on the pose relationship of the manipulator joints. When the task type is the object task, the object pose feature is extracted in the demonstration video using the object 6D pose estimation model to obtain the object pose motion trajectory, wherein the object pose motion trajectory includes the object's 6D pose information.

[0007] In some feasible methods, the step of analyzing the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position includes: When the manipulator is the robot's hand: When the task type is the operation body task, the HaMeR model is used to extract features of the key joint postures of the hand in the demonstration video to obtain the hand posture motion trajectory. The key joints of the hand include the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger. The remapping pose is calculated based on the pose relationship of the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger. When the task type is the object task, the FoundationPose model is used to extract features of the 6D pose of the object in the demonstration video to obtain the object's pose motion trajectory. In the demonstration video, in the keyframe when the object is successfully grasped, the hand pose is extracted using the HaMeR model, and the hand gripper pose executed by the robot is obtained based on the relative transformation relationship between the object pose and the hand pose, as well as the remapping pose of the key joints of the hand.

[0008] In some feasible methods, the step of interpolating the position and orientation of the manipulator's pose trajectory and the object's pose trajectory to obtain a training dataset includes: The positional portions of the manipulator's posture motion trajectory and the object's pose motion trajectory are interpolated using the warp interpolation method to obtain the interpolated positional trajectories of the manipulator and the object. The attitude portion of the motion trajectory of the manipulator and the pose of the object is interpolated using the spherical linear interpolation method to obtain the interpolated attitude trajectory of the manipulator and the attitude trajectory of the object. The interpolated position trajectories of the manipulator and the object, as well as the interpolated pose trajectories of the manipulator and the object, are combined to obtain the training dataset.

[0009] In some feasible methods, the step of training the diffusion policy model using the training dataset to obtain the target diffusion policy model includes: The pose trajectory sequence in the training dataset is used as training samples and input into the diffusion-type Transformer policy model for training. Through the back diffusion process, the diffusion-type Transformer policy model is trained to learn the denoising mapping from noise to the target pose trajectory, thus obtaining the target diffusion-type policy model. The pose trajectory sequence represents the sequence formed by the interpolated manipulator posture motion trajectory and the object pose motion trajectory in time. The target diffusion-type policy model can generate continuous motion trajectories based on real-time observations.

[0010] In some feasible methods, the step of training the diffusion policy model using the training dataset to obtain the target diffusion policy model further includes: For manipulation tasks involving multiple sub-stages, during the training phase of the diffusion-type Transformer policy model, a training phase switching policy network is used to obtain a target phase switching policy network. The phase switching policy network represents the identification of the completion status of each sub-stage in the manipulation task based on the real-time pose of the object and the state of the robot's end effector, and triggers the transition between stages to form a phase switching decision signal.

[0011] In some feasible methods, the step of obtaining joint control commands for the target manipulator by using the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model during deployment to form the manipulator's posture motion trajectory and the object's pose motion trajectory during the deployment phase includes: The poses of the robot's end effector and objects in the environment are acquired in real time and input into the target diffusion strategy model. The target diffusion strategy model is then used for processing to generate the motion trajectory of the target limb posture and the motion trajectory of the target object posture. Based on the target limb posture motion trajectory and the target object pose motion trajectory, the target limb posture motion trajectory is converted into joint control commands using robot inverse kinematics calculation, thereby obtaining the joint control commands for the target manipulator.

[0012] In some feasible methods, the step of obtaining joint control commands for the target manipulator by using the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model during deployment to form the manipulator's posture motion trajectory and the object's pose motion trajectory during the deployment phase, further includes: When deploying the current manipulation task, the pose of the robot end effector and the pose of objects in the environment are acquired in real time and input into the target diffusion strategy model and the target stage switching strategy network. The target diffusion strategy model and the target stage switching strategy network are used to process them respectively, and the target limb posture motion trajectory and the target object posture motion trajectory are generated accordingly, as well as whether the current manipulation task execution stage needs to be switched, forming the stage switching decision signal. Based on the stage switching decision signal, the target limb posture motion trajectory and the target object pose motion trajectory, the target limb posture motion trajectory is converted into joint control commands using robot inverse kinematics calculation, thus obtaining the joint control commands for the target manipulator.

[0013] In some feasible methods, the step of converting the target limb posture trajectory into joint control commands based on the stage switching decision signal, the target limb posture trajectory, and the target object pose trajectory, and using robot inverse kinematics calculation to obtain the joint control commands of the target manipulator includes: When the phase switching decision signal indicates that a phase switching is required, the target limb posture motion trajectory and the target object pose motion trajectory are regenerated based on the new phase target using the target diffusion strategy model. When the stage switching decision signal indicates that the current stage should be maintained, the target diffusion strategy model continuously generates the target limb posture motion trajectory and the target object pose motion trajectory based on the current manipulation task. Based on the target limb posture motion trajectory and the target object pose motion trajectory, the robot inverse kinematics calculation is used to convert the target limb posture motion trajectory into joint control commands, thereby obtaining the joint control commands for the target manipulator.

[0014] Secondly, this application provides a trajectory diffusion imitation learning system based on a single human video demonstration, applied to the aforementioned trajectory diffusion imitation learning method based on a single human video demonstration, including: The video analysis module is used to analyze the demonstration video to obtain the motion trajectory of the operator and the motion trajectory of the object's position. The demonstration video is a video of a human or machine operating an object. The trajectory generation module is used to interpolate the position and orientation of the manipulator's posture motion trajectory and the object's pose motion trajectory to obtain a training dataset. The trajectory generation module is also used to train the diffusion policy model using the training dataset to obtain the target diffusion policy model. The closed-loop execution module is used to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase by using the real-time pose of the robot's end effector as the observation input of the target diffusion strategy model, thereby obtaining the joint control commands of the target manipulator.

[0015] The technical solutions provided by the embodiments of this application may include the following beneficial effects: This application presents a trajectory diffusion imitation learning method based on a single human video demonstration. By using video analysis to replace robot teleoperation, the data acquisition threshold is lowered. The intermediate representation and enhancement methods of pose trajectory improve the cross-scene adaptability of the strategy, while the pose space decision mechanism in the deployment stage reduces the sensitivity to changes in the visual environment. Thus, while maintaining the task success rate, the practicality and scalability of the imitation learning method are improved. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart of a trajectory diffusion imitation learning method based on a single human video demonstration provided in this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0020] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0021] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] The following explanations of some terms used in this application are provided to aid in understanding the application: FoundationPose is an advanced model for 6D object pose estimation. 6D pose refers to an object's position (X, Y, Z coordinates) and orientation (rotation around the three axes) in 3D space, totaling six degrees of freedom. It can accurately estimate the 6D pose of a specific object in a scene based on RGB-D (color image + depth information) input. It exhibits strong generalization ability even when the object is partially occluded or viewed from a new perspective.

[0023] The HaMeR model is a Transformer-based 3D mesh reconstruction model of the hand. It can reconstruct a complete 3D hand model containing 21 joints from a single RGB image with high accuracy. The model can not only estimate the overall position and orientation of the hand in 3D space, but also accurately output the bending angle and posture of each finger joint, thus reproducing the complex movements and gestures of the hand in detail.

[0024] Remapping refers to the calculation of a new, standardized pose representation based on the geometric relationships between the joints of a manipulator (hand or object). The purpose of remapping is not to directly use the original hand pose, but to construct a new, stable local coordinate system through several key joints.

[0025] Warp interpolation is an interpolation method used to process three-dimensional spatial trajectory position data. Its core idea is to make the entire trajectory path smoothly adapt to new starting conditions by "bending" or "deforming" it.

[0026] Spherical linear interpolation is a classic algorithm used to smoothly interpolate two directions (or orientation) in three-dimensional space. Spherical linear interpolation guarantees the shortest and smoothest path on a quaternion-represented sphere of rotation.

[0027] like Figure 1 As shown, in a first aspect, this application provides a trajectory diffusion imitation learning method based on a single human video demonstration, comprising: S100 analyzes the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position.

[0028] The demonstration video is a video of a human or machine operating an object.

[0029] It should be noted that the purpose of step S100 is to extract the spatial motion information most relevant to task execution from a dynamic video, providing a data foundation for subsequent robot imitation learning.

[0030] Specifically, obtaining the motion trajectory of the manipulator and the motion trajectory of the object's pose may include the following steps: S101, Identify the task type in the demonstration video.

[0031] The task types include operation body tasks and object tasks.

[0032] Specifically, features are extracted from the demonstration video, and the extracted features are used to distinguish between the manipulated entity and the object. Then, different tasks are created for each of the extracted manipulated entity and object, and these tasks are processed separately. For feature recognition and extraction from the demonstration video, methods such as convolutional neural networks can be used.

[0033] S102, when the task type is the operation body task, the three-dimensional reconstruction model is used to extract features of the operation body posture in the demonstration video to obtain the operation body posture motion trajectory.

[0034] The motion trajectory of the manipulator represents the remapped pose calculated based on the pose relationship of the manipulator joints.

[0035] Specifically, in step S102, the 3D reconstruction model is used to extract features of the operator's posture in the demonstration video. For example, for each frame of the demonstration video, the operator is identified and the 3D coordinates of all its joints are estimated, thereby constructing a 3D model of the operator in each frame.

[0036] S103, when the task type is the object task, the object pose feature is extracted in the demonstration video using the object 6D pose estimation model to obtain the object pose motion trajectory.

[0037] The object's pose trajectory includes the object's 6D pose information.

[0038] Specifically, for object-oriented tasks, a 6D pose estimation model is used for each frame to identify the target object and estimate its 6D pose. The 6D pose includes the object's position (X, Y, Z coordinates) and rotational orientation (angles of rotation around the X, Y, Z axes, typically represented by quaternions or rotation matrices) in three-dimensional space. The 6D pose describes the object's "position and orientation" in each frame. Next, the estimated 6D poses of each frame are concatenated in chronological order to obtain the precise object's motion trajectory. This trajectory clearly demonstrates how the object is moved and rotated to ultimately complete the task.

[0039] In one embodiment, the step of parsing the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position further includes: When the manipulator is the robot's hand: S1011, when the task type is the operation body task, the HaMeR model is used to extract features of the key joint postures of the hand in the demonstration video to obtain the hand posture motion trajectory.

[0040] The key joints of the hand include the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger. The remapping pose is calculated based on the pose relationship between the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger.

[0041] Specifically, when the manipulator is a robot hand, the motion trajectory of key joints of the hand can be extracted from the demonstration video using the HaMeR hand 3D reconstruction model, so as to recover the 3D motion information of the hand from the 2D demonstration video sequence.

[0042] Key joint selection: The thumb metacarpophalangeal joint, thumb interphalangeal joint, and index finger metacarpophalangeal joint were chosen as key points because these three joints uniquely determine the hand's grasping posture and operational intent. These three non-collinear key points establish a stable local coordinate system, eliminating the influence of individual hand size differences.

[0043] Principle of remapping pose calculation: ; ; ; in, Represents the three-dimensional coordinates of the thumb metacarpophalangeal joint. Represents the three-dimensional coordinates of the thumb interphalangeal joint. Represents the three-dimensional coordinates of the metacarpophalangeal joint of the index finger; The X-axis basis vector is generated by the cross product of the thumb pointing direction (the vector from the thumb metacarpophalangeal joint to the interphalangeal joint) and the thumb-index finger direction (the vector from the thumb metacarpophalangeal joint to the index finger metacarpophalangeal joint). The X-axis is roughly along the hand grasping direction. This means that the X-axis basis vector is cross-producted with the vector pointing from the metacarpophalangeal joint of the index finger to the metacarpophalangeal joint of the thumb, generating the Z-axis basis vector. The Z-axis is perpendicular to the hand plane and points towards the back of the hand. This means that the Y-axis basis vector is generated by the cross product of the Z and X axes, ensuring that the coordinate system conforms to the right-hand rule, with the Y-axis along the width of the hand. The formula constructs a stable local coordinate system through three non-collinear key joints. Remapping the pose standardizes hand pose data under different hand sizes and viewpoints, thereby eliminating the influence of individual differences and viewpoint variations, and providing a foundation for subsequent trajectory generalization. , , These are three orthogonal basis vectors of the local coordinate system, used to construct the remapping pose. (A 4x4 transformation matrix) maps the hand pose uniformly to this local coordinate system.

[0044] Next, the key point detection and remapping calculations are repeated for each frame in the demonstration video, and the hand poses of each frame are connected in chronological order to form a continuous hand pose motion trajectory.

[0045] By calculating the remapped pose, the consistency of the trajectory under different hand sizes and viewpoints is ensured, so as to have generalization ability.

[0046] S1012, when the task type is the object task, the FoundationPose model is used to extract features of the 6D pose of the object in the demonstration video to obtain the object's pose motion trajectory.

[0047] Specifically, when the task type is an object task, the FoundationPose 6D object pose estimation model is used to extract the complete motion trajectory of the object from the demonstration video. 6D pose contains the object's position and orientation information in three-dimensional space, which can completely describe the object's motion state.

[0048] It should be noted that the 6D pose estimation can be performed using the FoundationPose model, which estimates the six-DOF pose of the object at each time step from the demonstration video sequence. The position component describes the object's translational motion, and the pose component describes the object's rotational state; the combination of both fully characterizes the object's kinematic features. Next, the estimated 6D poses of the object from each frame are concatenated in temporal order to form the object's trajectory throughout the entire task execution process. This trajectory reflects the object's state changes during the operation, providing a data foundation for understanding the operational logic.

[0049] S1013, in the demonstration video, in the keyframe when the object is successfully grasped, the hand pose is extracted using the HaMeR model, and the hand gripper pose executed by the robot is obtained based on the relative transformation relationship between the object pose and the hand pose and the remapping pose of the key joints of the hand.

[0050] Specifically, in the keyframe where the object is successfully grasped, the gripper target pose required for robot execution is calculated by comprehensively utilizing hand pose, object pose, and remapped pose information.

[0051] For example, a keyframe for successful grasping refers to the moment when a stable contact relationship is established between the hand and the object. Furthermore, when the shortest distance between the hand and the object is consistently below a set threshold, it is determined as a keyframe indicating successful grasping of the object. At this point, the relative pose of the hand and the object contains important semantic information, reflecting an effective grasping configuration. Based on the spatial transformation relationship between the hand pose and the object pose, the hand pose relative to the object is calculated. This transformation relationship not only captures the characteristics of the grasping operation but also reflects the geometric constraints of hand-object interaction.

[0052] By combining the remapped pose (representing the internal joint relationships of the hand) with the relative transformation relationship between the hand and the object, a suitable gripper target pose for robot execution is calculated. Specifically, this is achieved through the following steps: Based on the 3D coordinates of the key hand joints output by the HaMeR model, the hand pose is then applied using the remapping pose transformation matrix. The hand pose is standardized to a standard coordinate system to obtain the standardized hand pose. Based on the standardized hand pose and the object pose estimated by the FoundationPose model, the spatial transformation relationship is calculated to obtain the hand-object relative transformation matrix. This transformation matrix captures the essential geometric relationship of the grasping operation and reflects the spatial constraints of how the hand grasps the object. Based on the relative transformation matrix between the hand and the object, and combined with the robot's kinematic parameters, the target pose of the robot's gripper is calculated through coordinate transformation and feasibility verification, thereby generating the hand gripper pose executed by the robot.

[0053] S200, interpolation calculations are performed on the position and orientation of the operation body's posture motion trajectory and the object's pose motion trajectory to obtain a training dataset.

[0054] This involves randomly sampling the effective initial pose of the target object within the operating space to construct a large-scale, diverse training dataset. This is done to achieve the original pose trajectory. With generating pose trajectory For smooth transitions between positions, warp interpolation is used for position and spherical linear interpolation is used for attitude.

[0055] Specifically, obtaining the training dataset may include the following steps: S201, the position portions of the motion trajectory of the manipulator and the motion trajectory of the object are interpolated using the warp interpolation method to obtain the interpolated position trajectory of the manipulator and the position trajectory of the object.

[0056] Specifically, the positional portion of the manipulator's posture motion trajectory and the object's pose motion trajectory is transformed and enhanced using the warp interpolation method. The essence of warp is to map the original trajectory from the initial pose space of the actual demonstration to a new target pose space through geometric transformation, while preserving the shape characteristics and motion laws of the trajectory.

[0057] For example, first calculate the vectors pointing from the original starting point and the new starting point to the fixed ending point, denoted as follows: and And project it onto the XY-plane to obtain the projection vector. and Then, press the original trajectory... Perform normalization and use | is used as a scaling factor to scale the trajectory. Then the angle between the two vectors is... The rotation represented is applied to the trajectory, resulting in a transformed trajectory with the new starting point as the origin. Finally, the displacement vector is used. This means that the offset from the original starting point to the new starting point translates the entire trajectory, completing the reconstruction of the position portion. The final result is the interpolated position trajectory that maintains the original motion patterns under the new starting pose.

[0058] S202, the attitude portion of the motion trajectory of the manipulator and the motion trajectory of the object is interpolated using the spherical linear interpolation method to obtain the interpolated attitude trajectory of the manipulator and the attitude trajectory of the object.

[0059] Specifically, a spherical linear interpolation method is used for the attitude component to achieve a smooth transition in the unit quaternion space. This method ensures that the attitude change during the interpolation process is uniform and conforms to the geometric characteristics of rotational motion.

[0060] For example, to ensure consistency in the operation at the end of the trajectory, a fixed proportion of the original pose is preserved near the endpoint of the trajectory. The range of this region (referring to a trajectory interval in the time dimension) is determined by the scaling parameter. (when Approaching 1: Almost the entire trajectory retains the original pose, and the interpolation effect is minimal. Approaching 0: Only a very short end region is preserved, with a large interpolation influence range. This is controlled by the new starting pose (the pose corresponding to the new starting point generated through position warp interpolation) and the end region (the time interval during which the end of the original trajectory remains unchanged, its length determined by...). Between the control points, a spherical linear interpolation algorithm is used for attitude interpolation, thereby achieving a smooth attitude transition throughout the trajectory process. This results in a smooth interpolated attitude trajectory that preserves motion characteristics.

[0061] S203, combine the interpolated position trajectory of the manipulator and the position trajectory of the object, as well as the interpolated posture trajectory of the manipulator and the posture trajectory of the object, to obtain a training dataset.

[0062] Specifically, in step S203, the position trajectory and attitude trajectory are recombine to construct a complete pose trajectory training sample. The recombination method can be spatiotemporal alignment to ensure that the position and attitude trajectories are synchronized on the time axis. Through spatiotemporal alignment, data standardization (normalizing the trajectory data to eliminate dimensional differences) and sample augmentation (generating diverse training samples through multiple interpolations), a complete time-series pose trajectory variant containing multiple different initial conditions is generated, thereby constructing a training dataset that can adapt to diverse configurations and environments.

[0063] S300, Using the training dataset, train the diffusion policy model to obtain the target diffusion policy model.

[0064] Specifically, obtaining the target diffusion strategy model may include the following steps: The pose trajectory sequence in the training dataset is used as a training sample and input into the diffusion Transformer policy model for training. Through the back diffusion process, the diffusion Transformer policy model is trained to learn the denoising mapping from noise to the target pose trajectory, thus obtaining the target diffusion policy model.

[0065] The pose trajectory sequence represents the sequence of the interpolated manipulator posture motion trajectory and the object pose motion trajectory formed by time, and the target diffusion strategy model can generate continuous motion trajectories based on real-time observations.

[0066] It should be noted that the diffusion-based Transformer policy model is trained using pose trajectory sequences from the training dataset. This process is based on the principle of diffusion models, learning a denoising mapping from noise to the target pose trajectory through a back-diffusion process, enabling the model to generate continuous motion trajectories based on real-time observations.

[0067] During training, the forward diffusion training process involves progressively adding noise to the target pose trajectory, generating a noisy trajectory sequence. Next, the reverse denoising training process involves the Transformer policy learning to gradually recover the mapping relationship of the original trajectory from the noisy trajectory. The architectural advantage of the Transformer policy model lies in its use of a self-attention mechanism to capture long-range dependencies in the trajectory sequence, enhancing its ability to learn complex motion patterns.

[0068] For example, in the training process of the Transformer policy model, the loss function can be designed using the mean squared error loss function to optimize the difference between the model's predicted trajectory and the actual trajectory. Next, the pose trajectory sequence is used as a training sample (containing time-seriesd manipulator pose and object pose data) and trained through the loss function until convergence, thereby obtaining the target diffusion policy model. This target diffusion policy model can generate continuous and smooth motion trajectories based on observations (such as the pose of the robot's end effector).

[0069] In one embodiment, obtaining the target diffusion strategy model may further include the following steps: For manipulation tasks involving multiple sub-stages, during the training phase of the diffusion-type Transformer policy model, a training phase switching policy network is used to obtain a target phase switching policy network. The phase switching policy network represents the identification of the completion status of each sub-stage in the manipulation task based on the real-time pose of the object and the state of the robot's end effector, and triggers the transition between stages to form a phase switching decision signal.

[0070] Specifically, for complex manipulation tasks involving multiple sub-stages, a stage-switching policy network is trained in parallel while the diffusion-based policy model is being trained. This network identifies the completion status of task sub-stages based on real-time state information and triggers transitions between stages.

[0071] For example, during the training of the diffusion policy model, object pose and hand pose are used as inputs to the stage switching policy network. The stage switching policy network identifies and analyzes stages, which may include grasping, moving, and placing. These stages combine to form a manipulation task. During training, data with sub-stage boundary annotations is needed to train the stage switching policy network. This annotation data can be obtained manually or generated automatically. Automatic annotation can use significant changes in object pose or abrupt changes in motion as signals for sub-stage transitions. For example, a threshold is set for these changes, and time points where the change is greater than or equal to the set threshold are identified as potential stage transition boundaries, thereby automatically generating annotation information for network training. Using this annotation data (whether manually or automatically generated), the stage switching policy network is trained to identify stage transition opportunities. This network can be trained collaboratively with the diffusion policy model, sharing feature representations. Furthermore, the stage switching policy network itself (or its decision logic) can be optimized using gradient descent to ensure the accuracy of transition timing judgments in real-time deployment. After training, the network can output stage switching decision signals based on real-time observed object and hand poses.

[0072] It should be noted that the time point when a significant change in object pose or a sudden change in motion state is greater than or equal to a set threshold is taken as the reference time point. Preset time intervals are added forward and backward on the timeline of the reference time point, forming a time window centered on that time point. In the next training iteration, changes greater than or equal to the set threshold are searched within the time window. The time window is then recalibrated, and training is repeated until the optimal time window is determined. Changes greater than or equal to the set threshold occurring within this time window serve as the stage switching decision signal.

[0073] For example, based on the training dataset, the training data with initial time window annotations is obtained by traversing the data using a preset initial mutation threshold and initial window radius. In the training dataset, positions greater than or equal to the initial mutation threshold are time-marked, and these time marks are used as reference time points to form time windows with the initial window radius on the timeline, moving forward and backward.

[0074] The stage switching policy network was trained using training data with initial time window annotations to obtain the first-generation stage switching policy model. This model was trained using this labeled data. The learning objective of this stage switching policy model is to accurately determine whether the center point of a time window should be a stage switching point when given trajectory data within that time window. In other words, the stage switching policy model learns to identify which mutation patterns are effective given a given window size.

[0075] Based on the first-generation stage switching strategy model, the training data in the training dataset is re-analyzed to obtain updated time window annotations. Specifically, the first-generation stage switching strategy model is used to re-analyze the training data. This model predicts a switching probability for each point on the trajectory. Regions with consistently high probabilities are identified; these regions represent the more precise new time windows as determined by the stage switching strategy model. Then, within these new time windows, state changes are re-scanned to find points whose changes are greater than or equal to a threshold, thus obtaining new, more precise baseline time points.

[0076] The updated time window labels are compared with those formed during the previous training. If convergence is achieved, the iteration ends, yielding the final optimal time window and the corresponding stage switching decision signal. Convergence is determined by comparing the differences between the old and new labels; if the difference is less than a set threshold, convergence is considered achieved. If the predicted time window position and range of the stage switching strategy model remain relatively stable and no longer change significantly, convergence is considered achieved, and the iteration ends. At this point, the time window obtained from the last training iteration is the optimal time window, and the mutation pattern identified within this window is the most reliable stage switching decision signal.

[0077] The phase switching strategy network and the diffusion strategy model form a closed-loop control system. The phase switching decision signal serves as the conditional input to the diffusion model, ensuring that appropriate trajectory generation strategies are adopted at different task stages, thereby improving the success rate of complex tasks.

[0078] In summary, the S300 process, through the collaborative training of a diffusion-based policy model and a phase-switching policy network, achieves a complete technical chain from a single demonstration video to the execution of complex tasks. This design ensures both the accuracy of action generation and intelligent management of the task flow, significantly improving the practicality and reliability of the imitation learning system in complex environments.

[0079] S400, during deployment, uses the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase, thereby obtaining the joint control commands of the target manipulator.

[0080] The S400 step realizes a complete closed-loop control process from perception to action execution, applying the trained diffusion strategy model to the actual robot deployment environment, and completing the final conversion from observation to control commands.

[0081] Specifically, obtaining joint control commands for the target manipulator may include the following steps: S401, the pose of the robot's end effector and the pose of objects in the environment are acquired in real time, and input into the target diffusion strategy model. The target diffusion strategy model is used for processing to generate the motion trajectory of the target limb posture and the motion trajectory of the target object posture.

[0082] Specifically, in the deployment environment, the pose information of the robot's end effector and objects in the environment is acquired in real time. This real-time data stream serves as input to a target diffusion strategy model, generating the target limb pose trajectory and the target object pose trajectory adapted to the current environmental state. The method for acquiring the pose of the robot's end effector and objects in the environment can be set as needed, for example, through multi-sensor fusion, cameras, etc.

[0083] It should be noted that when acquiring the pose of objects in the environment, in order to avoid visual estimation drift caused by gripper occlusion, an anti-occlusion calculation method is adopted: that is, by combining the real-time pose of the robot end effector with the fixed attachment transformation learned in the demonstration stage (which was completed in step S1013), a stable and reliable current pose of the object is calculated.

[0084] S402, based on the target limb posture motion trajectory and the target object pose motion trajectory, the robot inverse kinematics calculation is used to convert the target limb posture motion trajectory into joint control commands to obtain the joint control commands of the target manipulator.

[0085] Specifically, after obtaining the target limb posture trajectory and the target object pose trajectory in step S401, robot inverse kinematics calculations are used to obtain the rotation angles of each joint required for the hand to reach that position. Next, these parameters, such as rotation angles, are converted into joint control commands, such as current, voltage, or position commands, which are then sent to the motor driver. This yields the joint control commands for the target manipulator. Using these joint control commands, the hand can perform corresponding actions based on the joint control commands of the target manipulator.

[0086] During deployment, after steps S401 and S402 are completed within one control cycle (e.g., 100 times per second), the robot executes the received joint control commands, causing minor changes to its own body and the environment's state. The system then immediately reacquires the new real-time pose and restarts step S401, forming a continuous closed loop of perception-decision-execution. This mechanism enables real-time adjustments based on the latest environmental state, thereby addressing uncertainties and achieving robust control.

[0087] In one embodiment, step S400, which involves using the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model to form the manipulator's posture motion trajectory and the object's pose motion trajectory during the deployment phase, and obtaining the joint control commands for the target manipulator, further includes the following steps: S403, when deploying the current manipulation task execution, the pose of the robot end effector and the pose of objects in the environment are acquired in real time and input into the target diffusion strategy model and the target stage switching strategy network. The target diffusion strategy model and the target stage switching strategy network are used to process them respectively, and the target limb posture motion trajectory and the target object posture motion trajectory are generated accordingly, as well as whether the current manipulation task execution stage needs to be switched, forming the stage switching decision signal.

[0088] S404, based on the stage switching decision signal, the target limb posture motion trajectory and the target object pose motion trajectory, the robot inverse kinematics calculation is used to convert the target limb posture motion trajectory into joint control commands to obtain the joint control commands of the target manipulator.

[0089] Specifically, obtaining joint control commands for the target manipulator may include the following steps: S4041, when the stage switching decision signal indicates that a stage switching is required, the target limb posture motion trajectory and the target object pose motion trajectory are regenerated based on the new stage target through the target diffusion strategy model.

[0090] S4042, when the stage switching decision signal indicates that the current stage should be maintained, the target diffusion strategy model continuously generates the target limb posture motion trajectory and the target object pose motion trajectory based on the current manipulation task.

[0091] S4043, based on the target limb posture motion trajectory and the target object pose motion trajectory, the robot inverse kinematics calculation is used to convert the target limb posture motion trajectory into joint control commands to obtain the joint control commands of the target manipulator.

[0092] Specifically, after obtaining the target limb posture trajectory and stage switching decision signal in step S403, if the signal indicates no switching is needed, the first target point (i.e., the pose to be reached in the next control cycle) is selected from the target limb posture trajectory, and the inverse kinematics calculation is initiated. If the signal indicates a switching is needed, the system immediately or according to a predetermined strategy switches to the target diffusion strategy model pre-trained for the next stage task, and begins executing the target limb posture trajectory generated by the new target diffusion strategy model. This means that different task stages can correspond to different action strategies. For related explanations of robot inverse kinematics calculation, please refer to step S402, which will not be repeated here. Then, after executing the stage switching decision signal, the joint control commands for the target manipulator are obtained.

[0093] In summary, this application aims to address several key issues in existing imitation learning methods and proposes a trajectory-based diffusion strategy learning framework to learn manipulation tasks from single human video segments, and possesses strong generalization capabilities across environments and morphologies.

[0094] To address the current issues of reliance on teleoperation for data acquisition and high data costs in imitation learning, this application eliminates the need for robot demonstrations, directly extracting task-related motion trajectories from human demonstration videos, and requires only one human video demonstration, significantly reducing the data acquisition threshold. To address the issues of large differences in visual domains and difficulty in transferring strategies, this application constructs modality-independent trajectory data by using the pose trajectory of 6D objects as an intermediate representation, effectively avoiding visual interference from viewpoint, background, etc. To address the issues of insufficient training samples and policy overfitting, this application achieves efficient data augmentation through trajectory perturbation and interpolation, generating diverse manipulation data. To enable autonomous execution of complex tasks, this application supports multi-stage task switching and closed-loop control, and can perform dynamic planning and adjustment based on real-time object pose during deployment.

[0095] This application addresses the issues of data efficiency, cross-modal generalization, and task complexity in imitation learning from multiple perspectives, including data sources, representation methods, model design, and execution mechanisms, providing a new path for building embodied intelligent systems with general manipulation capabilities.

[0096] This application has the following beneficial effects: 1. No robot demonstration required, significantly reducing data collection costs. This application relies solely on a single human demonstration video as input, extracting task-related trajectory information through a video analysis module. It eliminates the need to collect robot demonstration data via remote operation or other means, significantly reducing training costs and operational barriers, and solving the problems of high cost and low efficiency in existing imitation learning.

[0097] 2. Employing mode-independent trajectory representation enhances policy generalization ability. This application uses hand pose or 6D object pose trajectory as a unified intermediate representation, which shields the interference of visual style, camera view and background environment, and can effectively improve the transferability of the strategy in different visual domains and different robot platforms, and solve the problem that existing methods are difficult to deploy across modalities and entities.

[0098] 3. Supports multi-stage task switching and closed-loop control, suitable for complex operation processes. This application introduces a phase switching strategy to determine the timing of transitions between task sub-phases and combines it with real-time pose estimation to achieve closed-loop execution. It is suitable for manipulation tasks that include multiple sub-objectives or long-term planning requirements, effectively improving execution stability and reliability.

[0099] 4. The task success rate is significantly improved, demonstrating excellent simulation and physical generalization capabilities. Experimental results show that this application outperforms existing methods by more than 20% in tasks within the RLBench simulation environment, achieving an average success rate of 91.8%. In physical experiments, the success rate approaches 100% even with only 50 enhanced trajectories, significantly improving the algorithm's adaptability and robustness in real-world environments.

[0100] Secondly, this application provides a trajectory diffusion imitation learning system based on a single human video demonstration, applied to the aforementioned trajectory diffusion imitation learning method based on a single human video demonstration, including: The video analysis module is used to analyze the demonstration video to obtain the motion trajectory of the operator and the motion trajectory of the object's position. The demonstration video is a video of a human or machine operating an object. The trajectory generation module is used to interpolate the position and orientation of the manipulator's posture motion trajectory and the object's pose motion trajectory to obtain a training dataset. The trajectory generation module is also used to train the diffusion policy model using the training dataset to obtain the target diffusion policy model. The closed-loop execution module is used to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase by using the real-time pose of the robot's end effector as the observation input of the target diffusion strategy model, thereby obtaining the joint control commands of the target manipulator.

[0101] It should be noted that this application provides a trajectory parsing method for extracting operational intentions from human demonstration videos, which is used to learn robot manipulation strategies from human demonstration videos. It has the advantages of low demonstration cost, high spatial generalization ability and cross-visual domain robustness.

[0102] This application aims to address three key problems in existing imitation learning methods: (1) reliance on teleoperation to collect robot demonstration data, resulting in high data acquisition costs and low efficiency; (2) weak policy generalization ability, prone to failure under changes in initial state or visual domain, making it difficult to directly extract transferable operational intentions from human videos, thus limiting cross-modal applications; and (3) insufficient training samples and policy overfitting. To address these problems, this application proposes a unified solution based on trajectory representation and diffusion modeling, which includes the following three core technical modules: Video analysis module: Input a human demonstration video, automatically identify task-related objects and hand areas, and extract operation trajectories using the HaMeR 3D hand reconstruction model and the FoundationPose 6D object pose estimation model. Based on task type, it is divided into hand tasks and object tasks. Hand tasks are suitable for single-target manipulation, while object tasks can handle multiple targets or long-duration, multi-stage operations.

[0103] The trajectory generation module performs perturbation and interpolation operations on the original trajectory, especially sampling and changing the initial pose of the object in the task, generating a large-scale and diverse training dataset, thereby improving the policy's adaptability to spatial changes.

[0104] Closed-loop execution module: During deployment, the system estimates the poses of the reference object and the target object in real time. For operations after grasping, steady-state tracking control is achieved through the fixed attachment transformation between the gripper and the object, thereby maintaining the consistency and stability of strategy execution under different viewpoints and initial configurations.

[0105] Compared with existing imitation learning methods, this application has significant technical advantages: it eliminates the need for robot demonstrations, training the policy solely through human videos, significantly reducing the cost and complexity of data acquisition; it employs 6D pose trajectories as modality-independent intermediate representations, effectively enhancing the policy's generalization ability under different visual domains and environmental conditions; it generates high-precision, multi-stage manipulation policies through a trajectory diffusion model, supporting fine-grained control of complex task processes; and it combines real-time pose estimation with stage switching strategies to achieve closed-loop execution of the policy and adaptive switching during task processes. This application is applicable to different types of robots and operational task scenarios. Experimental results show that the average success rate of this application reaches 91.8% in simulation tasks, and in real robot operations, a success rate approaching 100% can be achieved simply by enhancing the number of trajectories to 50. It significantly outperforms existing imitation learning methods in terms of manipulation accuracy, robustness, and spatial generalization ability, demonstrating good practicality and promotional value.

[0106] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0107] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0108] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A trajectory diffusion imitation learning method based on a single human video demonstration, characterized in that, include: The demonstration video is analyzed to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position. The demonstration video is a video of a human or machine operating an object. The position and orientation of the manipulator's posture motion trajectory and the object's pose motion trajectory are interpolated to obtain a training dataset. Using the training dataset, the diffusion policy model is trained to obtain the target diffusion policy model; During deployment, the real-time pose of the robot's end effector is used as the observation input to the target diffusion strategy model to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase, thereby obtaining the joint control commands of the target manipulator.

2. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 1, characterized in that, The steps of analyzing the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position include: Identify the task types in the demonstration video, wherein the task types include operator tasks and object tasks; When the task type is the manipulator task, the manipulator posture features are extracted in the demonstration video using a 3D reconstruction model to obtain the manipulator posture motion trajectory, wherein the manipulator posture motion trajectory represents the remapped posture calculated based on the pose relationship of the manipulator joints. When the task type is the object task, the object pose feature is extracted in the demonstration video using the object 6D pose estimation model to obtain the object pose motion trajectory, wherein the object pose motion trajectory includes the object's 6D pose information.

3. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 2, characterized in that, The steps of analyzing the demonstration video to obtain the motion trajectory of the operator's posture and the motion trajectory of the object's position include: When the manipulator is the robot's hand: When the task type is the operation body task, the HaMeR model is used to extract features of the key joint postures of the hand in the demonstration video to obtain the hand posture motion trajectory. The key joints of the hand include the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger. The remapping pose is calculated based on the pose relationship of the metacarpophalangeal joint of the thumb, the interphalangeal joint of the thumb, and the metacarpophalangeal joint of the index finger. When the task type is the object task, the FoundationPose model is used to extract features of the 6D pose of the object in the demonstration video to obtain the object's pose motion trajectory. In the demonstration video, in the keyframe when the object is successfully grasped, the hand pose is extracted using the HaMeR model, and the hand gripper pose executed by the robot is obtained based on the relative transformation relationship between the object pose and the hand pose, as well as the remapping pose of the key joints of the hand.

4. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 1, characterized in that, The step of interpolating the position and orientation of the manipulator's motion trajectory and the object's pose motion trajectory to obtain the training dataset includes: The positional portions of the manipulator's posture motion trajectory and the object's pose motion trajectory are interpolated using the warp interpolation method to obtain the interpolated positional trajectories of the manipulator and the object. The attitude portion of the motion trajectory of the manipulator and the pose of the object is interpolated using the spherical linear interpolation method to obtain the interpolated attitude trajectory of the manipulator and the attitude trajectory of the object. The interpolated position trajectories of the manipulator and the object, as well as the interpolated pose trajectories of the manipulator and the object, are combined to obtain the training dataset.

5. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 1, characterized in that, The step of training the diffusion policy model using the training dataset to obtain the target diffusion policy model includes: The pose trajectory sequence in the training dataset is used as training samples and input into the diffusion-type Transformer policy model for training. Through the back diffusion process, the diffusion-type Transformer policy model is trained to learn the denoising mapping from noise to the target pose trajectory, thus obtaining the target diffusion-type policy model. The pose trajectory sequence represents the sequence formed by the interpolated manipulator posture motion trajectory and the object pose motion trajectory in time. The target diffusion-type policy model can generate continuous motion trajectories based on real-time observations.

6. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 1, characterized in that, The step of training the diffusion policy model using the training dataset to obtain the target diffusion policy model further includes: For manipulation tasks involving multiple sub-stages, during the training phase of the diffusion-type Transformer policy model, a training phase switching policy network is used to obtain a target phase switching policy network. The phase switching policy network represents the identification of the completion status of each sub-stage in the manipulation task based on the real-time pose of the object and the state of the robot's end effector, and triggers the transition between stages to form a phase switching decision signal.

7. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 1, characterized in that, The step of obtaining joint control commands for the target manipulator by using the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model during deployment to form the manipulator's posture motion trajectory and the object's pose motion trajectory during the deployment phase includes: The poses of the robot's end effector and objects in the environment are acquired in real time and input into the target diffusion strategy model. The target diffusion strategy model is then used for processing to generate the motion trajectory of the target limb posture and the motion trajectory of the target object posture. Based on the target limb posture motion trajectory and the target object pose motion trajectory, the target limb posture motion trajectory is converted into joint control commands using robot inverse kinematics calculation, thereby obtaining the joint control commands for the target manipulator.

8. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 6, characterized in that, The step of obtaining joint control commands for the target manipulator by using the real-time pose of the robot's end effector as the observation input to the target diffusion strategy model during deployment to form the manipulator's posture motion trajectory and the object's pose motion trajectory during the deployment phase, further includes: When deploying the current manipulation task, the pose of the robot end effector and the pose of objects in the environment are acquired in real time and input into the target diffusion strategy model and the target stage switching strategy network. The target diffusion strategy model and the target stage switching strategy network are used to process them respectively, and the target limb posture motion trajectory and the target object posture motion trajectory are generated accordingly, as well as whether the current manipulation task execution stage needs to be switched, forming the stage switching decision signal. Based on the stage switching decision signal, the target limb posture motion trajectory and the target object pose motion trajectory, the target limb posture motion trajectory is converted into joint control commands using robot inverse kinematics calculation, thus obtaining the joint control commands for the target manipulator.

9. The trajectory diffusion imitation learning method based on a single human video demonstration as described in claim 8, characterized in that, The step of converting the target limb posture trajectory into joint control commands based on the stage switching decision signal, the target limb posture motion trajectory, and the target object pose motion trajectory, and obtaining the joint control commands of the target manipulator using robot inverse kinematics calculation includes: When the phase switching decision signal indicates that a phase switching is required, the target limb posture motion trajectory and the target object pose motion trajectory are regenerated based on the new phase target using the target diffusion strategy model. When the stage switching decision signal indicates that the current stage should be maintained, the target diffusion strategy model continuously generates the target limb posture motion trajectory and the target object pose motion trajectory based on the current manipulation task. Based on the target limb posture motion trajectory and the target object pose motion trajectory, the robot inverse kinematics calculation is used to convert the target limb posture motion trajectory into joint control commands, thereby obtaining the joint control commands for the target manipulator.

10. A trajectory diffusion imitation learning system based on a single human video demonstration, characterized in that, The trajectory diffusion imitation learning method based on a single human video demonstration, applied to any one of claims 1-8, includes: The video analysis module is used to analyze the demonstration video to obtain the motion trajectory of the operator and the motion trajectory of the object's position. The demonstration video is a video of a human or machine operating an object. The trajectory generation module is used to interpolate the position and orientation of the manipulator's posture motion trajectory and the object's pose motion trajectory to obtain a training dataset. The trajectory generation module is also used to train the diffusion policy model using the training dataset to obtain the target diffusion policy model. The closed-loop execution module is used to form the motion trajectory of the manipulator and the motion trajectory of the object during the deployment phase by using the real-time pose of the robot's end effector as the observation input of the target diffusion strategy model, thereby obtaining the joint control commands of the target manipulator.

Images