A human-machine skill transfer method and system based on object center motion abstraction

Abstract

The application discloses a human-machine skill transfer method and system based on object center motion abstraction. The method comprises the following steps: acquiring dense object center motion flow of human operation demonstration; extracting task evidence latent variables through a task evidence bottleneck, and adopting unconditional prior and task classification loss joint constraint; generating motion abstraction of K source anchor points on P ordered time stages based on the task evidence latent variables; performing cross-instance structure alignment on the motion abstraction through source-time prototype constraint; predicting the final motion abstraction according to single-frame RGB-D observation and task description; and decoding the prediction result into target object rigid body motion and converting the target object rigid body motion into a robot end effector path. The application can effectively decouple task information and performer information, establish a cross-instance unified space-time organization structure, reduce the motion representation dimension, and improve the success rate and generalization ability of robot skill transfer.

Description

Technical Field

[0001] This invention relates to the fields of intelligent robot control, computer vision, machine learning, and human-machine skill transfer technology, and in particular to a human-machine skill transfer method and system based on object-centered motion abstraction. Background Technology

[0002] Human operation videos contain rich task knowledge and motion process information, are inexpensive to collect, and cover a wide range of task types, making them suitable as a data source for robot skill learning. However, humans and robots differ in body structure, joint degrees of freedom, end effectors, range of motion, viewing angle, and movement habits, meaning robots cannot directly replicate the joint trajectories of a human hand or body.

[0003] To bridge this "embodiment gap," existing systems typically establish an intermediate motion representation between "human demonstration" and "robot control commands." This object-centric motion representation does not directly record human hand movements, but rather records the displacement, rotation, or surface point trajectories of the manipulated object, thereby reducing the impact of differences in the executing agent on skill transfer. In recent years, a series of representative works have emerged focusing on object-centric motion representation.

[0004] The existing commonly used methods for representing the motion of object centers and their limitations are as follows: Dense Object-Centric Motion Field (DOCMF) uses an object-centric 3D motion field as a motion representation for learning robot control strategies from human videos. This method samples a large number of points on the surface of the target object and records the 3D motion trajectory of each point at multiple time points. It then trains a "denoised" 3D motion field estimator to robustly extract fine-grained object 3D motion from noisy RGB-D videos. Experiments show that this method reduces 3D motion estimation errors by more than 50%. However, while DOCMF offers high spatial coverage, it contains a large number of repetitive trajectories with similar directions, amplitudes, and temporal variations. This results in high output dimensionality, easily encoding actor habits, scene layout, and camera perspective, increasing the training difficulty of the prediction model.

[0005] Keypoint trajectories reduce representation dimensionality by selecting a small number of keypoints from dense trajectories through random sampling, farthest point sampling, motion amplitude sorting, or other strategies. Existing techniques further leverage pre-trained vision-language models to automatically generate task-relevant and cross-instance consistent keypoints, enabling robots to learn keypoint conditionalization strategies with only 5 to 10 demonstrations. However, this method remains inherently sparsity-based and cannot actively distinguish between motion evidence required to complete the task and actor-specific information. Furthermore, depth noise, point tracking errors, pose estimation errors, and sampling strategies can cause keypoint trajectory jitter, affecting representation stability.

[0006] The existing technologies currently have the following common defects: This indicates redundancy. Dense motion fields and point cloud methods contain a large amount of approximately repetitive motion information, increasing the dimensionality of the prediction model output and the difficulty of training.

[0007] This indicates instability. Depth noise, point tracking errors, pose estimation errors, and sampling strategies can cause keypoint trajectory jitter.

[0008] Task information is coupled with execution information. Execution speed, movement range, gripping position, camera viewpoint, and scene layout may be mistaken by the model as the task itself, making it difficult to generalize during skill transfer.

[0009] There is a lack of a unified structure across instances. Different executors of the same task may use different contact points and different action speeds, resulting in the source point and time phase not being able to be directly mapped.

[0010] Training objectives are difficult to establish. Different demonstrations often lack point-to-point correspondences, strict synchronization relationships, or paired supervision samples, making it difficult to form stable robot prediction objectives. Summary of the Invention

[0011] This invention aims to provide a human-machine skill transfer method and system based on object-centric motion abstraction. It aims to extract a low-dimensional, cross-instance unified, and decoupled task information and execution information structured motion abstraction from the original, dense object motion flow that is significantly influenced by the executor, so as to serve as a cross-embodied intermediate interface from human demonstration to robot execution.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: A human-computer skill transfer method based on object-centric motion abstraction includes the following steps: S1. Obtain the motion flow at the center of dense objects in the human operation demonstration; S2. Extract task evidence latent variables from the dense object center motion flow through the task evidence bottleneck, wherein the task evidence bottleneck adopts a joint constraint of unconditional prior and task classification loss. S3. Generating latent variables based on the aforementioned task evidence. Individual source anchor point An ordered time-phase motion abstraction, the motion abstraction being composed of... OK The column is composed of three-dimensional displacement vectors; S4. Perform cross-instance structure alignment on the motion abstraction through source-time prototype constraints to obtain the final motion abstraction; S5. Predict the final motion abstraction based on single-frame RGB-D observations and task description; S6. The predicted motion is abstracted and decoded into the rigid body motion of the target object, and then converted into the robot end effector path.

[0013] 2. The method according to claim 1, wherein step S1, obtaining the dense object center motion flow of the human operation demonstration specifically includes: Based on the target object segmentation results, object pose information, and depth information, samples are taken from the surface of the target object. Each point; the sampling method can be any one of uniform sampling, farthest point sampling, grid sampling, random sampling within a segmented mask, or curvature-based sampling; Track the The point is at Three-dimensional motion at a given time location; Representing all displacements in the target object coordinate system constitutes the dense object center motion flow; the dense object center motion flow is represented as... ,in This indicates the number of sampling points on the surface of the target object. This represents the number of time samples for the motion trajectory, and each three-dimensional vector represents the displacement of the corresponding surface point in the target object's coordinate system.

[0014] Further, in step S2, the extraction of task evidence latent variables from the dense object-centered motion flow through the task evidence bottleneck specifically includes: The motion flow at the center of the dense object Input the motion encoder to obtain point-by-point local motion features. and global motion characteristics The The The motion encoder is implemented using any one or a combination of point cloud network, ensemble encoder, spatiotemporal Transformer or graph neural network; Based on the global motion characteristics The posterior distribution is obtained through the posterior parameter generator. ; From the posterior distribution The latent variables of the task evidence were obtained by sampling. ; With unconditional standard Gaussian prior Constraining the latent variables of the task evidence Calculate the KL divergence loss : ; in, Denotes the KL divergence operator, Represents the identity matrix; Losses categorized by task The constraints enable the motion abstraction to identify task categories; Joint training objectives for: ; in, The balance coefficient is the equilibrium coefficient. The value is linearly increased from 0 to the target value during the initial training period.

[0015] Furthermore, in step S3, based on the task evidence latent variables... generate Individual source anchor point The abstraction of motion over an ordered time phase specifically includes: The latent variables of the task evidence and Each learnable slot is embedded and combined to form One source anchor query vector; The The source anchor point query vector and the point-by-point local motion features Perform slot cross-attention to obtain the source assignment matrix. The source allocation matrix This represents the soft-assigned weight of each source anchor point to each object surface sampling point; According to the source allocation matrix For the dense object center motion flow Perform weighted aggregation to form The bar represents the trajectory; Based on the task evidence latent variables, the stage prediction head is used. Output An orderly time phase ; According to the above An orderly time phase In the The motion abstraction is obtained by interpolating time along the representative trajectory. The .

[0016] Furthermore, the latent variables of the task evidence... and Each learnable slot is embedded and combined to form Each source anchor query vector specifically includes: Using the latent variables of the aforementioned task evidence As a condition, for the above Conditional modulation is performed on learnable slot embeddings to generate semantically relevant data for the current task. One source anchor query vector; The slot cross-attention is calculated by... The source anchor point query vector and the point-by-point local motion features The attention weights between them are used to obtain the source assignment matrix. .

[0017] Furthermore, in step S4, the cross-instance structural alignment of the motion abstraction through source-time prototype constraints specifically includes: For task categories Collect the original motion abstractions of this task category from the training set, and perform a weighted average of each source-time unit according to slot activity to form the task source-time prototype. : ; ; in, Indicates task category Next The source anchor point is at the first Prototype displacement vector at each time stage Indicates the task category The same source-time unit in all training instances The operation is a weighted average based on slot activity. Representing the abstraction of primitive motion in the first... Source anchor point, the first Three-dimensional displacement vectors at each time stage; For the original motion abstraction of the current instance Generate at most Abstract of candidate motions Candidate layout is generated by replacing the least active source anchor in the current motion abstraction with a more active source anchor in the task prototype, while keeping the source anchor capacity and stage order unchanged. For candidate motion abstraction Calculate its relationship with the task source-time prototype. Source-time prototype distance between : ; in Representing candidate motion abstraction Source-time unit index in Indicates the first The source configuration confidence of each unit, Represents the cutoff distance constant. Represents the task source-time prototype Source-time unit index in Indicates task category The source-time prototype occupies the support set. Indicates the first The unit and the first Structural distance between units This is an operation to find the minimum value; Computing candidate motion abstractions Overall rating : ; in, Representing candidate motion abstraction The sum of the source configuration confidence level, This refers to the distance weighting coefficient. When the candidate motion abstraction The candidate motion abstraction is selected when the following conditions are met. Otherwise, retain the original motion abstraction. : ; ; in, For confidence level decline tolerance, To raise the threshold for the overall score.

[0018] Further, in step S5, the prediction of the final motion abstract based on single-frame RGB-D observations and task description specifically includes: Construct the overall scene point cloud and target object point cloud from a single frame of RGB-D observation; Extract features from the task text description; The scene features, target object features, task text features, and camera parameter features are input into the conditional fusion network; Predicted by a dedicated decoder Individual source anchor point The three-dimensional motion, effectiveness state, and internal stage time of each stage.

[0019] Further, in step S6, the step of abstracting and decoding the predicted motion into the rigid body motion of the target object specifically includes: The source anchor point displacement field between adjacent time stages is centered. Singular value decomposition is performed on the centered source anchor displacement field to estimate the shared rotation axis of the target object; The translation assumption and the shared axis rotation assumption are established respectively, and the motion assumption is selected based on the reconstruction residual and the validity of the source anchor point. Based on the selected motion assumptions, the discrete values ​​between adjacent time stages are obtained. Rigid body transformation, through interpolation, obtains the pose trajectory of a continuous object.

[0020] Further, in step S6, converting the predicted motion abstraction into a robot end effector path specifically includes: The transformation matrix from the camera coordinate system to the robot base coordinate system is obtained through eye-to-hand external calibration; Transform the pose trajectory of the target object into the robot's base coordinate system; Based on the initial relative pose of the object and the end effector, the object's pose trajectory is converted into end effector path points.

[0021] A human-computer skill transfer system based on object-centric motion abstraction is provided to implement the aforementioned human-computer skill transfer method based on object-centric motion abstraction. The system includes: The motion abstraction construction module is configured to acquire dense object-centric motion flows from human operation demonstrations, extract task evidence latent variables from these flows through task evidence bottlenecks, and generate [something] based on these task evidence latent variables. Individual source anchor point An ordered time-phase motion abstraction, the motion abstraction being composed of... OK The column is composed of three-dimensional displacement vectors; The source-time prototype constraint module is configured to perform cross-instance structure alignment of the motion abstraction through source-time prototype constraints to obtain the final motion abstraction. The motion representation prediction module is configured to predict the final motion abstraction based on a single-frame RGB-D observation and a task description; The robot execution module is configured to abstract and decode the predicted motion into rigid body motion of the target object and convert it into a robot end effector path.

[0022] As can be seen from the above technical solutions, compared with the prior art, the beneficial effects of the present invention are: 1. This invention compresses the original dense object-centered motion flow into a low-dimensional structured motion abstraction by using the Task Evidence Bottleneck (TEB), which significantly reduces the dimensionality and redundancy of motion representation, while improving task decodeability.

[0023] 2. This invention addresses the bottleneck of task evidence by employing unconditional standard Gaussian prior and task classification loss as joint constraints. This ensures that latent variables do not record all input details by default. The executor's habits, action speed, action amplitude, and scene changes that do not contribute to task classification are compressed, while the object movement patterns that have a distinguishing effect on the task are preserved, thus achieving effective decoupling between task information and executor information.

[0024] 3. This invention establishes a unified source-time organization structure (ST-PTC) across instances, enabling different execution instances of the same task to converge to a unified time frame without requiring point-level geometric correspondence or strict time synchronization between different demonstrations. Indexed grid structure.

[0025] 4. This invention, through the Motion Representation Prediction Module (MRP), can predict the complete motion abstraction with only a single frame of RGB-D observation and task description during deployment, without the need to input a complete human demonstration video or recalculate human trajectories, significantly reducing the perception and data requirements during the deployment phase. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the steps of the human-computer skill transfer method based on object-centered motion abstraction of the present invention. Figure 2 This is a schematic diagram of the architecture constructed by the CRAFT motion abstraction of the present invention; Figure 3 This is a schematic diagram illustrating the process of motion representation prediction, rigid body decoding, and robot execution in this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The illustrative embodiments and descriptions of this invention are for illustrative purposes only and are not intended to limit the invention.

[0028] like Figure 1 As shown, the present invention provides a human-computer skill transfer method based on object-centered motion abstraction. The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0029] 1. Obtain the motion flow at the center of dense objects in the human operation demonstration (step S1) The process of obtaining the motion flow at the center of a dense object in a human operation demonstration specifically includes the following steps: S11. Object Surface Sampling. Based on the target object segmentation results, object pose information, and depth information, sample the surface of the target object. Each point. The sampling method can be any one of uniform sampling, farthest point sampling, grid sampling, random sampling within a segmented mask, or curvature-based sampling. In this embodiment, The value is 128.

[0030] S12, Motion Tracking. Tracking the... The point is at Three-dimensional motion at a specific time position. In this embodiment, The value is 8. Tracking can be achieved using optical flow, point tracking algorithms, or deep learning-based tracking methods.

[0031] S13. Coordinate System Representation. All displacements are represented in the target object's coordinate system, forming the dense object's central motion flow. The dense object's central motion flow is represented as... ,in This indicates the number of sampling points on the surface of the target object. This represents the number of time samples for the motion trajectory, with each 3D vector representing the displacement of the corresponding surface point in the target object's coordinate system. Representing motion in the target object's coordinate system eliminates the differences in absolute position across different scenes, allowing the model to focus on the relative motion structure of the target object itself.

[0032] In this embodiment, the motion flow at the center of the dense object .

[0033] II. Extracting latent variables for task evidence (Step S2) Extracting latent variables of task evidence from the dense object-centric motion flow by overcoming the task evidence bottleneck specifically includes the following steps: S21, Motion Encoding. This involves encoding the motion flow at the center of the dense object. Input the motion encoder to obtain point-by-point local motion features. and global motion characteristics The aforementioned The The motion encoder can be implemented using any one or a combination of point cloud networks, ensemble encoders, spatiotemporal Transformers, or graph neural networks. In this embodiment, the motion encoder is implemented using a point cloud network.

[0034] Point-by-point local motion characteristics Preserve the motion differences between different object surface regions for subsequent source anchor point generation; global motion features. Aggregate motion information of the entire object to generate latent variables for task evidence.

[0035] S22, Posterior distribution generation. Based on the global motion features... The posterior distribution is obtained through the posterior parameter generator. The posterior parameter generator can be implemented using a multilayer perceptron (MLP), outputting the mean of the posterior distribution. and variance .

[0036] S23. Latent variable sampling. From the posterior distribution... The latent variables of the task evidence were obtained by sampling. In this embodiment, It is a 64-dimensional vector. Sampling can be achieved using the reparameterization trick, that is... Represented as ,in To obtain from the standard normal distribution The random noise vector obtained by sampling in the middle, This is an element-wise product.

[0037] S24, KL divergence constraint. Based on unconditional standard Gaussian prior. Constraining the latent variables of the task evidence Calculate the KL divergence loss : ; in, Denotes the KL divergence operator, This represents the identity matrix. The KL divergence loss encourages the posterior distribution. The near-standard Gaussian prior ensures that latent variables do not record all input details by default, compressing executor habits, action speed, action amplitude, and scene changes that do not contribute to task classification.

[0038] S25, Task Classification Constraint. Based on task classification loss. Constraints are imposed to enable the motion abstraction to identify task categories. The task classification loss can be implemented using the cross-entropy loss function.

[0039] S26. Joint Training. Joint Training Objectives for: ; in, This is a balancing coefficient used to adjust the weight of the KL divergence loss in the total loss. The balancing coefficient... The value is linearly increased from 0 to the target value during the early training period to reduce the impact of strong information compression on task feature learning in the early stages of training. In this embodiment, It increases linearly from 0 to 0.01 over the first 40 training cycles.

[0040] Through the aforementioned task evidence bottleneck mechanism, executor habits, action speed, action range, and scene changes that do not contribute to task classification are compressed, while object movement patterns that have a distinguishing effect on tasks are preserved, thereby achieving effective decoupling between task information and executor information.

[0041] III. Generate the motion abstraction of the source anchor point in the ordered time phase (step S3) Based on the latent variables of the task evidence generate Individual source anchor point The abstraction of motion over an ordered time phase includes the following steps: S31, Source Anchor Point Query Vector Generation. (This refers to the generation of the task evidence latent variables.) and Each learnable slot is embedded and combined to form Each source anchor query vector. Specifically, using the task evidence latent variables. As a condition, for the above Conditional modulation is performed on learnable slot embeddings to generate semantically relevant data for the current task. Each source anchor point query vector. Conditional modulation can be achieved through feature concatenation followed by linear layer mapping, affine transformation, or gating mechanisms. In this embodiment, The value is 8.

[0042] In one implementation, the conditional modulation is achieved by: adjusting the task evidence latent variables... Mapped to modulation parameters via linear transformation (Scaling parameter, multiplicative factor of affine transformation) and (Bias parameter, additive factor of affine transformation), for the Learnable slot embedding Perform affine transformation: The conditional source anchor query vector is obtained. .

[0043] S32, Slot Cross Attention. The source anchor point query vector and the point-by-point local motion features Perform slot cross-attention to obtain the source assignment matrix. The source allocation matrix This represents the soft-assigned weight of each source anchor point to each object surface sampling point, i.e. ,in Indicates the first The source anchor point for the first The weights are assigned to the sampling points on the object surface. The slot cross-attention is calculated by... The source anchor point query vector and the point-by-point local motion features The attention weights between them are used to obtain the source assignment matrix. .

[0044] Specifically, let the source anchor query vector be... Point-by-point local motion characteristics The source allocation matrix is ​​calculated as keys and values ​​in the following manner. : ; in , Indicates the first The source anchor point for the first Attention weights for each surface point.

[0045] S33, Weighted Aggregation. Based on the source allocation matrix. For the dense object center motion flow Perform weighted aggregation to form Each line represents a trajectory. Specifically, for the first line... The source anchor point, which is the first source anchor point, in the... Representative trajectory at each time position for: ; in, Indicates the first The surface point at the first Displacement vectors at each time position.

[0046] S34. Ordered time phase generation. This is achieved by using the phase prediction head based on the task evidence latent variables. Output An orderly time phase The stage sequence is maintained through model structure or monotonic parameterization, and manual time labeling is not required for each stage. In this embodiment, The value is set to 4. The start and end times of the motion are fixed as boundary stages. The stage prediction head outputs two internally ordered stages, forming a total of four stages.

[0047] S35, Time Interpolation. As described... An orderly time phase In the The motion abstraction is obtained by interpolating time along the representative trajectory. The Time interpolation can be achieved using linear interpolation or spline interpolation.

[0048] In this embodiment, motion abstraction The number of source anchor points The number of ordered time stages can be adjusted between 4 and 32. It can be adjusted between 3 and 8, and the specific value depends on the complexity of the task.

[0049] IV. Cross-instance structure alignment (step S4) The motion abstraction is aligned across instances using source-time prototype constraints to obtain the final motion abstraction, specifically including the following steps: S41, Prototype Library Construction. (For task categories) Collect the original motion abstractions of this task category from the training set, and perform a weighted average of each source-time unit according to slot activity to form the task source-time prototype. : ; ; in, Indicates task category Next The source anchor point is at the first Prototype displacement vector at each time stage Indicates the task category The same source-time unit in all training instances The operation is a weighted average based on slot activity. Representing the abstraction of primitive motion in the first... Source anchor point, the first The three-dimensional displacement vectors at each time stage. The prototype library is built after training and before the inference stage, without introducing additional trainable parameters.

[0050] S42, Candidate Layout Generation. This involves the initial motion abstraction for the current instance. Generate at most Abstract of candidate motions Candidate layouts are generated by replacing the least active source anchor in the current motion abstraction with a more active source anchor in the task prototype, while maintaining the source anchor capacity and phase order.

[0051] S43, Prototype distance calculation. For candidate motion abstraction... Calculate its relationship with the task source-time prototype. Source-time prototype distance between : ; in, Representing candidate motion abstraction Source-time unit index in Indicates the first The source configuration confidence of each unit, Represents the cutoff distance constant. Represents the task source-time prototype Source-time unit index in Indicates task category The source-time prototype occupies the support set. Indicates the first The unit and the first Structural distance between units The minimum value operation is performed. The source-time prototype distance can be implemented using any one of the following: truncated distance, Chamfer-type distance, Earth Mover-type distance, or kernel distance.

[0052] This distance metric does not force a one-to-one point-level matching relationship between candidate motion abstractions and prototypes, thus solving the technical problem of lack of point-level correspondence and time synchronization between different demonstration instances.

[0053] S44. Comprehensive score calculation. Calculate the candidate motion abstraction. Overall rating : ; in, Representing candidate motion abstraction The sum of the source configuration confidence level, This represents the distance weighting coefficient. The overall score considers both the source configuration reliability of the current instance and its structural differences from the task prototype.

[0054] S45, Utility Gating Selection. When the candidate motion abstraction... The candidate motion abstraction is selected when the following conditions are met. Otherwise, retain the original motion abstraction. : ; ; in, For confidence level decline tolerance, To increase the threshold for the overall score. In this embodiment, , , .

[0055] The aforementioned utility gating mechanism ensures that candidate layouts are selected only when the confidence level is not lower than the allowable decrease range of the original layout and the overall score is at least higher than a set threshold; otherwise, the original motion abstraction is retained.

[0056] By using source-time prototype constraints, different execution instances of the same task can converge to a unified form without requiring point-level geometric correspondence or strict time synchronization between different demonstrations. The index grid structure establishes a unified source-time organization structure across instances.

[0057] like Figure 2 As shown, the motion abstraction construction in steps S1 to S4 above constructs an overall architecture, including dense object-centered motion flow input, motion encoder, task evidence bottleneck (TEB), and task evidence latent variables. The module connection relationships and data flow of core components such as slot cross attention, source anchor point generation, ordered time stage generation, and source-time prototype constraint (ST-PTC) are analyzed.

[0058] V. Final Motion Abstraction Prediction (Step S5) The prediction of the final motion abstract based on single-frame RGB-D observations and task description specifically includes the following steps: S51. Point Cloud Construction. Construct the overall scene point cloud and the target object point cloud from a single frame of RGB-D observation. Specifically, using the RGB image and depth image, the depth map is back-projected into a 3D point cloud using camera intrinsic parameters to obtain the overall scene point cloud; using the target object segmentation results, the target object point cloud is segmented from the overall scene point cloud.

[0059] S52. Feature Extraction. Text Feature Extraction. The overall scene point cloud and the target object point cloud are input into a shared point cloud backbone network PTv3, respectively, to extract scene point cloud features and target object point cloud features; a frozen visual-language encoder (such as SigLIP2) is used to extract the feature vector of the task description. Here, "shared" means that the overall scene point cloud and the target object point cloud share the same parameters of the PTv3 backbone network.

[0060] S53. Feature Fusion. Scene features, target object features, task text features, and camera parameter features are input into the conditional fusion network. The conditional fusion network can be implemented using a multilayer perceptron or a Transformer, and outputs the fused conditional features.

[0061] S54, Motion Abstraction Prediction. Predicted by a dedicated representation decoder. Individual source anchor point The three-dimensional motion, validity state, and internal stage time of each stage are represented. This indicates that the dedicated decoder can be implemented using a Transformer decoder or an MLP.

[0062] In this embodiment, the Motion Representation Prediction Model (MRP) is trained using mask coordinate mean squared error loss, validity binary cross-entropy loss, and stage-time Smooth L1 loss, with weights of 1, 0.1, and 0.5, respectively. MRP is trained for 100 epochs with a batch size of 64, using the AdamW optimizer with a learning rate of... .

[0063] With the MRP module described above, only a single frame of RGB-D observation and task description are needed to predict the complete motion abstraction during deployment, without the need to input a complete human demonstration video or recalculate human trajectories, which significantly reduces the perception and data requirements during the deployment phase.

[0064] VI. Motion Abstraction Decoding and Conversion (Step S6) The predicted motion is abstracted and decoded into rigid body motion of the target object, and then converted into a robot end effector path. This process includes the following steps: S61, Rigid Body Motion Decoding: Displacement Field Centering. The displacement field of the source anchor point between adjacent time stages is centered. For the first... Phase to the Displacement field of stage Calculate its mean Then, the displacement field is centered: .

[0065] S62. Rigid Body Motion Decoding: Rotation Axis Estimation. Singular Value Decomposition (SVD) is performed on the centered source anchor displacement field to estimate the shared rotation axis of the target object. Specifically, a matrix is ​​constructed. ,in This is the position matrix of the source anchor point in the target object's coordinate system. Perform SVD decomposition: The rotation matrix is ​​obtained. Then, the rotation axis and rotation angle are extracted.

[0066] S63. Rigid Body Motion Decoding: Motion Assumption Selection. Establish translation and shared-axis rotation assumptions separately, and select the motion assumption based on the reconstructed residuals and the validity of the source anchors. The translation assumption considers only overall translational motion; the shared-axis rotation assumption considers rotational motion about the estimated shared rotation axis. Based on the reconstructed residuals and the number of valid source anchors under both assumptions, select the motion assumption that best matches the current prediction.

[0067] S64. Rigid Body Motion Decoding: Trajectory Generation. Based on the selected motion assumptions, discrete values ​​between adjacent time stages are obtained. Rigid body transformation, through interpolation, obtains the pose trajectory of a continuous object. Transformation includes rotation matrix Translation vector Interpolation can be performed using quaternion spherical linear interpolation (SLERP) for rotation and linear interpolation for translation.

[0068] S65. Coordinate Transformation. The transformation matrix from the camera coordinate system to the robot's base coordinate system is obtained through eye-to-hand external calibration. Transform the pose trajectory of the target object from the camera coordinate system to the robot base coordinate system.

[0069] S66. End-effector path generation. Based on the initial relative pose of the object and the end effector. This converts the object's pose trajectory into end effector path points. Specifically, for the object's pose in the base coordinate system... The pose of the end effector in the base coordinate system is: ; in, Indicates the first The homogeneous transformation matrix of the object's coordinate system in the robot's base coordinate system at time n. This represents the initial relative homogeneous transformation matrix of the object coordinate system in the end effector coordinate system. Indicates the first The homogeneous transformation matrix of the end effector coordinate system in the robot base coordinate system at time step. The superscript of the homogeneous transformation matrix represents the target coordinate system, and the subscript represents the source coordinate system.

[0070] S67. Robot Execution. The robot controller moves according to the generated end effector path points to complete tasks such as grasping, pushing, pulling, switching, or tilting.

[0071] like Figure 3 As shown, the overall process of motion representation prediction, rigid body decoding and robot execution in steps S5 to S6 above includes a complete link starting from single-frame RGB-D observation and task description, predicting motion abstraction through the motion representation prediction module (MRP), then through rigid body motion decoding, coordinate transformation and end-point path generation, and finally executed by the robot. VII. Specific Implementation Methods The following uses the task of pouring water from a kettle as an example to illustrate the technical solution of this invention in detail.

[0073] Data Acquisition and Preprocessing. Acquire synchronized RGB-D demonstration sequences of multiple executors completing the kettle-pouring task. Each instance includes an RGB image sequence, a depth image sequence, kettle segmentation results, kettle 3D pose, task label, and executor identifier. Based on the kettle segmentation results and pose information, sample data is taken from the kettle surface. Points, and in Tracking its 3D motion at specific time points yields the motion flow at the center of dense objects. The motion is represented in the kettle coordinate system to eliminate the differences in absolute position in different scenes, allowing the model to focus on the relative motion structure of the kettle itself.

[0074] Motion abstraction training. This involves understanding the motion flow... Input to the motion encoder to obtain 128 point-by-point local features and a global feature The posterior parameter generator outputs 64-dimensional latent variables of task evidence. TEB constraints are used to compress details that are relevant to the executor but do not improve task classification; these details are generated by slot cross-attention. Each source anchor point, and in Motion abstraction is formed in an ordered time phase. .

[0075] The training configuration is as follows: the method (CRAFT) in this embodiment is trained for 200 epochs, with a batch size of 128, using the AdamW optimizer, and the learning rate is preheated to [value missing] in the first 5 epochs. Subsequently, the cosines annealed to , It increases linearly to 0.01 within the first 40 cycles.

[0076] Construct a pouring task source-time prototype. After completing TEB training, process all pouring kettle instances again using a unified method. The index grid performs weighted statistics on source anchor points and time phase units to form a source-time prototype of the pouring task. This process does not require different executors to have the same pouring speed, nor does it require a one-to-one correspondence between manually labeled points on the kettle surface.

[0077] Single-frame RGB-D prediction and execution. During deployment, a single-frame RGB image and depth image containing the kettle are acquired by the camera to construct scene point clouds and kettle point clouds, and the task description "pouring the kettle" is input. MRP predicts the 3D motion, validity state, and internal stage time of 8 source anchors in 4 stages; the kettle pose trajectory is obtained through rigid body motion decoding; the end effector path is generated using camera-robot calibration and initial grasping pose; the robot completes the pouring around the estimated shared rotation axis.

[0078] VIII. Implementation Results and Experimental Basis To verify the technical effectiveness of this invention, the inventors conducted experiments based on the HOI4D dataset. The training set contained 9543 instances, and the validation set contained 1063 instances, covering 41 fine-grained tasks. The experiments evaluated the motion representation quality from four aspects: Task Decodeability (TD), Execution Style Hiding (SL), Geometric-Task Alignment (GTA), and Temporal Structure Alignment (TSA). Higher TD, GTA, and TSA values ​​were considered better, while lower SL values ​​were preferred.

[0079] Table 1 Comparison Experiment of Motion Representation Mass

[0080] As shown in Table 1, the motion abstraction of this invention achieves a TD index of 0.7625, which is significantly better than the 0.6514 of the dense motion field and the 0.6058 of the key point trajectory method, proving that while compressing the representation dimension, task-related motion information is better preserved; the SL index reaches 0.2299, which is significantly better than the comparison method, proving that this invention effectively suppresses the interference of executor-specific information on motion representation; the GTA and TSA indices are both significantly better than the comparison method, proving that this invention establishes a unified spatiotemporal organization structure across instances.

[0081] Table 2. Ablation experiments of TEB and ST-PTC

[0082] As shown in Table 2, all indicators reached their optimal levels when both TEB and ST-PTC modules were used simultaneously. TD reached 0.7625, SL decreased to 0.2299, GTA reached 0.6294, and TSA reached 0.6307, all of which were better than using either module alone, proving that TEB and ST-PTC have a synergistic effect.

[0083] Table 3. Comparison of Prediction Quality of Single-Frame RGB-D Motion Representation

[0084] As shown in Table 3, the MRP prediction results of this invention achieve a TD index of 0.8725, an SL index of 0.1154, a GTA index of 0.7646, and a TSA index of 0.7804, all of which are significantly better than the comparison methods, proving that the motion abstraction quality of single-frame prediction is high and the information loss is small.

[0085] Table 4 Overall Execution Results of the Real Robot

[0086] In real-world robotic experiments, a UFACTORY xArm 7 robotic arm, a UFACTORY Gripper G2 gripper, and an Intel RealSense D435 depth camera were used to cover 9 types of objects and 23 object-action combinations. Each combination underwent 10 independent trials, for a total of 230 executions. Table 4 shows that the overall success rate of the method presented in this invention is 83.5%, significantly better than the 64.3% of the dense motion field method and the 66.5% of the keypoint trajectory method. In highly constrained actions such as opening, closing, pulling, pushing, and tilting, the success rates of the three methods were 45.0%, 47.5%, and 78.8%, respectively; in free-space actions, the success rates were 74.7%, 76.7%, and 86.0%, respectively. These results demonstrate that the present invention exhibits stable and superior performance in various types of tasks.

[0087] IX. System Implementation Examples This invention also provides a human-computer skill transfer system based on object-centric motion abstraction, used to implement the above method. The system includes: The motion abstraction construction module is configured to acquire dense object-centric motion flows from human operation demonstrations, extract task evidence latent variables from these flows through task evidence bottlenecks, and generate [something] based on these task evidence latent variables. Individual source anchor point An ordered time-phase motion abstraction, the motion abstraction being composed of... OK The column is composed of three-dimensional displacement vectors; The source-time prototype constraint module is configured to perform cross-instance structure alignment of the motion abstraction through source-time prototype constraints to obtain the final motion abstraction. The motion representation prediction module is configured to predict the final motion abstraction based on a single-frame RGB-D observation and a task description; The robot execution module is configured to abstract and decode the predicted motion into rigid body motion of the target object and convert it into a robot end effector path.

[0088] The specific implementation methods of each module are consistent with the descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0089] 10. Alternative Implementation Methods Those skilled in the art should understand that the above embodiments are merely preferred embodiments of the present invention, and the present invention can also employ the following alternative embodiments: Object surface sampling can employ any of the following methods: uniform sampling, farthest-point sampling, grid sampling, random sampling within a segmented mask, or curvature-based sampling. The number of sampling points... The value can be adjusted between 64 and 512 depending on the complexity of the task.

[0090] Motion encoders can be implemented using point cloud networks, ensemble encoders, spatiotemporal Transformers, graph neural networks, or combinations thereof.

[0091] Priors for task-related evidence bottlenecks can use standard Gaussian, diagonal covariance Gaussian, or other unconditional priors with explicit compression effects; the core requirement is that the priors are not directly conditional on the task labels.

[0092] Number of source anchor points The number of time stages can be adjusted between 4 and 32 depending on the task complexity. It can be adjusted between 3 and 8.

[0093] Slot cross-attention can be replaced by set queries with task latent variable conditions, dynamic weight pooling, or other mechanisms that can generate source soft assignments.

[0094] The source-time prototype distance can be truncated distance, Chamfer-type distance, Earth Mover-type distance, kernel distance, or other set distances that do not require a one-to-one correspondence.

[0095] Rigid body motion decoding can be achieved using shared axis rotation, general rigid body registration, weighted Procrustes analysis, SVD, optimized pose estimation, or a combination thereof.

[0096] The point cloud backbone can be PTv3, PointNet++, sparse convolutional networks, point cloud Transformer, or other 3D visual feature extraction networks; the text encoder can be a frozen or jointly trained visual-language model.

[0097] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A human-computer skill transfer method based on object-centric motion abstraction, characterized in that, Includes the following steps: S1. Obtain the motion flow at the center of dense objects in the human operation demonstration; S2. Extract task evidence latent variables from the dense object center motion flow through the task evidence bottleneck, wherein the task evidence bottleneck adopts a joint constraint of unconditional prior and task classification loss. S3. Generating latent variables based on the aforementioned task evidence. Individual source anchor point An ordered time-phase motion abstraction, the motion abstraction being composed of... OK The column is composed of three-dimensional displacement vectors; S4. Perform cross-instance structure alignment on the motion abstraction through source-time prototype constraints to obtain the final motion abstraction; S5. Predict the final motion abstraction based on single-frame RGB-D observations and task description; S6. The predicted motion is abstracted and decoded into the rigid body motion of the target object, and then converted into the robot end effector path.

2. The method according to claim 1, characterized in that, In step S1, obtaining the motion flow of the dense object center in the human operation demonstration specifically includes: Based on the target object segmentation results, object pose information, and depth information, samples are taken from the surface of the target object. Each point; the sampling method can be any one of uniform sampling, farthest point sampling, grid sampling, random sampling within a segmented mask, or curvature-based sampling; Track the The point is at Three-dimensional motion at a given time location; Representing all displacements in the target object coordinate system constitutes the dense object center motion flow; the dense object center motion flow is represented as... ,in This indicates the number of sampling points on the surface of the target object. This represents the number of time samples for the motion trajectory, and each three-dimensional vector represents the displacement of the corresponding surface point in the target object's coordinate system.

3. The method according to claim 2, characterized in that, In step S2, extracting latent variables of task evidence from the dense object-centered motion flow through the task evidence bottleneck specifically includes: The motion flow at the center of the dense object Input the motion encoder to obtain point-by-point local motion features. and global motion characteristics The The The motion encoder is implemented using any one or a combination of point cloud network, ensemble encoder, spatiotemporal Transformer or graph neural network; Based on the global motion characteristics The posterior distribution is obtained through the posterior parameter generator. ; From the posterior distribution The latent variables of the task evidence were obtained by sampling. ; With unconditional standard Gaussian prior Constraining the latent variables of the task evidence Calculate the KL divergence loss : ； in, Denotes the KL divergence operator, Represents the identity matrix; Losses categorized by task The constraints enable the motion abstraction to identify task categories; Joint training objectives for: ； in, The balance coefficient is the equilibrium coefficient. The value is linearly increased from 0 to the target value during the initial training period.

4. The method according to claim 3, characterized in that, In step S3, based on the task evidence latent variables generate Individual source anchor point The abstraction of motion over an ordered time phase specifically includes: The latent variables of the task evidence and Each learnable slot is embedded and combined to form One source anchor query vector; The The source anchor point query vector and the point-by-point local motion features Perform slot cross-attention to obtain the source assignment matrix. The source allocation matrix This represents the soft-assigned weight of each source anchor point to each object surface sampling point; According to the source allocation matrix For the dense object center motion flow Perform weighted aggregation to form The bar represents the trajectory; Based on the task evidence latent variables, the stage prediction head is used. Output An orderly time phase ; According to the above An orderly time phase In the The motion abstraction is obtained by interpolating time along the representative trajectory. The .

5. The method according to claim 4, characterized in that, The latent variables of the task evidence and Each learnable slot is embedded and combined to form Each source anchor query vector specifically includes: Using the latent variables of the aforementioned task evidence As a condition, for the above Conditional modulation is performed on learnable slot embeddings to generate semantically relevant data for the current task. One source anchor query vector; The slot cross-attention is calculated by... The source anchor point query vector and the point-by-point local motion features The attention weights between them are used to obtain the source assignment matrix. .

6. The method according to claim 1, characterized in that, In step S4, the cross-instance structural alignment of the motion abstraction through source-time prototype constraints specifically includes: For task categories Collect the original motion abstractions of this task category from the training set, and perform a weighted average of each source-time unit according to slot activity to form the task source-time prototype. : ； ； in, Indicates task category Next The source anchor point is at the first Prototype displacement vector at each time stage Indicates the task category The same source-time unit in all training instances The operation is a weighted average based on slot activity. Representing the abstraction of primitive motion in the first... Source anchor point, the first Three-dimensional displacement vectors at each time stage; For the original motion abstraction of the current instance Generate at most Abstract of candidate motions Candidate layout is generated by replacing the least active source anchor in the current motion abstraction with a more active source anchor in the task prototype, while keeping the source anchor capacity and stage order unchanged. For candidate motion abstraction Calculate its relationship with the task source-time prototype. Source-time prototype distance between : ； in Representing candidate motion abstraction Source-time unit index in Indicates the first The source configuration confidence of each unit, Represents the cutoff distance constant. Represents the task source-time prototype Source-time unit index in Indicates task category The source-time prototype occupies the support set. Indicates the first The unit and the first Structural distance between units This is an operation to find the minimum value; Computing candidate motion abstractions Overall rating : ； in, Representing candidate motion abstraction The sum of the source configuration confidence level, This refers to the distance weighting coefficient. When the candidate motion abstraction The candidate motion abstraction is selected when the following conditions are met. Otherwise, retain the original motion abstraction. : ； ； in, For confidence level decline tolerance, To raise the threshold for the overall score.

7. The method according to claim 1, characterized in that, In step S5, predicting the final motion abstract based on single-frame RGB-D observations and task description specifically includes: Construct the overall scene point cloud and target object point cloud from a single frame of RGB-D observation; Extract features from the task text description; The scene features, target object features, task text features, and camera parameter features are input into the conditional fusion network; Predicted by a dedicated decoder Individual source anchor point The three-dimensional motion, effectiveness state, and internal stage time of each stage.

8. The method according to claim 1, characterized in that, In step S6, the step of abstracting and decoding the predicted motion into rigid body motion of the target object specifically includes: The source anchor point displacement field between adjacent time stages is centered. Singular value decomposition is performed on the centered source anchor displacement field to estimate the shared rotation axis of the target object; The translation assumption and the shared axis rotation assumption are established respectively, and the motion assumption is selected based on the reconstruction residual and the validity of the source anchor point. Based on the selected motion assumptions, the discrete values ​​between adjacent time stages are obtained. Rigid body transformation, through interpolation, obtains the pose trajectory of a continuous object.

9. The method according to claim 1, characterized in that, In step S6, converting the predicted motion abstraction into a robot end effector path specifically includes: The transformation matrix from the camera coordinate system to the robot base coordinate system is obtained through eye-to-hand external calibration; Transform the pose trajectory of the target object into the robot's base coordinate system; Based on the initial relative pose of the object and the end effector, the object's pose trajectory is converted into end effector path points.

10. A human-computer skill transfer system based on object-centric motion abstraction, characterized in that, include: The motion abstraction construction module is configured to acquire dense object-centric motion flows from human operation demonstrations, extract task evidence latent variables from these flows through task evidence bottlenecks, and generate [something] based on these task evidence latent variables. Individual source anchor point An ordered time-phase motion abstraction, the motion abstraction being composed of... OK The column is composed of three-dimensional displacement vectors; The source-time prototype constraint module is configured to perform cross-instance structure alignment of the motion abstraction through source-time prototype constraints to obtain the final motion abstraction. The motion representation prediction module is configured to predict the final motion abstraction based on a single-frame RGB-D observation and a task description; The robot execution module is configured to abstract and decode the predicted motion into rigid body motion of the target object and convert it into a robot end effector path.

Images