A method and apparatus for learning robot operation strategy from human video

Abstract

The application belongs to the technical field of embodied intelligence technology and robot technology, and particularly relates to a method and device for learning robot operation strategy from human video. The method comprises the following steps: acquiring an RGB-D video of a human operation from an egocentric perspective, and converting the RGB-D video into a three-dimensional scene point cloud sequence with time sequence; segmenting a hand point cloud and an object point cloud from the three-dimensional scene point cloud sequence, estimating a hand pose based on the hand point cloud to form a motion trajectory, and after simplification by a key frame, mapping the hand trajectory to a seed action trajectory of a robot end effector through a coordinate system; performing spatial transformation augmentation on the seed action trajectory to generate a plurality of enhanced action trajectories, combining the object point cloud to synthesize a synchronous observation point cloud sequence; and constructing a training data set by using the enhanced action trajectory and the corresponding synthesized observation point cloud sequence, training a 3D diffusion strategy model based on the data set, and obtaining the robot operation strategy.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence technology and robotics, specifically relating to a method and apparatus for learning robot operation strategies from human videos. Background Technology

[0002] Imitation learning is the mainstream paradigm for enabling robots to acquire operational skills. It trains policy models using demonstration data, mapping environmental observations into robot actions. However, the development of this field still faces significant challenges.

[0003] Early mainstream imitation learning methods heavily relied on large-scale, high-quality real-world robot demonstration data. This type of data required remote robot collection by human experts, a time-consuming, labor-intensive process requiring expensive hardware, resulting in extremely high data costs and limiting the scale and diversity of the data. Policy space generalization ability based on limited data was poor. Changes in object pose often led to a sharp drop in performance, requiring repeated data collection under various spatial configurations to improve generalization ability, further exacerbating the cost problem.

[0004] To leverage abundant human video resources, researchers have turned to learning from human demonstrations. However, humans and robots differ significantly in morphology and movement. To address this difference, existing methods often process images at the two-dimensional level, such as erasing human hands and rendering virtual robots using image inpainting techniques. However, these methods are complex, prone to introducing visual artifacts, and, more importantly, lose precise three-dimensional geometric information, making them unsuitable for tasks requiring high-precision three-dimensional interaction. Furthermore, while some studies have attempted to extend demonstration data with synthetic data, these typically still require real robot demonstrations as a starting point, and synthetic visual data often necessitates expensive robot verification, failing to achieve low-cost end-to-end synthesis. Summary of the Invention

[0005] This application proposes a solution for learning robot operation strategies from human videos, aiming to solve the problems in the prior art, such as the lack of three-dimensional information due to reliance on two-dimensional image processing, insufficient spatial generalization ability due to the lack of data augmentation mechanisms, and the inability to synthesize training data end-to-end due to the reliance on real robot data in the process.

[0006] A first aspect of this application provides a method for learning robot operation strategies from human videos, including: Acquire RGB-D videos of human manipulation from an egocentric perspective and convert them into a temporal sequence of 3D scene point clouds; The hand point cloud and the target object point cloud are segmented from the three-dimensional scene point cloud sequence. The hand pose is estimated based on the hand point cloud to form a motion trajectory. After simplification by keyframes, the hand trajectory is redirected to the seed motion trajectory of the robot end effector through coordinate system mapping. Spatial transformation augmentation is performed on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence. A training dataset is constructed using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence. A 3D diffusion strategy model is then trained based on the dataset to obtain the robot's operation strategy.

[0007] In some embodiments of this application, acquiring egocentric human-operated RGB-D video includes: Record synchronized RGB and depth images directly using an RGB-D camera; or The RGB-D video is synthesized by recording RGB video using an RGB camera and generating corresponding depth information from the RGB video using a depth estimation model.

[0008] In some embodiments of this application, the step of segmenting the hand point cloud and the target object point cloud from the three-dimensional scene point cloud sequence includes: Based on the RGB image synchronized with the point cloud, a two-dimensional segmentation mask for the human hand and the target object is obtained using an image segmentation model. The two-dimensional segmentation mask is projected onto a three-dimensional space to obtain the corresponding three-dimensional point cloud clusters from the three-dimensional scene point cloud; Using a multimodal visual language model, semantic recognition is performed on the 3D point cloud clusters based on text prompts to distinguish between hand point clouds and target object point clouds. In some embodiments of this application, the keyframe simplification includes: Analyze the kinematic features of the motion trajectory and the hand interaction state, and identify keyframes based on at least one of the following conditions: Points of change in the opening and closing state of the hand, local minimum points of hand movement velocity, and local maximum points of hand movement acceleration; Temporal redundancy filtering is performed on the identified keyframes. In some embodiments of this application, the redirection of the hand trajectory via coordinate system mapping includes: Define the human hand coordinate system and the robot gripper coordinate system respectively, where: The hand coordinate system is established by multiple predefined semantic hand joints output by the hand pose estimation model. The semantic hand joints include at least 21 joints, and explicitly include joints representing the wrist, the base of the index finger, the base of the middle finger, and the base of the little finger. The human hand coordinate system is defined as follows: the origin is the wrist joint, the positive Y-axis is the direction from the wrist joint to the middle finger root joint, the positive Z-axis is the direction of the vector obtained by the cross product of the vector from the index finger root joint to the little finger root joint and the Y-axis, and the positive X-axis is determined by the cross product of the Y-axis and the Z-axis. The robot gripper coordinate system is defined as follows: the origin is located at the center of the gripper flange, the positive Y-axis is the opening and closing direction of the gripper fingers, the positive Z-axis is the approach direction of the gripper, and the positive X-axis is determined by the cross product of the Y-axis and Z-axis according to the right-hand rule. Pose repositioning is achieved through the following calculations: Position mapping: Set the position of the robot's end effector to the position of the origin of the human hand coordinate system; Attitude mapping: Align the Z-axis of the human hand coordinate system with the Z-axis of the gripper coordinate system, and align the X-axis of the human hand coordinate system with the Y-axis of the gripper coordinate system, in order to calculate the attitude of the robot end effector; By combining the results of position mapping and attitude mapping, a complete transformation relationship from the human hand pose to the robot end effector pose is obtained. In some embodiments of this application, the step of spatially transforming and augmenting the seed motion trajectory to generate multiple enhanced motion trajectories includes: The initial pose of the target object in the seed motion trajectory is randomly sampled to obtain multiple new initial poses; Calculate the transformation matrix of each new initial pose relative to the original initial pose; The transformation matrix is ​​applied to the keyframe poses related to object interaction in the seed motion trajectory to generate the enhanced motion trajectory. In some embodiments of this application, the synthesized synchronized observation point cloud sequence includes: Obtain the standard point cloud model of the robot gripper and the standard point cloud model of the target object; For each time step of the enhanced motion trajectory, perform the following operations: Based on the robot end effector pose at that time step, the standard point cloud model of the gripper is spatially transformed. The pose of the target object at this time step is determined based on the transformation relationship between the initial pose of the target object in the seed motion trajectory and the new initial pose of the target object in the enhanced trajectory. Based on the determined pose of the target object, a spatial transformation is performed on the standard point cloud model of the object; The spatially transformed gripper point cloud and the object point cloud are merged to generate the synthetic observation point cloud for that time step.

[0009] In some embodiments of this application, the 3D diffusion strategy model learns the probability distribution of generating robot action sequences by taking the synthetic observation point cloud sequence as a conditional input. In some embodiments of this application, the training of the 3D diffusion strategy model is achieved by minimizing the noise prediction loss, wherein the noise prediction loss function is: in, To extract from synthetic datasets The motion trajectory samples were collected in the middle. For observing point clouds; For the noise prediction network to be trained, To obtain from the standard normal distribution N(0, I Gaussian noise sampled in the sample; K This represents the number of diffusion steps. For the first k The noisy motion trajectory of the step is given by the formula The calculation yielded, where For predefined noise scheduling parameters, This represents the actual movement trajectory.

[0010] A second aspect of this application provides an apparatus for learning robot operation strategies from human videos, comprising: The conversion module is used to acquire RGB-D videos of human manipulation from an egocentric perspective and convert them into a temporal sequence of 3D scene point clouds; The redirection module is used to segment the hand point cloud and the target object point cloud from the three-dimensional scene point cloud sequence, estimate the hand pose based on the hand point cloud to form a motion trajectory, and after simplification by keyframes, redirect the hand trajectory to the seed motion trajectory of the robot end effector through coordinate system mapping. The synthesis module is used to perform spatial transformation augmentation on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence. The training module is used to construct a training dataset using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence, and to train a 3D diffusion strategy model based on the dataset to obtain the robot operation strategy.

[0011] In summary, the methods and apparatus provided in the embodiments of this application for learning robot operation strategies from human videos bridge the gap between human and robot forms by migrating from human hand operations to robotic arm operations at both the pose and point cloud observation levels. Specifically, trajectory redirection achieves pose mapping from the human hand to the robotic arm end effector, and point cloud camouflage replaces the human hand point cloud in the scene with the robot gripper point cloud, thereby providing a clean and consistent 3D observation input for strategy learning and overcoming the limitations of 2D image processing methods. Trajectory augmentation is achieved by directly applying geometric transformations to the object point cloud in the point cloud space. Because the generated data covers a wide range of object pose changes, the trained strategy possesses spatial generalization capabilities and can adapt to scenes not seen in the initial demonstration. Finally, by synergistically integrating trajectory redirection, point cloud space augmentation, and point cloud camouflage, a new end-to-end, data-efficient paradigm for robot imitation learning is formed, achieving low-cost, end-to-end learning that does not rely on real robot demonstration data. Attached Figure Description

[0012] The features and advantages of this application will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is an algorithmic framework diagram of the robot operation strategy learning method proposed in this application; Figure 2 This is a schematic diagram of a computer system to which this application applies; Figure 3 This is a flowchart illustrating a method for learning robot operation strategies from human videos, according to some embodiments of this application; Figure 4 This is a schematic diagram of an apparatus for learning robot operation strategies from human videos, according to some embodiments of this application. Detailed Implementation

[0013] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0014] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0015] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0016] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0017] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0018] Currently, the mainstream method for enabling robots to master general operational skills is imitation learning, which involves training a policy model that maps observed actions by collecting a large amount of demonstration data. However, existing technologies mainly face the following challenges: Data collection is costly: Mainstream methods rely on large-scale real-world robot demonstration data, typically collected through remote robot operation by human operators. This process is not only time-consuming and labor-intensive but also requires expensive hardware, severely limiting the scale and diversity of the data and becoming a major bottleneck for robot learning.

[0019] Poor spatial generalization ability: Policies trained with limited data are often only effective in scenarios highly similar to the demonstration data. When the position or pose of the target object changes, the policy performance drops sharply, indicating insufficient spatial generalization. To address this issue, data must be repeatedly collected under various spatial configurations, further increasing data costs.

[0020] The differences between human and machine forms are difficult to overcome: To leverage the abundant human video resources available on the internet, some studies have attempted to learn from human videos. However, there is a significant embodiment gap between humans and robots in terms of morphology and movement. Existing methods typically process data at the 2D image level, such as erasing a human hand using image inpainting techniques before rendering a virtual robot. These methods are complex, prone to visual artifacts, and lose precise 3D geometric information, making them unsuitable for tasks requiring high-precision manipulation.

[0021] The synthetic data generation process is incomplete: Some works have attempted to extend demonstrations with synthetic data, but they typically require a real robot demonstration as a starting point, and generating the corresponding visual data may still require costly on-robot rollouts, thus failing to achieve low-cost synthesis.

[0022] In view of this, this application proposes a robot operation strategy learning method. Its core lies in utilizing only egocentric viewpoint human operation videos, achieving cross-morphological data transfer through trajectory redirection and point cloud camouflage, and synthesizing diverse training data using point cloud spatial augmentation. Finally, a robot strategy with strong generalization ability is obtained through imitation learning. The algorithmic framework of the method is as follows: Figure 1 As shown.

[0023] Figure 2 This is a schematic diagram of a computer system to which this application applies. Figure 2 The computer system shown includes a teach-transfer server connected to a human video database and a policy training unit. The teach-transfer server uses human video data from the human video database to construct a large-scale synthetic training dataset. The policy training unit trains the model based on this training dataset, ultimately obtaining the robot's operation policy. Wherein: The human video database stores RGB-D videos of human actions recorded from an egocentric perspective.

[0024] The teaching migration server is used to acquire data from the video library and convert it into a time-series 3D scene point cloud sequence; segment the point cloud and estimate the hand pose to extract the human hand motion trajectory; redirect the human hand trajectory to the seed motion trajectory of the robot end effector through coordinate system mapping; drive the point cloud spatial augmentation module to generate a large number of enhanced motion trajectories; and perform point cloud camouflage to synthesize the corresponding robot observation point cloud sequence, finally constructing a large-scale synthetic training dataset and providing it to the policy training unit for model training.

[0025] The policy training unit receives a synthetic dataset generated by the teaching migration server and uses it to train a 3D diffusion policy model.

[0026] The teaching migration server and policy training unit can be deployed on a single server, server cluster, or distributed computing environment; wherein, the policy training unit can be physically integrated into the teaching migration server or run as an independent policy training server.

[0027] Figure 3 This is a flowchart illustrating a method for learning robot operation strategies from human videos according to some embodiments of this application. In some embodiments, the method for learning robot operation strategies from human videos is... Figure 2 The teaching migration server shown is executed, and the method includes the following steps: S310 acquires RGB-D video of human manipulation from an egocentric perspective and converts it into a temporal sequence of 3D scene point clouds.

[0028] First, an egocentric RGB-D video of human operation is acquired. In some embodiments of this application, the RGB-D video is generated by recording video of a human hand performing a target operation task from the operator's egocentric perspective using an RGB-D camera. In other embodiments, depth information is generated from RGB-only video using a depth estimation model (such as the DepthAnything model) to obtain the RGB-D video.

[0029] Then, for a series of synchronized RGB and depth image frames in the video, each pair of RGB-D frames is converted into a 3D point cloud scene. This yields an original point cloud sequence with temporal information, denoted as... ,in, It is the first Scene point cloud of a frame, It is the total number of frames in the video.

[0030] S320, the hand point cloud and the target object point cloud are segmented from the three-dimensional scene point cloud sequence. The hand pose is estimated based on the hand point cloud to form a motion trajectory. After simplification by keyframes, the hand trajectory is redirected to the seed motion trajectory of the robot end effector through coordinate system mapping.

[0031] In some embodiments of this application, the redirection specifically includes: 1. Point cloud segmentation: For each frame in the original point cloud sequence Perform point cloud segmentation Identify and separate hand point clouds Point cloud of the manipulated object The formal representation is as follows: .

[0032] Point cloud segmentation The specific process is as follows: (1) 2D universal segmentation: Using a pre-trained general image segmentation model SAM (SegmentAnythingModel), the point cloud frames are segmented... Synchronized RGB images The process is performed. SAM outputs a set of candidate 2D segmentation masks. Each mask corresponds to a potential entity in the scene.

[0033] (2) Projection of mask to point cloud cluster: Using camera intrinsic and extrinsic parameters, each 2D mask Projected into 3D space, thus from scene point clouds The corresponding 3D point cloud clusters are segmented from the middle. .

[0034] (3) Semantic recognition: The CLIP multimodal visual language model is used to identify hands and target objects based on text cues. Embedding vectors for the text cues "hand" and "object being manipulated" are calculated separately. For each point cloud cluster... , and its image The visual embedding vector of the corresponding region on the image is calculated. By comparing the cosine similarity between the text embedding and the visual embedding, the point cloud cluster that best matches the "hand" is selected as the target. The best match for the "operated object" is... .

[0035] 2. Attitude estimation: For each frame of RGB image Perform hand pose estimation Specifically, WiLoR, a 3D hand pose estimation model based on a parametric model, is used for localization. The hand was analyzed, and a 3D mesh of the hand and the 3D coordinates of 21 key joints were reconstructed. From these joint coordinates, the position of the palm center and a coordinate system consisting of the palm normal and finger directions were obtained, thus enabling the analysis of the hand's structure. The system accurately calculates the 6D pose of the human hand end in the world coordinate system. ,in For position vectors, Let be the pose quaternion. All poses constitute a continuous hand trajectory: .

[0036] 3. Keyframe simplification: To eliminate noise and extract core motion, keyframe selection is used. Continuous trajectory Simplified to a sparse but crucial sequence of poses , which includes 1 keyframe. Formal representation:

[0037] Keyframe selection The process is as follows: (1) Kinematic analysis: Calculate human hand trajectory Linear velocity of each frame and linear acceleration .

[0038] (2) Interaction state analysis: Estimate the opening and closing state of the hand in each frame Calculate the three-dimensional distance between the tips of the thumb and index finger. When the distance is less than a threshold, the state is closed; otherwise, it is open.

[0039] (3) Keyframe recognition: Traverse all frames, and a frame that meets any of the following conditions is identified as a candidate keyframe: a. Interaction state change point: The frame where the hand opening / closing state changes, i.e. This corresponds to the starting point of the grab or release action.

[0040] b. Start and stop points of movement: local minimum points of hand speed, that is, frames where the speed is close to zero, representing the start, end or pause in the middle of the movement.

[0041] c. Inflection point of motion trajectory: The local maximum value of hand acceleration, which represents a drastic change in the direction or speed of motion, and is an important turning point of the movement.

[0042] (4) Redundant frame filtering: Deduplicat candidate keyframes temporally. If the time interval between two candidate keyframes is less than a preset threshold... If the event is more significant, then only the more significant one will be retained (e.g., retaining the point of change in interaction state rather than the inflection point of motion). At the same time, ensure that the start and end frames of the trajectory are always included.

[0043] 4. Track redirection: Define a method for redirecting the kinematic mapping from a human hand to a robotic gripper. This function achieves pose transformation by aligning the coordinate system of the human hand's palm with the coordinate system of the robot's gripper center. Specifically, it maps the position of the human hand's palm to the position of the gripper center, and aligns the palm normal and finger directions to the approach and grasping directions of the gripper, respectively. This function is applied to each hand keyframe to obtain the seed motion trajectory corresponding to the robot's end effector. The formal representation is as follows:

[0044] Redirect The specific process is as follows: (1) Define the human hand coordinate system : 21 3D key points obtained based on hand pose estimation ( arrive ,in Define a right-handed coordinate system for the wrist: a.Origin : Take key points on the wrist The location.

[0045] bY axis (Main axis): Defined as the axis from the wrist Pointing to the base of the middle finger joint The unit vector represents the main direction of the palm.

[0046] cZ axis (Normal / Approach axis): Through the base of the index finger joint and the joint at the base of the little finger Construct a vector and then perform a cross product with the Y-axis to obtain an approximate palm normal direction, which represents the approach direction when grasping.

[0047] dX axis It is obtained by the cross product of the Y-axis and Z-axis. .

[0048] (2) Define the coordinate system of the robot gripper : The origin of the coordinate system is located at the center of the flange of the gripper, the Z-axis is the approach direction of the tool, and the Y-axis is the opening and closing direction of the gripper.

[0049] (3) Calculate the mapping: For each hand keyframe pose Its rotation matrix in the world coordinate system is The position vector is Redirected robot gripper pose The calculation method is as follows: A. Location mapping: The position of the robot gripper is directly set as the origin of the hand coordinate system: .

[0050] B. Attitude mapping: The robot gripper's orientation is determined by aligning the axes of the two coordinate systems. The final rotation matrix... Make Z-axis alignment Align the Z and Y axes The X-axis.

[0051] S330, spatial transformation augmentation is performed on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence.

[0052] First, enhance the motion trajectory space: Record the initial pose of the object in the seed demo. In a virtual 3D workspace, the initial pose of the manipulated object is randomly sampled N times to obtain a new set of initial poses. For each new initial pose, calculate its transformation matrix relative to the seed pose: .

[0053] To maintain the relative pose between the robot's end effector and the object during interaction, the transformation matrix will be... Left multiply by the seed action trajectory The keyframe poses related to all object interactions are used to generate a new, spatially valid, and enhanced motion trajectory that matches the pose of the new object. , .

[0054] Thus, expanding from a seed trajectory Robot motion trajectories with diverse spatial positions .

[0055] Then, point cloud synthesis and point cloud camouflage are performed: For each enhanced motion trajectory Generate a corresponding, time-synchronized sequence of observation point clouds. Prepare a standard point cloud model of the object to be manipulated in advance. Standard point cloud model of robot gripper For trajectory Each time step in (Obtained through keyframe interpolation): (1) Obtain the end effector pose of the robot gripper at this time step. .

[0056] (2) Calculate the pose of the manipulated object at that time step based on the motion history and physical constraints. .

[0057] (3) Point cloud camouflage: The scene is synthesized by combining the transformed standard model. First, the standard point cloud model of the robot gripper is... By pose First, perform the transformation; second, use the standard point cloud model of the object being manipulated. By pose The transformation is performed; finally, the two transformed point clouds are merged to generate the synthetic observation point cloud for that time step. This process is equivalent to replacing the original hand point cloud in the scene with a point cloud from a robotic gripper. Point cloud camouflage can be formally represented as: .

[0058] This process directly replaces the point cloud of the human hand with the point cloud of the robot gripper in three-dimensional space, making up for the difference in morphological domain between the human hand and the end of the robotic arm, and avoiding the complexity and artifact problems of 2D image processing.

[0059] S340, a training dataset is constructed using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence, and a 3D diffusion strategy model is trained based on the dataset to obtain the robot operation strategy.

[0060] Utilizing enhanced motion trajectories A large-scale, fully synthetic training dataset was constructed using the corresponding observation point cloud sequences. Each sample contains an observation point cloud sequence and a corresponding augmented motion trajectory.

[0061] Using the synthetic dataset To train a 3D diffusion strategy model The model learns from point cloud observations. Under the given conditions, the probability distribution of generating action trajectory A The training objective is to minimize a noisy prediction loss function. In the diffusion process... Steps, noisy movements From real action and Gaussian noise constitute: .

[0062] in These are predefined noise scheduling parameters. Policy network. The task is to base the actions on the noise. diffusion steps and observation To predict the added noise The training loss function Defined as:

[0063] By optimizing this loss function, the model Ultimately, it learns to generate complete, executable action sequences from initial observations.

[0064] One embodiment of this application implements the method described in S310-S340 on a system integrating specific hardware and software. The hardware and software configuration of the system is as follows: 1. Hardware Configuration (1) Computing device (PC): a. Processor (CPU): Intel Core i9-13900K or equivalent, used for handling routine computing tasks and system scheduling.

[0065] b. Graphics Processing Unit (GPU): NVIDIA GeForce RTX 4090 (24GB VRAM) or higher, used to accelerate the training and inference of deep learning models, as well as the parallel processing of point cloud data.

[0066] c. Memory (RAM): 64GB DDR5 to ensure sufficient cache space when processing large-scale point cloud data and training deep models.

[0067] d. Storage: 2TB NVMe SSD for fast access to video data, point cloud files, and model checkpoints.

[0068] (2) Data acquisition equipment: An Intel RealSense D435i or equivalent RGB-D camera is available. This camera can simultaneously output high-resolution color (RGB) and depth images, and has a built-in inertial measurement unit (IMU) to provide the raw data necessary for generating 3D point clouds and attitude estimation. The camera can be handheld or mounted on the operator's chest / head-mounted device to simulate an egocentric perspective.

[0069] (3) Robot platform: a. Robotic arm: Franka Emika Panda seven-DOF collaborative robotic arm, or any six-axis / seven-axis robotic arm with similar workspace and load capacity.

[0070] b. End effector: Robotiq2F-85 two-finger parallel gripper, or a gripper with similar function.

[0071] c. Communication: The computing device is connected to the robot controller via Ethernet to enable the sending of real-time control commands and the receiving of status data.

[0072] 2. Software Configuration The embodiments described herein implement the methods described in S310-S340 through a series of open-source software libraries and customized scripts. The system runs on the Ubuntu 20.04 operating system and uses Python 3.8 as the main programming language.

[0073] (1) Data acquisition and preprocessing module: The IntelRealSenseSDK (librealsense) Python interface is used to capture RGB and depth data streams. The data acquisition script synchronously records RGB frames, depth frames, and camera intrinsics. After acquisition, the Open3D library is used to convert each pair of RGB-D frames into a colored 3D point cloud file (such as .pcd or .ply format), forming the raw point cloud sequence.

[0074] (2) Hand trajectory redirection module: 1. Point Cloud Segmentation: First, load the RGB image and feed it into the SegmentAnythingModel (SAM) model to obtain candidate 2D masks. Then, using camera intrinsics, project these masks back onto the depth map to filter out the corresponding 3D points, thereby completing the segmentation of the point cloud cluster in Open3D.

[0075] 2. Pose estimation: The segmented hand image region is fed into the WiLoR model to infer the parameters of the MANO model and reconstruct 21 three-dimensional hand key points.

[0076] 3. Trajectory Redirection and Keyframes: Based on the algorithm detailed in Part 5, "Methods, Steps, and Processes," NumPy is used to calculate velocity, acceleration, and fingertip distance to automatically extract keyframes. Coordinate system transformation and pose mapping are performed using the spatial transformation libraries in NumPy and SciPy.

[0077] (3) Point cloud spatial augmentation and compositing module: First, the seed motion trajectory and the standard point cloud model of the object being manipulated are loaded. A random SE(3) transformation matrix is ​​generated using NumPy to sample the initial pose of the object. Then, the enhanced motion trajectory is calculated based on these transformations. At each time step, the script calls Open3D's transformation functions to transform the standard point cloud models of the gripper and the object to the calculated pose. Finally, the point cloud merging function is used to generate the final synthetic observation point cloud.

[0078] (4) 3D diffusion strategy training module: a. Model architecture: The backbone network of the policy model can use PointNet++ or sparse convolutional networks to process point cloud inputs, and combine it with a noise prediction network based on Transformer or U-Net.

[0079] b. Training Loop: Following the diffusion model training process detailed in S340, write a standard PyTorch training script. Use the AdamW optimizer, combined with a learning rate scheduler, to optimize the model. Accelerate the training process using a GPU.

[0080] With the above-mentioned hardware and software configuration, the method proposed in this application can be implemented completely and efficiently, realizing end-to-end learning from human videos to robot operation strategies.

[0081] Figure 4 This is a schematic diagram of an apparatus for learning robot operation strategies from human videos, according to some embodiments of this application. Figure 4 As shown, the apparatus 400 for learning robot operation strategies from human videos includes a conversion module 410, a redirection module 420, a synthesis module 430, and a training module 440. In some embodiments of this application, the function of learning robot operation strategies from human videos is... Figure 2 The teaching migration server shown is executed, wherein the apparatus includes: The conversion module 410 is used to acquire RGB-D video of human manipulation from an egocentric perspective and convert it into a temporal sequence of 3D scene point clouds; The redirection module 420 is used to segment the hand point cloud and the target object point cloud from the three-dimensional scene point cloud sequence, estimate the hand pose based on the hand point cloud to form a motion trajectory, and after simplification by keyframes, redirect the hand trajectory to the seed motion trajectory of the robot end effector through coordinate system mapping. The synthesis module 430 is used to perform spatial transformation augmentation on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence. The training module 440 is used to construct a training dataset using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence, and to train a 3D diffusion strategy model based on the dataset to obtain the robot operation strategy.

[0082] In summary, the methods and apparatus provided in the embodiments of this application for learning robot operation strategies from human videos bridge the gap between human and robot forms by migrating from human hand operations to robotic arm operations at both the pose and point cloud observation levels. Specifically, trajectory redirection achieves pose mapping from the human hand to the robotic arm end effector, and point cloud camouflage replaces the human hand point cloud in the scene with the robot gripper point cloud, thereby providing a clean and consistent 3D observation input for strategy learning and overcoming the limitations of 2D image processing methods. Trajectory augmentation is achieved by directly applying geometric transformations to the object point cloud in the point cloud space. Because the generated data covers a wide range of object pose changes, the trained strategy possesses spatial generalization capabilities and can adapt to scenes not seen in the initial demonstration. Finally, by synergistically integrating trajectory redirection, point cloud space augmentation, and point cloud camouflage, a new end-to-end, data-efficient paradigm for robot imitation learning is formed, achieving low-cost, end-to-end learning that does not rely on real robot demonstration data.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0084] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0085] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0086] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for learning robot operation strategies from human videos, characterized in that, include: Acquire RGB-D videos of human manipulation from an egocentric perspective and convert them into a temporal sequence of 3D scene point clouds; The hand point cloud and the target object point cloud are segmented from the three-dimensional scene point cloud sequence. The hand pose is estimated based on the hand point cloud to form a motion trajectory. After simplification by keyframes, the hand trajectory is redirected to the seed motion trajectory of the robot end effector through coordinate system mapping. Spatial transformation augmentation is performed on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence. A training dataset is constructed using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence, and a 3D diffusion strategy model is trained based on the dataset to obtain the robot operation strategy. The 3D diffusion strategy model uses the synthetic observation point cloud sequence as a conditional input to learn the probability distribution of generating robot action sequences. The training of the 3D diffusion strategy model is achieved by minimizing the noise prediction loss, the noise prediction loss function being: in, To extract from synthetic datasets The motion trajectory samples were collected in the middle. For observing point clouds; For the noise prediction network to be trained, To obtain from the standard normal distribution N(0, I Gaussian noise sampled in the sample; K This represents the number of diffusion steps. For the first k The noisy motion trajectory of the step is given by the formula The calculation yielded, where For predefined noise scheduling parameters, This represents the actual movement trajectory.

2. The method according to claim 1, characterized in that, The acquisition of egocentric viewpoint human-operated RGB-D video includes: Record synchronized RGB and depth images directly using an RGB-D camera; or The RGB-D video is synthesized by recording RGB video using an RGB camera and generating corresponding depth information from the RGB video using a depth estimation model.

3. The method according to claim 1, characterized in that, The step of segmenting the hand point cloud and the target object point cloud from the three-dimensional scene point cloud sequence includes: Based on the RGB image synchronized with the point cloud, a two-dimensional segmentation mask for the human hand and the target object is obtained using an image segmentation model. The two-dimensional segmentation mask is projected onto a three-dimensional space to obtain the corresponding three-dimensional point cloud clusters from the three-dimensional scene point cloud; Using a multimodal visual language model, semantic recognition is performed on the 3D point cloud clusters based on text prompts to distinguish between hand point clouds and target object point clouds.

4. The method according to claim 1, characterized in that, The keyframe simplification includes: Analyze the kinematic features of the motion trajectory and the hand interaction state, and identify keyframes based on at least one of the following conditions: Points of change in the opening and closing state of the hand, local minimum points of hand movement velocity, and local maximum points of hand movement acceleration; Temporal redundancy filtering is performed on the identified keyframes.

5. The method according to claim 1, characterized in that, The redirection of the human hand trajectory through coordinate system mapping includes: Define the human hand coordinate system and the robot gripper coordinate system respectively, where: The hand coordinate system is established by multiple predefined semantic hand joints output by the hand pose estimation model. The semantic hand joints include at least 21 joints, and explicitly include joints representing the wrist, the base of the index finger, the base of the middle finger, and the base of the little finger. The human hand coordinate system is defined as follows: the origin is the wrist joint, the positive Y-axis is the direction from the wrist joint to the middle finger root joint, the positive Z-axis is the direction of the vector obtained by the cross product of the vector from the index finger root joint to the little finger root joint and the Y-axis, and the positive X-axis is determined by the cross product of the Y-axis and the Z-axis. The robot gripper coordinate system is defined as follows: the origin is located at the center of the gripper flange, the positive Y-axis is the opening and closing direction of the gripper fingers, the positive Z-axis is the approach direction of the gripper, and the positive X-axis is determined by the cross product of the Y-axis and Z-axis according to the right-hand rule. Pose repositioning is achieved through the following calculations: Position mapping: Set the position of the robot's end effector to the position of the origin of the human hand coordinate system; Attitude mapping: Align the Z-axis of the human hand coordinate system with the Z-axis of the gripper coordinate system, and align the X-axis of the human hand coordinate system with the Y-axis of the gripper coordinate system, in order to calculate the attitude of the robot end effector; By combining the results of position mapping and attitude mapping, a complete transformation relationship from the human hand pose to the robot end effector pose is obtained.

6. The method according to claim 1, characterized in that, The step of spatially transforming and augmenting the seed motion trajectory to generate multiple enhanced motion trajectories includes: The initial pose of the target object in the seed motion trajectory is randomly sampled to obtain multiple new initial poses; Calculate the transformation matrix of each new initial pose relative to the original initial pose; The transformation matrix is ​​applied to the keyframe poses related to object interaction in the seed motion trajectory to generate the enhanced motion trajectory.

7. The method according to claim 1, characterized in that, The synthesized and synchronized observation point cloud sequence includes: Obtain the standard point cloud model of the robot gripper and the standard point cloud model of the target object; For each time step of the enhanced motion trajectory, perform the following operations: Based on the robot end effector pose at this time step, perform spatial transformation on the standard point cloud model of the gripper; The pose of the target object at this time step is determined based on the transformation relationship between the initial pose of the target object in the seed motion trajectory and the new initial pose of the target object in the enhanced trajectory. Based on the determined pose of the target object, perform spatial transformation on the standard point cloud model of the object; The spatially transformed gripper point cloud and the object point cloud are merged to generate the synthetic observation point cloud for that time step.

8. An apparatus for learning robot operation strategies from human videos, characterized in that, include: The conversion module is used to acquire RGB-D videos of human manipulation from an egocentric perspective and convert them into a temporal sequence of 3D scene point clouds; The redirection module is used to segment the hand point cloud and the target object point cloud from the three-dimensional scene point cloud sequence, estimate the hand pose based on the hand point cloud to form a motion trajectory, and after simplification by keyframes, redirect the hand trajectory to the seed motion trajectory of the robot end effector through coordinate system mapping. The synthesis module is used to perform spatial transformation augmentation on the seed motion trajectory to generate multiple enhanced motion trajectories. For each enhanced motion trajectory, at the three-dimensional scene point cloud level, the corresponding hand point cloud is replaced with the standard point cloud of the robot gripper in the same pose, and combined with the target object point cloud to synthesize a synchronized observation point cloud sequence. The training module is used to construct a training dataset using the enhanced motion trajectory and the corresponding synthetic observation point cloud sequence, and to train a 3D diffusion strategy model based on the dataset to obtain the robot operation strategy. The 3D diffusion strategy model uses the synthetic observation point cloud sequence as a conditional input to learn the probability distribution of generating robot action sequences. The training of the 3D diffusion strategy model is achieved by minimizing the noise prediction loss, the noise prediction loss function being: in, To extract from synthetic datasets The motion trajectory samples were collected in the middle. For observing point clouds; For the noise prediction network to be trained, To obtain from the standard normal distribution N(0, I Gaussian noise sampled in the sample; K This represents the number of diffusion steps. For the first k The noisy motion trajectory of the step is given by the formula The calculation yielded, where For predefined noise scheduling parameters, This represents the actual movement trajectory.

Images