A Method and System for Robot Skill Transfer Based on Multi-View Object Trajectory Prediction

Abstract

This invention relates to the field of robot skill transfer technology, and in particular to a robot skill transfer method and system based on multi-view object trajectory prediction. The method includes: constructing a first skill transfer model, which includes a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; constructing a second skill transfer model based on the trained first skill transfer model, which includes a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; inputting multi-view images of the robot to be tested and language commands to be tested into the second skill transfer model to obtain predicted trajectories of the target object from multiple perspectives; and generating executable robot actions based on the predicted trajectories of the target object from multiple perspectives to complete the actions in the language commands to be tested. This invention can efficiently and robustly achieve robot skill transfer.

Description

Technical Field

[0001] This invention relates to the field of robot skill transfer technology, and in particular to a robot skill transfer method and system based on multi-view object trajectory prediction. Background Technology

[0002] Robot skill transfer, the ability of a robot to learn and adapt to tasks from demonstrations or prior experience, has become a key area of ​​research in robotics. However, existing technologies still face numerous problems and shortcomings in robot skill transfer. Specifically, while advanced methods such as imitation learning, meta-learning, and continuous learning have addressed some issues through knowledge reuse and shared task representations, these methods often exhibit poor robustness in unfamiliar scenarios, require the collection of large amounts of training data, and suffer from low scalability when facing complex and dynamic environments. This means that in practical robot deployments, these methods struggle to efficiently and stably cope with environmental changes and task diversity. Furthermore, while methods such as hierarchical skill transfer and differentiable physical simulation have shown some potential, they rely excessively on accurate physical modeling and consume significant computational resources. This not only increases development and deployment costs but also limits the feasibility of these methods in resource-constrained applications or those requiring rapid iteration.

[0003] In summary, the main problems faced by existing technologies in robot skill transfer include: insufficient robustness in complex dynamic environments, high dependence on large amounts of data, low scalability, and high requirements for accurate modeling and computing resources. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a robot skill transfer method and system based on multi-view object trajectory prediction. This invention can efficiently and robustly achieve robot skill transfer and has good generalization ability. The invention aims to solve the following problems: how to effectively fuse multimodal inputs (such as multi-view images and language commands) and extract task-related skill information from them, overcoming the challenges of feature encoding and information alignment in existing methods; how to construct a model that can generate accurate object motion trajectories from multi-camera perspectives while ensuring its generalization ability and efficiency in different tasks and scenarios; and how to efficiently merge multi-view motion trajectories and map them to a unified three-dimensional space, thereby converting them into executable robot actions without additional training, reducing training complexity and computational cost.

[0005] On the one hand, a robot skill transfer method based on multi-view object trajectory prediction is provided, including:

[0006] A training set is constructed, which consists of multimodal data of the known trajectory of a target object. The multimodal data includes: multi-view robot images and preset language commands; the target object is held in the robot's end effector.

[0007] Construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model;

[0008] A second skill transfer model is constructed based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; the multi-view images of the robot to be tested and the language commands to be tested are input into the second skill transfer model to obtain the predicted trajectory of the target object from multiple perspectives.

[0009] Based on the predicted trajectory of the target object from multiple perspectives, executable robot actions are generated. The robot then follows these executable robot actions to complete the actions in the language instructions to be tested.

[0010] On the other hand, a robot skill transfer system based on multi-view object trajectory prediction is provided, including:

[0011] The training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language commands; the target object is held in the robot's end effector;

[0012] The training module is configured to: construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model;

[0013] The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; input the multi-view images of the robot to be tested and the language commands to be tested into the second skill transfer model to obtain the predicted trajectory of the target object from multiple perspectives.

[0014] The execution module is configured to generate executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completes the actions in the language instructions to be tested according to the executable robot actions.

[0015] The above technical solution has the following advantages or beneficial effects:

[0016] It overcomes the key limitations of existing technologies in the transfer of robot skills in complex dynamic environments, such as insufficient robustness, high dependence on large amounts of data, low scalability, and excessive reliance on accurate physical modeling and high computing resources, thus bringing significant and multiple practical benefits.

[0017] Specifically, this invention innovatively introduces a skill perception coding module, which enables the effective fusion of multimodal inputs and the accurate capture of task-related skill features. This allows the robot's skills to still exhibit excellent robustness and generalization ability in complex dynamic environments where the color, shape, and background of objects change significantly, greatly improving the adaptability and reliability of the robot system in unfamiliar or unknown scenarios.

[0018] Meanwhile, the multi-view trajectory prediction diffusion model utilizes comprehensive observation data from multiple camera perspectives and performs object trajectory modeling in an efficient latent space. With the dual optimization of trajectory prediction and skill classification tasks, it not only significantly improves the accuracy and stability of trajectory prediction, but more importantly, it greatly reduces the dependence on massive training data, thereby effectively reducing the cost and difficulty of data collection and accelerating the learning and deployment cycle of robot skills.

[0019] Furthermore, the multi-view object trajectory merging and motion mapping strategy proposed in this invention can take the accurately predicted two-dimensional trajectory under multiple views as an overall constraint and directly transform it into a three-dimensional motion sequence that can be executed by the robot through an efficient nonlinear optimization method. This completely avoids the time-consuming and computationally intensive additional motion learning or inverse dynamics model training steps that are usually required in traditional methods. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of the method in Example 1;

[0022] Figure 2 The image shown is the initial frame image observed from the first perspective in Example 1.

[0023] Figure 3 The image shown is the initial frame image observed from the second perspective in Example 1.

[0024] Figure 4 The bounding box of the object identified in the first viewpoint of Example 1;

[0025] Figure 5 The bounding box of the object identified in the second view of Example 1;

[0026] Figure 6 The object's trajectory was identified from the first perspective in Example 1;

[0027] Figure 7 This is the trajectory of the object identified from the second perspective in Example 1. Detailed Implementation

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] Example 1

[0030] This embodiment provides a robot skill transfer method based on multi-view object trajectory prediction;

[0031] Robot skill transfer methods based on multi-view object trajectory prediction include:

[0032] S101: Construct a training set, which is multimodal data of the known target object's movement trajectory. The multimodal data includes: multi-view robot images and preset language commands; the target object is held in the robot's end effector.

[0033] S102: Construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model;

[0034] S103: Construct a second skill transfer model based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; input the multi-view images of the robot to be tested and the language commands to be tested into the second skill transfer model to obtain the multi-view target object prediction trajectory.

[0035] S104: Based on the predicted trajectory of the target object from multiple perspectives, generate executable robot actions. The robot then performs the actions in the language instructions to be tested by following the executable robot actions.

[0036] Further, the multi-view trajectory prediction diffusion module B1 includes: a pre-trained encoder, a noise scheduler, and a U-net network B11 connected in sequence; the skill-aware encoding module A1 in the trained first skill transfer model is regarded as the skill-aware encoding module A2; the U-net network B11 in the trained skill transfer model is regarded as the U-net network B12; the multi-view trajectory prediction diffusion module B2 includes: a U-net network B12, a subtractor, and a pre-trained decoder connected in sequence.

[0037] For example, the multi-view robot images are images taken by multiple cameras from different shooting angles of the same robot and the target object placed in front of the robot. The different shooting angles include: a frontal shooting angle of the robot, a shooting angle at a 45-degree angle to the left front of the robot, or a shooting angle at a 45-degree angle to the right front of the robot. Figure 2 The image shown is the initial frame image observed from the first perspective in Example 1. Figure 3 This is the initial frame image observed from the second perspective of Example 1.

[0038] For example, the target objects placed in front of the robot include, but are not limited to: teapots, kettles, coffee pots, water cups, pots, cans, etc.

[0039] For example, predictive language instructions include, but are not limited to: Please pour the water from the yellow cup into the red bowl.

[0040] For example, a robot's end effector, also known as an end tool, refers to an actuator device installed at the end of a robot to perform specific operational tasks. The end effector is the robot's "hand," responsible for performing various actions and operations. Depending on different task requirements, end effectors can be implemented in different forms and structures, such as grippers, clamps, and suction cups.

[0041] Furthermore, the working process of the skill perception coding module A1 is the same as that of the skill perception coding module A2; the working process of the skill perception coding module A1 includes:

[0042] (11) Receive robot images from multiple perspectives and preset language commands, and complete the initial encoding of multimodal information;

[0043] (111) Using a pre-trained visual language model, the bounding box of the target object in front of the robot is identified for each view of the robot image, and the coordinates of the bounding box are encoded to obtain the position encoding result of the bounding box.

[0044] (112) A pre-trained image-text contrast learning model is used to encode the robot image and the predicted language command for each viewpoint, and the image encoding result and the language command encoding result are obtained.

[0045] (113) A multilayer perceptron is used to encode the skill perception vector to obtain the encoding result of the skill perception; the skill perception vector is a set of learnable vectors, and the initial skill perception vector is obtained by random initialization;

[0046] (12) A multi-head self-attention mechanism and a multi-head cross-attention mechanism are adopted to realize the interaction and fusion between the multimodal coding results and the skill perception coding results, so as to obtain the skill perception feature sequence; among them, the multimodal coding results include: the bounding box position coding results, the image coding results and the language instruction coding results.

[0047] Further, step (111) employs a pre-trained visual language model to identify the bounding box of the target object in front of the robot for each viewpoint of the robot image, encodes the coordinates of the bounding box, and obtains the position encoding result of the bounding box, including:

[0048] Receive multi-view RGB-D images and language instructions It also utilizes pre-trained vision-language models (such as Qwen-VL, Qwen Large Vision Language Model) to identify target objects and extract their corresponding bounding boxes. ;

[0049] The bounding box of an object is represented by the coordinates of its top-left and bottom-right corners, denoted as . , used to describe the spatial location of the target in the image;

[0050] The value of each coordinate Through position encoding function Map coordinate values ​​to a dimension. The vector. The encoding method uses sine and cosine positional encoding:

[0051] ;

[0052] ;

[0053] in, For encoding dimensions, Index representing the encoded dimension, The internal variables of the formula are independent of the coordinate values, when Use the sine formula when the number is even. When the number is odd, use the cosine formula. These are coordinate values.

[0054] Then, the encoded vectors of the four coordinate values ​​are concatenated sequentially to obtain the complete positional encoding of the bounding box of a single object:

[0055] ;

[0056] in, The position encoding result of the bounding box. For a numerical value, the sine and cosine coding functions are... This indicates a splicing operation.

[0057] When the task involves multiple objects, the above process will be applied to the bounding boxes of each object and then concatenated to obtain the final encoding result of multiple objects. Figure 4 This is the bounding box of the object identified from the first viewpoint in Example 1. Figure 5 This is the object bounding box identified from the second viewpoint in Example 1.

[0058] Further, step (112) employs a pre-trained image-text contrast learning model to encode the robot image and predicted language command for each viewpoint, obtaining image encoding results and language command encoding results, including:

[0059] The image and language are independently encoded using a pre-trained contrastive language-image pre-training (CLIP) model:

[0060] ;

[0061] ;

[0062] in, For image encoding results, The result of encoding language instructions. and These are a pre-trained visual encoder and a text encoder, respectively. and These consist of observed images and verbal commands, respectively.

[0063] Furthermore, step (113) employs a multilayer perceptron to encode the skill perception vector, obtaining the encoded result of the skill perception, including:

[0064] For skill information, a skill classifier composed of a multilayer perceptron (MLP) is used to observe the task image. and language instructions Output the corresponding skill index, and then from a set of learnable skill perception vectors Select the corresponding initial skill information, and then use a skill encoder composed of MLP to encode the skill information:

[0065] ;

[0066] ;

[0067] in, It is a skills classifier. For the predicted skill index, This is a task-related skill perception vector obtained based on the skill index. It is a skill encoder. This represents the encoding result of skill characteristics.

[0068] Furthermore, step (12) employs a multi-head self-attention (MHSA) and a multi-head cross-attention (MHCA) mechanism to achieve interaction and fusion between the multimodal encoding results and the skill-aware encoding results, resulting in a skill-aware feature sequence, including:

[0069] First, self-attention encoding is performed using MHSA:

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] in, This indicates a multi-head self-attention mechanism. , , , They are respectively through The result after further encoding.

[0075] Subsequently, the MHSA-encoded skill feature sequence is used as the key and value input, and the remaining feature sequences are used as the query input. Information exchange is then carried out through MHCA to promote the alignment and fusion of multimodal information.

[0076] ;

[0077] ;

[0078] ;

[0079] in, This is a multi-head cross-attention mechanism. , , They are , , pass With skill characteristics The result after information exchange.

[0080] All multimodal feature sequences fused by MHCA, along with skill feature sequences, are input into a feed-forward network (FFN). The results from each part of the FFN processing are then... splicing:

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] in, It is a feedforward neural network. , , , It was through The result after further encoding, It's a splicing operation. This represents the final sequence of skill perception features.

[0087] The beneficial effects of the above technical solution are: to realize the interaction and fusion between multimodal information and skill features by using multi-head self-attention (MHSA) and multi-head cross-attention (MHCA) mechanisms.

[0088] Self-attention encoding using MHSA aims to capture the contextual dependencies within each modality, laying the foundation for subsequent cross-modal fusion. MHSA's parallel processing capability allows the model to simultaneously focus on different representational subspaces of the feature sequence, thereby gaining a more comprehensive understanding of the intrinsic structure of each modality.

[0089] MHCA (Multimodal Matrix Association) achieves information exchange between different modalities by calculating the similarity between queries and keys and using values ​​for weighted summation. This mechanism enables the model to associate information from other modalities with core skill features, effectively extracting task-related cross-modal knowledge. All multimodal feature sequences fused by MHCA, as well as skill feature sequences, are input into a feedforward neural network (FFN) for further nonlinear processing to enhance their information representation capabilities.

[0090] The beneficial effects of the above technical solution are: the skill perception coding module A1 can effectively address the challenges of feature encoding and information alignment for multimodal inputs (multi-view images, language instructions, and object position bounding boxes) and capture task-specific skill features.

[0091] The beneficial effects of the above technical solution are: by introducing learnable skill-aware vectors and combining them with a multi-head attention mechanism, it effectively aligns multimodal inputs and accurately captures information related to the skill information of the current task. Specifically, the learnable skill-aware vectors act as an "anchor," guiding the model to focus more on the features most relevant to a specific robot skill when processing multi-source information such as images, language, and bounding boxes. This focus on the skill of the task enables the model to effectively cope with complex and dynamic environments, including object color, object category, and significant scene differences. The multi-head attention mechanism further enhances this focusing ability, while also enabling efficient information exchange and alignment between multimodal features. In this way, robust and adaptive conditions can be provided for the subsequent multi-view diffusion model process.

[0092] Furthermore, the multi-view trajectory prediction diffusion module B1 operates as follows:

[0093] (21) Compress the object trajectories from all perspectives using an encoder to obtain the encoded object trajectories;

[0094] (22) Latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. Add noise;

[0095] (23) The latent representation after adding noise perturbation Input into the conditional denoising model Denoising is performed; the conditional denoising model This is achieved through the U-net network.

[0096] Furthermore, step (21) compresses the object trajectories from all perspectives using an encoder to obtain the encoded object trajectory, specifically including:

[0097] The object trajectory of the ground truth labeled object in each viewpoint of the dataset consists of the trajectory of multiple key points on the object. For each viewpoint... ,in The number of viewpoints, the duration of the object's motion. M A frame, where the trajectory of an object in each frame can be represented as a set of two-dimensional coordinates of multiple key points on the object. So, the trajectory of an object from the perspective of a single camera. Represented as a sequence of coordinate points:

[0098]

[0099] This sequential representation method enables the model to capture the continuous motion of objects in the time dimension.

[0100] In the forward diffusion phase of the diffusion model, the object trajectories from all viewpoints are passed through an encoder based on a variational autoencoder (VAE). Independent compression into the latent representation space:

[0101]

[0102] in, This is the encoded trajectory of the object.

[0103] Figure 6 The object's trajectory was identified from the first perspective in Example 1; Figure 7 This is the trajectory of the object identified from the second perspective in Example 1.

[0104] Furthermore, the (22) latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. Add noise, including:

[0105] latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. Add noise:

[0106]

[0107] in, These are noise scheduling parameters. It's the noise added each time. This is the result after adding Gaussian noise. As the time step increases, the noise in the trajectory latent representation gradually accumulates, eventually drowning out the original information in the Gaussian noise.

[0108] Furthermore, step (23) adds noise perturbation to the latent representation. Input into the conditional denoising model Denoising is performed; the conditional denoising model This is achieved through the U-net network, including:

[0109] Skill perception feature sequence As a conditional input, this conditionalization mechanism enables the conditional denoising model Using skill-aware information to guide the denoising process, thereby more accurately predicting the additions to the denoising process. Noise in :

[0110]

[0111] in, Indicates U-net network, This represents the predicted added noise. A latent representation of the trajectory of an object after noise has been added. This indicates the current time step. This is a sequence of skill perception features.

[0112] The beneficial effect of the above technical solution is that, in the reverse denoising stage, the core task of the diffusion model is to gradually recover the original trajectory information from the noisy latent representation. The latent representation after adding noise perturbation... The conditional denoising model is input into the U-Net network. Denoising is performed. Through this conditional denoising, the model can generate 2D object trajectories that are highly correlated with the specific task and environmental context, significantly improving the accuracy and generalization ability of predictions.

[0113] For example, conditional denoising model It is implemented through the U-net network B11, where B11 is an index symbol used to distinguish the U-net network and has no special meaning.

[0114] The multi-view trajectory prediction diffusion module B1 is designed to generate the two-dimensional motion trajectory of the object to be manipulated in the task scene from multiple camera views, so as to obtain more comprehensive observation and improve the accuracy of the predicted trajectory.

[0115] To facilitate the modeling and efficient processing of diffusion models, the original high-dimensional trajectory data is first subjected to dimensionality reduction through an encoder-decoder structure, thereby mapping it from the original high-dimensional trajectory space to a more compact and information-rich potential representation space. The encoder... and decoder Before training the diffusion model, it is pre-trained on the dataset to ensure that it can effectively capture the structural features and spatiotemporal patterns of the trajectory.

[0116] Furthermore, the training set is input into the first skill transfer model to train the model, resulting in the trained first skill transfer model. The total loss function used during training is:

[0117] The total loss function is a weighted sum of the mean squared error (MSE) loss from trajectory generation and the cross-entropy loss (CE) from skill classification. Weighting coefficients:

[0118] ;

[0119] in, It is the noise prediction loss. It is a skill classification loss. These are the weighting coefficients.

[0120] ;

[0121] in, The noise added to the noise dispatcher, For denoising model Predicted noise.

[0122] ；

[0123] in, The total number of skill categories, As a true label for skills, The model predicts that the current task belongs to the first... The probability of each skill.

[0124] The beneficial effect of the above technical solution is that the multi-view trajectory prediction diffusion model, during training, minimizes prediction noise. Compared to the actual added noise The training is performed using the mean squared error (MSE) between the two values, and the loss function is... The present invention also utilizes the output of the skill classifier in the skill perception coding module. An auxiliary loss function was constructed to effectively regularize the skill-aware encoding module and facilitate the learning of skill representations. The loss function for this part... Using cross-entropy loss, where These are genuine skill labels.

[0125] Furthermore, the multi-view trajectory prediction diffusion module B2 operates as follows:

[0126] By iteratively applying the U-net network B12 denoising model, random Gaussian noise is extracted. Stepwise prediction of single-step noise The subtractor is used to remove single-step noise, and then the object trajectory of each viewpoint is inferred through the pre-trained decoder.

[0127] Furthermore, the multi-view trajectory prediction diffusion module B2 specifically includes the following working process:

[0128] From a randomly sampled noise at each camera viewpoint Initially, the denoising model U-net network B12 at each time step Iteratively estimate the noise and update the latent representation:

[0129]

[0130] in, and For predefined scheduling parameters, The noise predicted by the denoising model. It is noise introduced during the reasoning process. It is the standard deviation of the noise term. This represents the latent representation of the object's trajectory after subtracting noise once.

[0131] Repeat the iteration T times until the final latent representation of the object trajectory is obtained. ;

[0132] Then, A decoder based on a variational autoencoder (VAE) Restore the 2D trajectory of the object on the image from each viewpoint:

[0133]

[0134] After obtaining the 2D trajectories of the object from multiple camera perspectives, we obtain a set of multi-view 2D trajectories:

[0135] .

[0136] The beneficial effects of the above technical solution are: by using a diffusion model to model object trajectories, and by simultaneously optimizing trajectory prediction and skill classification tasks, the model can better understand the subtle differences between different skills, thereby achieving higher trajectory prediction accuracy. Ultimately, this model can accurately generate the two-dimensional motion trajectory of an object from different perspectives based on given task instructions and image observations.

[0137] A multi-view object trajectory merging and motion mapping strategy is proposed. This strategy aims to accurately map multi-view object trajectories to a unified 3D space and convert them into executable robot actions without additional training. A multi-view trajectory prediction diffusion model can be used to obtain a set of 2D trajectories from multiple perspectives. .

[0138] Further, S104: Based on the predicted trajectory of the target object from multiple perspectives, generate executable robot actions, including:

[0139] (41): From a multi-perspective two-dimensional trajectory set Randomly select a main perspective and use the other perspectives as auxiliary perspectives. Subsequently, at the initial moment... A set of key points is uniformly downsampled within the bounding box of the target object from the main viewpoint. The depth information obtained from the RGBD camera is then projected onto three-dimensional space to obtain three-dimensional coordinates. .

[0140] Next, the transformation matrix between the main camera's viewpoint and other viewpoints is pre-calibrated. Extract the 3D point set corresponding to the key points of the object from other perspectives. :

[0141]

[0142] By using the projection relationships within their respective perspectives, the corresponding two-dimensional point sets are obtained. :

[0143]

[0144] in This represents the intrinsic parameter matrix of the camera under different viewpoints.

[0145] (42): Based on a two-dimensional point set Based on the distance between the coordinates of a two-dimensional point and the starting point of the trajectory, from the set of two-dimensional trajectories Retrieve the motion trajectory corresponding to the sampling points in all viewpoints .

[0146] The retrieval process can be specifically represented as follows:

[0147]

[0148] in, This represents the distance calculation function. Indicates the first The starting point of the trajectory of multiple key points from different perspectives Indicates based on a two-dimensional point set The retrieved data and The corresponding trajectory.

[0149] Based on minimizing projection error, an object motion transformation matrix is ​​introduced from the main viewpoint. To describe the object from the start time To any subsequent time The three-dimensional rigid body motion trajectory is determined, and a projection error objective function is defined. To measure the difference between the predicted 2D trajectory and the estimated 3D trajectory, a nonlinear optimization method (such as the Levenberg-Marquardt algorithm) is used to optimize the projection error objective function. ,

[0150]

[0151] The estimated transformation matrix is ​​obtained through optimization. 3D transformation matrix It consists of a rotation matrix and a translation vector:

[0152]

[0153] Wherein, rotation matrix It describes the rotation information of the object in three-dimensional space. It describes the translation vector of an object in three-dimensional space, thus completing the transformation from a two-dimensional planar trajectory to a three-dimensional trajectory.

[0154] (43): When mapping a 3D trajectory to an executable action, from the estimated transformation matrix The intermediate converter extracts the motion sequence of the object over time, which serves as a reference for the motion planning of the end effector.

[0155] First, the 3D trajectory at each moment is transformed using a pre-calibrated coordinate transformation matrix between the camera coordinate system and the robot coordinate system. Mapped to the robot's base coordinate system:

[0156]

[0157] in The trajectory is represented in the camera coordinate system as a pose transformation matrix. This represents the trajectory, expressed as a pose transformation matrix, transformed into the robot's base coordinate system. Coordinate transformation matrix. It was pre-calibrated;

[0158] (44): Setting It is the desired initial gripping point pose of the robot's end effector in the robot's base coordinate system, while It generates the trajectory. The pose of the starting point in the robot's base coordinate system is then determined by calculating an initial transformation. To correct the offset:

[0159]

[0160] in This is the offset matrix between the robot's end effector and the starting point of the trajectory.

[0161] Based on this, the corrected trajectory of the robot end effector is represented as:

[0162]

[0163] in The trajectory of the robot end effector is represented in the form of a pose transformation matrix.

[0164] (45): Combine the inverse kinematics (IK) solver to transform the calculated robot gripper attitude trajectories into target angles of each joint of the robot. This allows the robotic arm to perform actual movements:

[0165] ;

[0166] in The trajectory of the robot end effector is represented in the form of a pose transformation matrix. For inverse kinematics solvers, This represents the final joint angle of the robotic arm.

[0167] The beneficial effects of the above technical solution are as follows: To achieve coordinated movement between the robot gripper and the object, it is necessary to eliminate the offset between the gripping point and the starting point of the generated trajectory. This ensures that the robot's end effector can be precisely aligned with the object at the start of movement. Multi-view prediction results are used as overall constraints. They collectively participate in optimizing the rigid body transformation matrix of the object to be manipulated, which enables a more accurate determination of the object's position and orientation in three-dimensional space. Furthermore, through nonlinear optimization, the accurately predicted two-dimensional trajectory can be directly converted into executable robot actions without additional training. This design significantly reduces implementation complexity and avoids the time-consuming and computationally intensive action learning stage typically required in traditional methods, thereby greatly improving the efficiency and practicality of robot skill transfer.

[0168] Example 2

[0169] This embodiment provides a robot skill transfer system based on multi-view object trajectory prediction;

[0170] A robot skill transfer system based on multi-view object trajectory prediction includes:

[0171] The training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language commands; the target object is held in the robot's end effector;

[0172] The training module is configured to: construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model;

[0173] The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; input the multi-view images of the robot to be tested and the language commands to be tested into the second skill transfer model to obtain the predicted trajectory of the target object from multiple perspectives.

[0174] The execution module is configured to generate executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completes the actions in the language instructions to be tested according to the executable robot actions.

[0175] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot skill transfer method based on multi-view object trajectory prediction, characterized by: include: A training set is constructed, which consists of multimodal data of the known trajectory of a target object. The multimodal data includes: multi-view robot images and preset language commands; the target object is held in the robot's end effector. Construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model; A second skill transfer model is constructed based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; the multi-view images of the robot to be tested and the language commands to be tested are input into the second skill transfer model to obtain the predicted trajectory of the target object from multiple perspectives. Based on the predicted trajectory of the target object from multiple perspectives, executable robot actions are generated. The robot then performs the actions in the language instructions to be tested by following the executable robot actions. The working process of skill perception coding module A1 and skill perception coding module A2 is the same; the working process of skill perception coding module A1 includes: (11) Receive robot images from multiple perspectives and preset language commands, and complete the initial encoding of multimodal information; (111) Using a pre-trained visual language model, the bounding box of the target object in front of the robot is identified for each view of the robot image, and the coordinates of the bounding box are encoded to obtain the position encoding result of the bounding box. (112) A pre-trained image-text contrast learning model is used to encode the robot image and the predicted language command for each viewpoint, and the image encoding result and the language command encoding result are obtained. (113) A multilayer perceptron is used to encode the skill perception vector to obtain the encoding result of the skill perception; the skill perception vector is a set of learnable vectors, and the initial skill perception vector is obtained by random initialization; (12) A multi-head self-attention mechanism and a multi-head cross-attention mechanism are adopted to realize the interaction and fusion between the multimodal coding results and the skill perception coding results to obtain the skill perception feature sequence; among which, the multimodal coding results include: bounding box position coding results, image coding results and language instruction coding results; The multi-head self-attention mechanism and the multi-head cross-attention mechanism are used to achieve the interaction and fusion between the multimodal encoding results and the skill-aware encoding results, resulting in a skill-aware feature sequence, including: First, self-attention encoding is performed using MHSA: ； ； ； ； in, The position encoding result of the bounding box. For image encoding results, The result of encoding language instructions. The encoding result representing skill characteristics; This indicates a multi-head self-attention mechanism. , , , They are respectively through The result after further encoding; Subsequently, the MHSA-encoded skill feature sequence is used as the key and value input, and the remaining feature sequences are used as the query input. Information exchange is then carried out through MHCA to promote the alignment and fusion of multimodal information. ； ； ； in This is a multi-head cross-attention mechanism. , , They are , , pass With skill characteristics The result after information exchange; All multimodal feature sequences fused by MHCA, along with skill feature sequences, are input into a feedforward neural network (FFN). The results from each part of the FFN processing are then... splicing: ； ； ； ； ； in, , , , It was through The result after further encoding, It's a splicing operation. This represents the final sequence of skill perception features.

2. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, The multi-view trajectory prediction diffusion module B1 includes: a pre-trained encoder, a noise scheduler, and a U-net network B11 connected in sequence; the skill-aware encoding module A1 in the trained first skill transfer model is regarded as the skill-aware encoding module A2; the U-net network B11 in the trained skill transfer model is regarded as the U-net network B12; the multi-view trajectory prediction diffusion module B2 includes: a U-net network B12, a subtractor, and a pre-trained decoder connected in sequence.

3. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, The multi-view trajectory prediction diffusion module B1 operates as follows: (21) Compress the object trajectories from all perspectives using an encoder to obtain the encoded object trajectories; (22) Latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. t Add noise; (23) The latent representation after adding noise perturbation Input into the conditional denoising model Denoising is performed; the conditional denoising model This is achieved through the U-net network.

4. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. t Add noise, including: latent representation of the encoded object trajectory for each viewpoint A time step is randomly selected by the noise scheduler. Add noise: ； in, These are noise scheduling parameters. It's the noise added each time. The result is after adding Gaussian noise; as the time step increases, the noise in the trajectory latent representation gradually accumulates, eventually drowning out the original information in the Gaussian noise.

5. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, The latent representation after adding noise perturbation Input into the conditional denoising model Denoising is performed; the conditional denoising model This is achieved through the U-net network, including: Skill perception feature sequence As a conditional input, this conditionalization mechanism enables the conditional denoising model Using skill-aware information to guide the denoising process, thereby more accurately predicting the additions to the denoising process. Noise in : ； in, This represents the predicted added noise. A latent representation of the trajectory of an object after noise has been added. This indicates the current time step. This is a sequence of skill perception features.

6. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, The multi-view trajectory prediction diffusion module B2, in its specific working process, includes: From a randomly sampled noise at each camera viewpoint Initially, the denoising model U-net network B12 at each time step Iteratively estimate the noise and update the latent representation: ； in, and For predefined scheduling parameters, The noise predicted by the denoising model. It is noise introduced during the reasoning process. It is the standard deviation of the noise term. This represents the latent representation of the object's trajectory after subtracting noise once; Repeat the iteration T times until the final latent representation of the object trajectory is obtained. ; Then, Through a decoder based on a variational autoencoder Restore the 2D trajectory of the object on the image from each viewpoint: ； After obtaining the 2D trajectories of the object from multiple camera perspectives, we obtain a set of multi-view 2D trajectories: 。 7. The robot skill transfer method based on multi-view object trajectory prediction as described in claim 1, characterized in that, Based on the predicted trajectory of the target object from multiple perspectives, executable robot actions are generated, including: (41): From a multi-perspective two-dimensional trajectory set Randomly select a main perspective and use the other perspectives as auxiliary perspectives. Subsequently, at the initial moment A set of key points is uniformly downsampled within the bounding box of the target object from the main viewpoint. The depth information obtained from the RGBD camera is then projected onto three-dimensional space to obtain three-dimensional coordinates. ; Next, the transformation matrix between the main camera's viewpoint and other viewpoints is pre-calibrated. Extract the 3D point set corresponding to the key points of the object from other perspectives. : ； By using the projection relationships within their respective perspectives, the corresponding two-dimensional point sets are obtained. : ； in This represents the intrinsic parameter matrix of the camera under different viewpoints; (42): Based on a two-dimensional point set Based on the distance between the coordinates of a two-dimensional point and the starting point of the trajectory, from the set of two-dimensional trajectories Retrieve the motion trajectory corresponding to the sampling points in all viewpoints ; The retrieval process is specifically represented as follows: ； in, This represents the distance calculation function. Indicates the first The starting point of the trajectory of multiple key points from different perspectives Indicates based on a two-dimensional point set The retrieved data and The corresponding trajectory; Based on minimizing projection error, an object motion transformation matrix is ​​introduced from the main viewpoint. To describe the object from the start time To any subsequent time The three-dimensional rigid body motion trajectory is determined, and a projection error objective function is defined. To measure the difference between the predicted 2D trajectory and the estimated 3D trajectory, a nonlinear optimization method is used to optimize the objective function of the projection error. , ； The estimated transformation matrix is ​​obtained through optimization. 3D transformation matrix It consists of a rotation matrix and a translation vector: ； Wherein, rotation matrix It describes the rotation information of the object in three-dimensional space. It describes the translation vector of an object in three-dimensional space, thus completing the transformation from a two-dimensional planar trajectory to a three-dimensional trajectory; (43): When mapping a 3D trajectory to an executable action, from the estimated transformation matrix The intermediate converter extracts the motion sequence of the object over time, which serves as a reference for the motion planning of the end effector; First, the 3D trajectory at each moment is transformed using a pre-calibrated coordinate transformation matrix between the camera coordinate system and the robot coordinate system. Mapped to the robot's base coordinate system: ； in, The trajectory is represented in the camera coordinate system as a pose transformation matrix. The trajectory, represented by a pose transformation matrix, is transformed into the robot's base coordinate system; coordinate transformation matrix It was pre-calibrated; (44): Setting It is the desired initial gripping point pose of the robot's end effector in the robot's base coordinate system, while It generates the trajectory. The pose of the starting point in the robot's base coordinate system is then determined by calculating an initial transformation. To correct the offset: ； in This is the offset matrix between the robot's end effector and the starting point of the trajectory; Based on this, the corrected trajectory of the robot end effector is represented as: ； in The trajectory of the robot end effector is represented in the form of a pose transformation matrix; (45): Combine the inverse kinematics IK solver to convert the calculated robot gripper attitude trajectories into target angles of each joint of the robot. This allows the robotic arm to perform actual movements: ； in The trajectory of the robot end effector is represented in the form of a pose transformation matrix. For inverse kinematics solvers, This represents the final joint angle of the robotic arm.

8. A robot skill transfer system based on multi-view object trajectory prediction, characterized in that, include: The training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language commands; the target object is held in the robot's end effector; The training module is configured to: construct a first skill transfer model, which includes: a skill perception encoding module A1 and a multi-view trajectory prediction diffusion module B1 connected in sequence; input the training set into the first skill transfer model, train the model, and obtain the trained first skill transfer model; The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model. The second skill transfer model includes: a skill perception encoding module A2 and a multi-view trajectory prediction diffusion module B2 connected in sequence; input the multi-view images of the robot to be tested and the language commands to be tested into the second skill transfer model to obtain the predicted trajectory of the target object from multiple perspectives. The execution module is configured to: predict the trajectory of the target object from multiple perspectives, generate executable robot actions, and the robot completes the actions in the language instructions to be tested according to the executable robot actions; The working process of skill perception coding module A1 and skill perception coding module A2 is the same; the working process of skill perception coding module A1 includes: (11) Receive robot images from multiple perspectives and preset language commands, and complete the initial encoding of multimodal information; (111) Using a pre-trained visual language model, the bounding box of the target object in front of the robot is identified for each view of the robot image, and the coordinates of the bounding box are encoded to obtain the position encoding result of the bounding box. (112) A pre-trained image-text contrast learning model is used to encode the robot image and the predicted language command for each viewpoint, and the image encoding result and the language command encoding result are obtained. (113) A multilayer perceptron is used to encode the skill perception vector to obtain the encoding result of the skill perception; the skill perception vector is a set of learnable vectors, and the initial skill perception vector is obtained by random initialization; (12) A multi-head self-attention mechanism and a multi-head cross-attention mechanism are adopted to realize the interaction and fusion between the multimodal coding results and the skill perception coding results to obtain the skill perception feature sequence; among which, the multimodal coding results include: bounding box position coding results, image coding results and language instruction coding results; The multi-head self-attention mechanism and the multi-head cross-attention mechanism are used to achieve the interaction and fusion between the multimodal encoding results and the skill-aware encoding results, resulting in a skill-aware feature sequence, including: First, self-attention encoding is performed using MHSA: ； ； ； ； in, The position encoding result of the bounding box. For image encoding results, The result of encoding language instructions. The encoding result representing skill characteristics; This indicates a multi-head self-attention mechanism. , , , They are respectively through The result after further encoding; Subsequently, the MHSA-encoded skill feature sequence is used as the key and value input, and the remaining feature sequences are used as the query input. Information exchange is then carried out through MHCA to promote the alignment and fusion of multimodal information. ； ； ； in This is a multi-head cross-attention mechanism. , , They are , , pass With skill characteristics The result after information exchange; All multimodal feature sequences fused by MHCA, along with skill feature sequences, are input into a feedforward neural network (FFN). The results from each part of the FFN processing are then... splicing: ； ； ； ； ； in, , , , It was through The result after further encoding, It's a splicing operation. This represents the final sequence of skill perception features.

Images