Robot skill migration method and system based on multi-view object trajectory prediction

Abstract

The invention relates to the technical field of robot skill migration, in particular to a robot skill migration method and system based on multi-view object trajectory prediction.The method comprises the steps that a first skill migration model is constructed, and the first skill migration model comprises a skill perception coding module A1 and a multi-view trajectory prediction diffusion module B1 which are connected in sequence; a second skill migration model is constructed based on the trained first skill migration model, and the second skill migration model comprises a skill perception coding module A2 and a multi-view trajectory prediction diffusion module B2 which are connected in sequence; inputting a to-be-tested robot multi-view image and a to-be-tested language instruction into the second skill migration model to obtain a multi-view target object prediction trajectory; and according to the multi-view target object prediction trajectory, an executable robot action is generated, and the action in the to-be-tested language instruction is completed. According to the method, robot skill migration can be efficiently and robustly realized.

Description

Technical Field

[0001] The present invention relates to the technical field of robot skill transfer, and in particular to a robot skill transfer method and system based on multi-view object trajectory prediction. Background Art

[0002] Robotic skill transfer—the ability of a robot to learn and adapt to tasks from demonstrations or prior experience—has become a key area of ​​robotics research. However, existing technologies for robotic skill transfer still face numerous challenges and shortcomings. Specifically, while advanced methods such as imitation learning, meta-learning, and continual learning have addressed some of these issues by reusing knowledge and sharing task representations, these methods often suffer from poor robustness in unfamiliar scenarios, require the collection of large amounts of training data, and scale poorly in complex and dynamic environments. This means that in practical robotic deployments, these methods struggle to efficiently and stably cope with environmental changes and task diversity. Furthermore, while methods such as hierarchical skill transfer and differentiable physics simulation show promise, they rely heavily on precise physical modeling and consume significant computational resources. This not only increases development and deployment costs but also limits their feasibility in resource-constrained applications or those requiring rapid iteration.

[0003] In summary, the main problems faced by existing technologies in robot skill transfer include: insufficient robustness in complex dynamic environments, high dependence on large amounts of data, low scalability, and high requirements for accurate modeling and computing resources. Summary of the Invention

[0004] To address the shortcomings of the existing technology, the present invention provides a method and system for robotic skill transfer based on multi-view object trajectory prediction. This method enables efficient and robust robotic skill transfer with good generalization capabilities. The present invention addresses how to effectively fuse multimodal inputs (such as multi-view images and language instructions) and extract task-related skill information from them, thereby overcoming the challenges of feature encoding and information alignment faced by existing methods; how to construct a model that can generate accurate object motion trajectories from multi-camera observations while ensuring its generalization and efficiency across different tasks and scenarios; and how to efficiently merge and map multi-view motion trajectories into a unified three-dimensional space, thereby converting them into executable robotic actions without requiring additional training, thereby reducing training complexity and computational cost.

[0005] On the one hand, a robot skill transfer method based on multi-view object trajectory prediction is provided, including: Constructing a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language instructions; the target object is clamped in the robot's end effector; Constructing a first skill transfer model, the first skill transfer model comprising: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; inputting a training set into the first skill transfer model, training the model, and obtaining a trained first skill transfer model; A second skill transfer model is constructed based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; a multi-view image of the robot to be tested and a language command to be tested are input into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; Based on the predicted trajectory of the target object from multiple perspectives, executable robot actions are generated. The robot completes the actions in the language instructions to be tested according to the executable robot actions.

[0006] On the other hand, a robot skill transfer system based on multi-view object trajectory prediction is provided, including: A training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: robot images from multiple perspectives and preset language instructions; the target object is clamped in the robot's end effector; The training module is configured to: construct a first skill transfer model, the first skill transfer model including: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; input a training set into the first skill transfer model, train the model, and obtain a trained first skill transfer model; The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; input a multi-view image of the robot to be tested and a language command to be tested into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; The execution module is configured to: generate executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completes the actions in the language instructions to be tested according to the executable robot actions.

[0007] The above technical solution has the following advantages or beneficial effects: It breaks through the key limitations faced by existing technologies in robot skill transfer in complex dynamic environments, such as insufficient robustness, high dependence on large amounts of data, low scalability, and over-reliance on precise physical modeling and high computing resources, thereby bringing significant and multiple practical benefits.

[0008] Specifically, the present invention achieves effective fusion of multimodal input and precise capture of task-related skill features through the innovative introduction of a skill perception coding module. This enables the robot's skills to demonstrate excellent robustness and generalization capabilities in complex dynamic environments where object colors, shapes, and scene backgrounds change significantly, greatly improving the adaptability and reliability of the robot system in unfamiliar or unknown scenarios.

[0009] At the same time, the multi-view trajectory prediction diffusion model utilizes comprehensive observation data from multiple camera perspectives and models object trajectories in an efficient latent space, supplemented by the dual optimization of trajectory prediction and skill classification tasks. It not only significantly improves the accuracy and stability of trajectory prediction, but more importantly, greatly reduces the dependence on massive training data, thereby effectively reducing the cost and difficulty of data collection and accelerating the learning and deployment cycle of robot skills.

[0010] In addition, the multi-view object trajectory merging and motion mapping strategy proposed in the present invention can use the accurately predicted two-dimensional trajectories under multiple viewpoints as overall constraints, and directly convert them into three-dimensional motion sequences executable by the robot through efficient nonlinear optimization methods, thereby completely avoiding the additional motion learning or inverse dynamics model training steps that are usually time-consuming and computationally intensive in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0012] Figure 1 This is a flow chart of the method of embodiment 1; Figure 2 is the initial frame image observed at the first viewing angle in Example 1; Figure 3 is the initial frame image observed at the second viewing angle of the first embodiment; Figure 4 The object bounding box recognized in the first perspective in the first embodiment; Figure 5 The object bounding box recognized at the second viewing angle in the first embodiment; Figure 6 The motion trajectory of the object recognized from the first viewing angle in the first embodiment; Figure 7This is the motion trajectory of the object identified from the second viewing angle in Example 1. DETAILED DESCRIPTION

[0013] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0014] Example 1 This embodiment provides a robot skill transfer method based on multi-view object trajectory prediction; The robot skill transfer method based on multi-view object trajectory prediction includes: S101: Constructing a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language instructions; the target object is clamped in the robot's end effector; S102: Constructing a first skill transfer model, the first skill transfer model comprising: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; inputting a training set into the first skill transfer model, training the model, and obtaining a trained first skill transfer model; S103: Constructing a second skill transfer model based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; inputting a multi-view image of the robot to be tested and a language command to be tested into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; S104: generating executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completing the actions in the language instruction to be tested according to the executable robot actions.

[0015] Furthermore, the multi-view trajectory prediction diffusion module B1 includes: a pre-trained encoder, a noise scheduler and a U-net network B11 connected in sequence; the skill perception encoding module A1 in the trained first skill transfer model is regarded as the skill perception encoding module A2; the U-net network B11 in the trained skill transfer model is regarded as the U-net network B12; the multi-view trajectory prediction diffusion module B2 includes: a U-net network B12, a subtractor and a pre-trained decoder connected in sequence.

[0016] For example, the multi-view robot images are images captured by multiple cameras from different angles, focusing on the same robot and a target object placed in front of the robot. The different angles include: a frontal view of the robot, a 45-degree view from the left front of the robot, or a 45-degree view from the right front of the robot. Figure 2 is the initial frame image observed at the first viewing angle in Example 1; Figure 3 This is the initial frame image observed at the second viewing angle of the first embodiment.

[0017] Exemplarily, the target objects placed in front of the robot include, but are not limited to, teapots, kettles, coffee pots, cups, pots, cans, etc.

[0018] Exemplarily, the predicted language instructions include, but are not limited to: Please pour the water in the yellow cup into the red bowl.

[0019] For example, a robot's end effector, also known as an end tool, refers to the actuator device mounted on the end of the robot to complete specific operational tasks. The end effector is the robot's "hand," responsible for performing various movements and operations. Depending on the task requirements, the end effector can adopt different forms and structures, such as grippers, fixtures, and suction cups.

[0020] Furthermore, the working process of the skill perception encoding module A1 is consistent with that of the skill perception encoding module A2; the working process of the skill perception encoding module A1 includes: (11) Receive multi-perspective robot images and preset language instructions to complete the preliminary encoding of multimodal information; (111) Using a pre-trained visual language model, the bounding box of the target object in front of the robot is identified for each robot image from each perspective, and the coordinates of the bounding box are encoded to obtain the position encoding result of the bounding box; (112) Using a pre-trained image-text comparison learning model, the robot image and predicted language instructions of each perspective are encoded separately to obtain the image encoding results and the language instruction encoding results; (113) A multi-layer perceptron is used to encode the skill perception vector to obtain a skill perception encoding result; the skill perception vector is a set of learnable vectors, and the initial skill perception vector is obtained by random initialization; (12) A multi-head self-attention mechanism and a multi-head cross-attention mechanism are used to realize the interaction and fusion between the multimodal encoding results and the skill perception encoding results, and obtain the skill perception feature sequence; among them, the multimodal encoding results include: the position encoding results of the bounding box, the image encoding results and the language instruction encoding results.

[0021] Furthermore, the (111) uses a pre-trained visual language model to identify the bounding box of the target object in front of the robot for each perspective image of the robot, encodes the coordinates of the bounding box, and obtains the position encoding result of the bounding box, including: Receive multi-view RGB-D images and language instructions , and use pre-trained vision-language models (such as Qwen-VL, Qwen Large Vision Language Model) to identify target objects and extract corresponding bounding boxes ; The bounding box of an object is represented by the coordinates of the upper left corner and the lower right corner, denoted as , used to describe the spatial position of the target in the image; The value of each coordinate By position encoding function Map the coordinate value to a dimension The encoding method uses the sine and cosine position encoding form: ; ; in, is the encoding dimension, represents the index of the encoding dimension, The internal variables of the formula are independent of the coordinate values. When is an even number, use the sine formula. When is an odd number, use the cosine formula, is the coordinate value.

[0022] Then, the encoding vectors of the four coordinate values ​​are concatenated in sequence to obtain the complete position encoding of the bounding box of a single object: ; in, The position encoding result of the bounding box, is a numerical sine-cosine encoding function, Represents a splicing operation.

[0023] When the task involves multiple objects, the above process will act on the bounding boxes of each object separately and splice them together to obtain the final encoding results of multiple objects. Figure 4 is the object bounding box recognized from the first perspective in Example 1, Figure 5 This is the object bounding box recognized from the second viewing angle in the first embodiment.

[0024] Furthermore, the (112) uses a pre-trained image-text comparison learning model to encode the robot image and predicted language instructions of each perspective respectively, and obtains image encoding results and language instruction encoding results, including: Use the pre-trained Constrastive Language-Image Pre-training (CLIP) model to encode images and language independently: ; ; in, is the image encoding result, Encoding results for language instructions, and are pre-trained visual encoders and text encoders, respectively. and They are observation images and language instructions respectively.

[0025] Furthermore, the (113) uses a multi-layer perceptron to encode the skill perception vector to obtain the encoding result of the skill perception, including: For skill information, a skill classifier composed of a multi-layer perceptron (MLP) is used to observe the task image. and language instructions Output the corresponding skill index, and then from a set of learnable skill perception vectors Select the corresponding initial skill information, and then use a skill encoder composed of MLP to encode the skill information: ; ; in, is a skill classifier, is the predicted skill index, is the task-related skill perception vector obtained based on the skill index, is a skill encoder, The encoding result representing the skill characteristics.

[0026] Furthermore, (12) adopts the multi-head self-attention mechanism (MHSA) and the multi-head cross-attention mechanism (MHCA) to realize the interaction and fusion between the multimodal encoding results and the skill perception encoding results, and obtains the skill perception feature sequence, including: First, self-attention encoding is performed through MHSA: ; ; ; ; in, represents the multi-head self-attention mechanism, , , , After The result after further encoding.

[0027] The skill feature sequences encoded by MHSA are then input as keys and values, and the remaining feature sequences are used as query inputs. Information exchange is performed through MHCA to promote the alignment and fusion of multimodal information: ; ; ; in, It is a multi-head cross attention mechanism. , , They are , , pass and skill characteristics The result after information interaction.

[0028] All multimodal feature sequences and skill feature sequences fused by MHCA are input into a feed-forward neural network (FFN). The results of each part after FFN processing are Splicing: ; ; ; ; ; in, is a feedforward neural network, , , , It is through The result after further encoding is It is a splicing operation. Represents the final skill-aware feature sequence.

[0029] The beneficial effect of the above technical solution is: using multi-head self-attention (MHSA) and multi-head cross attention (MHCA) mechanisms to realize the interaction and fusion between multimodal information and skill features.

[0030] Self-attention encoding through MHSA aims to capture contextual dependencies within each modality, laying the foundation for subsequent cross-modal fusion. MHSA's parallel processing capabilities enable the model to simultaneously focus on different representation subspaces of feature sequences, thereby more comprehensively understanding the inherent structure of each modality.

[0031] MHCA achieves information exchange between different modalities by calculating the similarity between the query and the key and using the weighted sum of the values. This mechanism enables the model to associate information from other modalities with core skill features, effectively extracting cross-modal knowledge relevant to the task. All multimodal feature sequences and skill feature sequences fused by MHCA are input into a feedforward neural network (FFN) for further nonlinear processing to enhance their information representation capabilities.

[0032] The beneficial effect of the above technical solution is that the skill-aware encoding module A1 solves the feature encoding and information alignment challenges of multimodal input (multi-view images, language instructions and object position bounding boxes) and effectively captures task-specific skill features.

[0033] The beneficial effects of the above technical solution are: introducing a learnable skill perception vector and combining it with a multi-head attention mechanism to effectively align multimodal inputs and accurately capture information related to the skill information belonging to the current task. Specifically, the learnable skill perception vector acts as an "anchor point" to guide the model to focus more on the features most relevant to specific robot skills when processing multi-source information such as images, language, and bounding boxes. This focus on the skills belonging to the task enables the model to effectively cope with complex and dynamic environments, including object colors, object categories, and significant scene differences. The multi-head attention mechanism further enhances this focusing ability, while efficiently exchanging and aligning information between multimodal features. In this way, it can provide robust and adaptive conditions for the subsequent multi-view diffusion model process.

[0034] Furthermore, the multi-view trajectory prediction diffusion module B1 has a working process including: (21) Compress the object trajectories of all viewpoints through the encoder to obtain the encoded object trajectories; (22) Potential representation of object trajectories encoded at each viewpoint , randomly select a time step through the noise scheduler Add noise; (23) The potential representation after adding noise perturbation , input to the conditional denoising model Denoising; the conditional denoising model This is achieved through the U-net network.

[0035] Furthermore, (21) compresses the object trajectories of all viewpoints through an encoder to obtain encoded object trajectories, specifically including: The real annotated object trajectory at each viewpoint in the dataset consists of the trajectories of multiple key points on the object. ,in is the number of viewing angles, the object's motion process lasts M Frame, where the object trajectory of each frame can be represented as a set of two-dimensional coordinate points of multiple key points on the object Then, the object trajectory from a single camera perspective is Represented as a sequence of coordinate point sets:

[0036] This sequence representation enables the model to capture the continuous motion of objects in the time dimension.

[0037] In the forward diffusion phase of the diffusion model, the object trajectories of all views are passed through an encoder based on a variational autoencoder (VAE) Independent compression into latent representation space:

[0038] in, is the object trajectory after encoding.

[0039] Figure 6 The motion trajectory of the object recognized from the first viewing angle in the first embodiment; Figure 7 This is the motion trajectory of the object identified from the second viewing angle in Example 1.

[0040] Furthermore, (22) represents the potential trajectory of the object encoded at each viewpoint. , randomly select a time step through the noise scheduler Add noise, including: Potential representation of object trajectory encoded at each viewpoint , randomly select a time step through the noise scheduler Add noise:

[0041] in, is the noise scheduling parameter, is the noise added each time, This is the result after adding Gaussian noise. As the time steps increase, the noise in the trajectory potential representation gradually accumulates, and eventually the original information is submerged in the Gaussian noise.

[0042] Furthermore, (23) adds the noise-perturbed latent representation , input to the conditional denoising model Denoising; the conditional denoising model This is achieved through the U-net network, including: Skill-based feature sequence As a conditional input, this conditioning mechanism makes the conditional denoising model The skill-aware information is used to guide the denoising process, thereby more accurately predicting the Noise in :

[0043] in, represents the U-net network, represents the predicted noise to be added, The potential representation of the object's trajectory after adding noise, represents the current corresponding time step, It is the skill perception feature sequence.

[0044] The beneficial effect of the above technical solution is that in the reverse denoising stage, the core task of the diffusion model is to gradually recover the original trajectory information from the noisy latent representation. Input into the conditional denoising model based on the U-Net network Through this conditional denoising, the model is able to generate two-dimensional object trajectories that are highly relevant to the specific task and environmental context, significantly improving prediction accuracy and generalization capabilities.

[0045] Exemplarily, the conditional denoising model This is achieved through the U-net network B11, which is an indexing symbol for distinguishing the U-net network and has no special meaning.

[0046] The multi-view trajectory prediction diffusion module B1 aims to generate the two-dimensional motion trajectory of the object to be operated in the task scene from multiple camera perspectives to obtain more comprehensive observations and improve the accuracy of the predicted trajectory.

[0047] In order to facilitate the modeling and efficient processing of the diffusion model, the original high-dimensional trajectory data is first reduced in dimension through an encoder-decoder structure, thereby mapping the original high-dimensional trajectory space into a more compact and informative latent representation space. and decoder Before the diffusion model is trained, it is pre-trained on the dataset to ensure that it can effectively capture the structural features and spatiotemporal patterns of the trajectories.

[0048] Furthermore, the training set is input into the first skill transfer model, and the model is trained to obtain the trained first skill transfer model. The total loss function used in the training process is: The total loss function is the weighted sum of the mean squared error (MSE) loss for trajectory generation and the cross-entropy loss (CE) for skill classification. is the weighting coefficient: ; in, is the noise prediction loss, is the skill classification loss, is the weighting coefficient.

[0049] ; in, Noise added to the noise scheduler, For the denoising model Predicted noise.

[0050] ； in, is the total number of skill categories, is the true label of the skill, The model predicts that the current task belongs to The probability of a skill.

[0051] The beneficial effect of the above technical solution is that the multi-view trajectory prediction diffusion model is trained by minimizing the prediction noise. The actual noise added The mean square error (MSE) between them is used for training, and the loss function is The present invention also uses the output of the skill classifier in the skill perception coding module to An auxiliary loss function is constructed to effectively regularize the skill-aware encoding module and promote the learning of skill representation. The loss function of this part is Using cross entropy loss, where It is the actual skill label.

[0052] Furthermore, the multi-view trajectory prediction diffusion module B2 has a working process including: The denoising model U-net network B12 is applied iteratively to extract the random Gaussian noise Step-by-step prediction of single-step noise , a subtractor is used to remove single-step noise, and then the object trajectory of each view is inferred through the pre-trained decoder.

[0053] Furthermore, the multi-view trajectory prediction diffusion module B2 specifically works as follows: At each camera viewpoint, a random noise sampled from Initially, the denoising model U-net network B12 is used at each time step Iteratively estimate the noise and update the latent representation:

[0054] in, and are predefined scheduling parameters, is the noise predicted by the denoising model, is the noise introduced during the inference process, is the standard deviation of the noise term, Represents the potential representation of the object trajectory after subtracting the noise.

[0055] Iterate T times until the final latent representation of the object trajectory is obtained ; Then, Through a decoder based on a variational autoencoder (VAE) Recover the two-dimensional trajectory of the object on the image at each perspective:

[0056] After obtaining the two-dimensional trajectory of the object under multiple camera perspectives, we can organize it into a set of multi-perspective two-dimensional trajectories: .

[0057] The beneficial effect of this technical solution is that it uses a diffusion model to model object trajectories. By simultaneously optimizing the trajectory prediction task and the skill classification task, the model can better understand the subtle differences between different skills, thereby achieving higher trajectory prediction accuracy. Ultimately, this model can accurately generate two-dimensional motion trajectories of objects from different perspectives based on given task instructions and image observations.

[0058] Multi-view object trajectory merging and action mapping strategy, which aims to accurately map multi-view object trajectories into a unified three-dimensional space and convert them into executable robot actions without additional training. Through the multi-view trajectory prediction diffusion model, a set of two-dimensional trajectories under multiple views can be obtained. .

[0059] Furthermore, the step S104 of generating executable robot actions based on the predicted trajectory of the target object from multiple perspectives includes: (41): From a multi-view two-dimensional trajectory set Randomly select a main perspective , and use the remaining perspectives as auxiliary perspectives Then, at the initial moment , uniformly sample a set of key points from the target object bounding box in the main view , and use the depth information obtained by the RGBD camera to project it into three-dimensional space to obtain three-dimensional coordinates .

[0060] Then, through the pre-calibrated transformation matrix between the main camera perspective and other perspectives , extract the 3D point set corresponding to the key points of the object in other perspectives :

[0061] Obtain the corresponding two-dimensional point set through the projection relationship within each perspective :

[0062] in is the intrinsic parameter matrix of the camera at different viewing angles.

[0063] (42): According to the two-dimensional point set , according to the distance between the two-dimensional point coordinates and the starting point of the trajectory from the two-dimensional trajectory set Retrieve the motion trajectory corresponding to the sampling points in all perspectives .

[0064] The retrieval process can be specifically expressed as:

[0065] in, represents the distance calculation function, Indicates the The starting point of multiple key point trajectories under different viewing angles, Represented by a two-dimensional point set The retrieved The corresponding trajectory.

[0066] According to the minimization of projection error, the object motion transformation matrix of the main perspective is introduced To describe the object from the start time To any subsequent time The three-dimensional rigid body motion trajectory and define a projection error objective function To measure the difference between the predicted two-dimensional trajectory and the estimated three-dimensional trajectory, a nonlinear optimization method (such as the Levenberg-Marquardt algorithm) is used to optimize the projection error objective function. ,

[0067] The estimated transformation matrix is ​​obtained by optimization , three-dimensional transformation matrix It consists of a rotation matrix and a translation vector:

[0068] Among them, the rotation matrix Describes the rotation information of an object in three-dimensional space. It describes the translation vector of the object in three-dimensional space, thereby completing the transformation from two-dimensional plane trajectory to three-dimensional trajectory.

[0069] (43): When mapping 3D trajectories to executable actions, the estimated transformation matrix The time-varying motion sequence of the object is converted into a reference for the motion planning of the end effector.

[0070] First, the three-dimensional trajectory at each moment is transformed through the pre-calibrated coordinate transformation matrix between the camera coordinate system and the robot coordinate system Mapping to the robot base coordinate system:

[0071] in is the trajectory expressed in the form of pose transformation matrix in the camera coordinate system, The trajectory is converted to the robot base coordinate system in the form of a pose transformation matrix. It is pre-calibrated; (44): Setting is the desired initial grasping point of the robot end gripper in the robot base coordinate system, and Is the generated trajectory The starting point of the robot is in the base coordinate system, and an initial transformation is calculated To correct for the offset:

[0072] in is the offset matrix between the robot end gripper and the starting point of the trajectory.

[0073] On this basis, the trajectory of the robot end gripper after correction is expressed as:

[0074] in is the trajectory of the robot end gripper expressed in the form of a pose transformation matrix.

[0075] (45): Combined with the inverse kinematics (IK) solver, these calculated robot gripper posture trajectories are converted into target angles for each joint of the robot. , so that the robot arm can actually move: ; in is the trajectory of the robot end gripper expressed in the form of a pose transformation matrix, is the inverse kinematics solver, is the final robotic arm joint angle.

[0076] The beneficial effect of the above technical solution is that in order to achieve the coordination of the robot gripper and the movement of the object, it is necessary to eliminate the offset between the grasping point and the starting point of the generated trajectory. This ensures that the robot end effector can be accurately aligned with the object at the beginning of the movement. The multi-view prediction results are used as the overall constraint. They jointly participate in optimizing the rigid body transformation matrix of the object to be operated, which can more accurately determine the position and posture of the object in three-dimensional space. In addition, through nonlinear optimization methods, the accurately predicted two-dimensional trajectory can be directly converted into an executable robot action without additional training. This design significantly reduces the implementation complexity and avoids the time-consuming and computationally intensive motion learning stage that is usually required in traditional methods, thereby greatly improving the efficiency and practicality of robot skill transfer.

[0077] Example 2 This embodiment provides a robot skill transfer system based on multi-view object trajectory prediction; The robot skill transfer system based on multi-view object trajectory prediction includes: A training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: robot images from multiple perspectives and preset language instructions; the target object is clamped in the robot's end effector; The training module is configured to: construct a first skill transfer model, the first skill transfer model including: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; input a training set into the first skill transfer model, train the model, and obtain a trained first skill transfer model; The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; input a multi-view image of the robot to be tested and a language command to be tested into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; The execution module is configured to: generate executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completes the actions in the language instructions to be tested according to the executable robot actions.

[0078] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A robot skill transfer method based on multi-view object trajectory prediction, characterized by: include: Constructing a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: multi-view robot images and preset language instructions; the target object is clamped in the robot's end effector; Constructing a first skill transfer model, the first skill transfer model comprising: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; inputting a training set into the first skill transfer model, training the model, and obtaining a trained first skill transfer model; A second skill transfer model is constructed based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; a multi-view image of the robot to be tested and a language command to be tested are input into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; Based on the predicted trajectory of the target object from multiple perspectives, executable robot actions are generated. The robot completes the actions in the language instructions to be tested according to the executable robot actions.

2. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The multi-view trajectory prediction diffusion module B1 includes: a pre-trained encoder, a noise scheduler and a U-net network B11 connected in sequence; the skill perception encoding module A1 in the trained first skill transfer model is regarded as the skill perception encoding module A2; the U-net network B11 in the trained skill transfer model is regarded as the U-net network B12; the multi-view trajectory prediction diffusion module B2 includes: a U-net network B12, a subtractor and a pre-trained decoder connected in sequence.

3. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The working process of the skill perception encoding module A1 and the skill perception encoding module A2 is consistent; The working process of the skill perception encoding module A1 includes: (11) Receive multi-perspective robot images and preset language instructions to complete the preliminary encoding of multimodal information; (111) Using a pre-trained visual language model, the bounding box of the target object in front of the robot is identified for each viewpoint of the robot image, and the coordinates of the bounding box are encoded to obtain the position encoding result of the bounding box; (112) Using a pre-trained image-text comparison learning model, the robot image and predicted language instructions of each perspective are encoded separately to obtain the image encoding results and the language instruction encoding results; (113) A multi-layer perceptron is used to encode the skill perception vector to obtain a skill perception encoding result; the skill perception vector is a set of learnable vectors, and the initial skill perception vector is obtained by random initialization; (12) A multi-head self-attention mechanism and a multi-head cross-attention mechanism are used to realize the interaction and fusion between the multimodal encoding results and the skill perception encoding results, and obtain the skill perception feature sequence; among them, the multimodal encoding results include: the position encoding results of the bounding box, the image encoding results and the language instruction encoding results.

4. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The multi-head self-attention mechanism and the multi-head cross-attention mechanism are used to realize the interaction and fusion between the multimodal encoding results and the skill perception encoding results, and obtain the skill perception feature sequence, including: First, self-attention encoding is performed through MHSA: ； ； ； ； in, represents the multi-head self-attention mechanism, , , , After The results after further coding; The skill feature sequences encoded by MHSA are then input as keys and values, and the remaining feature sequences are used as query inputs. Information exchange is performed through MHCA to promote the alignment and fusion of multimodal information: ； ； ； in It is a multi-head cross attention mechanism. , , They are , , pass and skill characteristics The result after information interaction; All multimodal feature sequences and skill feature sequences fused by MHCA are input into a feedforward neural network FFN. The results of each part after FFN processing are Splicing: ； ； ； ； ； in, , , , It is through The result after further encoding is It is a splicing operation. Represents the final skill-aware feature sequence.

5. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The multi-view trajectory prediction and diffusion module B1 has the following working process: (21) Compress the object trajectories of all viewpoints through the encoder to obtain the encoded object trajectories; (22) Potential representation of object trajectories encoded at each viewpoint , randomly select a time step through the noise scheduler Add noise; (23) The potential representation after adding noise perturbation , input to the conditional denoising model Denoising; the conditional denoising model This is achieved through the U-net network.

6. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: Potential representation of object trajectory encoded at each viewpoint , randomly select a time step through the noise scheduler Add noise, including: Potential representation of object trajectory encoded at each viewpoint , randomly select a time step through the noise scheduler Add noise: ； in, is the noise scheduling parameter, is the noise added each time, This is the result after adding Gaussian noise; as the time step increases, the noise in the potential representation of the trajectory gradually accumulates, and eventually the original information is submerged in the Gaussian noise.

7. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The potential representation after adding noise perturbation , input to the conditional denoising model Denoising; the conditional denoising model This is achieved through the U-net network, including: Skill-based feature sequence As a conditional input, this conditioning mechanism makes the conditional denoising model The skill-aware information is used to guide the denoising process, thereby more accurately predicting the Noise in : ； in, represents the U-net network, represents the predicted noise to be added, The potential representation of the object's trajectory after adding noise, represents the current corresponding time step, It is the skill perception feature sequence.

8. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: The multi-view trajectory prediction and diffusion module B2 specifically works as follows: At each camera viewpoint, a random noise sampled from Initially, the denoising model U-net network B12 is used at each time step Iteratively estimate the noise and update the latent representation: ； in, and are predefined scheduling parameters, is the noise predicted by the denoising model, is the noise introduced during the inference process, is the standard deviation of the noise term, Represents the potential representation of the object trajectory after subtracting the noise once; Iterate T times until the final latent representation of the object trajectory is obtained ; Then, Through a decoder based on variational autoencoder Recover the two-dimensional trajectory of the object on the image at each perspective: ； After obtaining the two-dimensional trajectory of the object under multiple camera perspectives, we can organize it into a set of multi-perspective two-dimensional trajectories: 。 9. The robot skill transfer method based on multi-view object trajectory prediction according to claim 1, characterized in that: Predict trajectories of target objects from multiple perspectives and generate executable robot actions, including: (41): From a multi-view two-dimensional trajectory set Randomly select a main perspective , and use the remaining perspectives as auxiliary perspectives ; Then, at the initial moment , uniformly sample a set of key points from the target object bounding box in the main view , and use the depth information obtained by the RGBD camera to project it into three-dimensional space to obtain three-dimensional coordinates ; Then, through the pre-calibrated transformation matrix between the main camera perspective and other perspectives , extract the 3D point set corresponding to the key points of the object in other perspectives : ； Obtain the corresponding two-dimensional point set through the projection relationship within each perspective : ； in is the intrinsic parameter matrix of the camera at different viewing angles; (43): According to the two-dimensional point set , according to the distance between the two-dimensional point coordinates and the starting point of the trajectory from the two-dimensional trajectory set Retrieve the motion trajectory corresponding to the sampling points in all perspectives ; The retrieval process is specifically expressed as: ； in, represents the distance calculation function, Indicates the The starting point of multiple key point trajectories under different viewing angles, Represented by a two-dimensional point set The retrieved The corresponding trajectory; According to the minimization of projection error, the object motion transformation matrix of the main perspective is introduced To describe the object from the start time To any subsequent time The three-dimensional rigid body motion trajectory and define a projection error objective function To measure the difference between the predicted two-dimensional trajectory and the estimated three-dimensional trajectory, the nonlinear optimization method is used to optimize the projection error objective function , ； The estimated transformation matrix is ​​obtained by optimization , three-dimensional transformation matrix It consists of a rotation matrix and a translation vector: ； Among them, the rotation matrix Describes the rotation information of an object in three-dimensional space. Describes the translation vector of the object in three-dimensional space, thereby completing the transformation from two-dimensional plane trajectory to three-dimensional trajectory; (43): When mapping 3D trajectories to executable actions, the estimated transformation matrix Convert the object's time-varying motion sequence into a reference for the end effector's motion planning; First, the three-dimensional trajectory at each moment is transformed through the pre-calibrated coordinate transformation matrix between the camera coordinate system and the robot coordinate system Mapping to the robot base coordinate system: ； in, is the trajectory expressed in the form of pose transformation matrix in the camera coordinate system, The trajectory is converted to the robot base coordinate system in the form of a pose transformation matrix; the coordinate system transformation matrix It is pre-calibrated; (44): Setting is the desired initial grasping point of the robot end gripper in the robot base coordinate system, and Is the generated trajectory The starting point of the robot is in the base coordinate system, and an initial transformation is calculated To correct for the offset: ； in is the offset matrix between the robot end gripper and the trajectory starting point; On this basis, the trajectory of the robot end gripper after correction is expressed as: ； in is the trajectory of the robot end gripper expressed in the form of a pose transformation matrix; (45): Combined with the inverse kinematics IK solver, these calculated robot gripper posture trajectories are converted into target angles of each robot joint , so that the robot arm can actually move: ； in is the trajectory of the robot end gripper expressed in the form of a pose transformation matrix, is the inverse kinematics solver, is the final robotic arm joint angle.

10. A robot skill transfer system based on multi-view object trajectory prediction, characterized by: include: A training set construction module is configured to: construct a training set, wherein the training set is multimodal data of a known target object's movement trajectory, the multimodal data including: robot images from multiple perspectives and preset language instructions; the target object is clamped in the robot's end effector; The training module is configured to: construct a first skill transfer model, the first skill transfer model including: a skill perception encoding module A1 and a multi-view trajectory prediction and diffusion module B1 connected in sequence; input a training set into the first skill transfer model, train the model, and obtain a trained first skill transfer model; The testing module is configured to: construct a second skill transfer model based on the trained first skill transfer model, the second skill transfer model comprising: a skill perception encoding module A2 and a multi-view trajectory prediction and diffusion module B2 connected in sequence; input a multi-view image of the robot to be tested and a language command to be tested into the second skill transfer model to obtain a multi-view predicted trajectory of the target object; The execution module is configured to: generate executable robot actions based on the predicted trajectory of the target object from multiple perspectives, and the robot completes the actions in the language instructions to be tested according to the executable robot actions.

Images