A robot motion skill learning method fusing text instructions and motion information

Abstract

This invention discloses a robot motion skill learning method that integrates text instructions and motion information, relating to the field of robot motion planning. The method includes the following steps: S1, acquiring the action sequence during robot movement, merging it into a dataset, preprocessing it, extracting keyframes from the action sequence, and simultaneously acquiring the text information corresponding to the task; S2, extracting the visual and text features of the task respectively; S3, fusing the text features and visual features, continuously updating and optimizing the dependency relationship between text and task through learning the text information, enabling the robot to better understand the task through text and predict future robot actions; S4, further understanding the dependency relationship between text information and actions through learning the text information. This invention combines text information with visual information, predicting the next action by using text information, past multi-view visual information, and previous action information of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot motion planning, and more specifically, to a method for learning robot motion skills that integrates text instructions and motion information. Background Technology

[0002] With the continuous development of modern technology, robots have played an important role in many fields. Traditional robots can no longer meet people's needs. The combination of robot motion planning and human-computer interaction has become a research hotspot in robot motion planning.

[0003] In human environments, robots are expected to perform complex tasks based on language instructions in order to replace humans. However, due to the different target shapes, operation times, and task complexities in different tasks, it is difficult to learn a single skill to meet the needs of different tasks. Furthermore, it is difficult to extend the learned skills to unlearned tasks, resulting in poor generalization performance.

[0004] Currently, there are two input formats: text data and visual data. Text data consists of text information corresponding to each task, which can be one set or multiple sets of information. Visual data is based on RGBD data format information and mask data information captured by multiple cameras. Researching the fusion of text and visual information to enable robots to better understand human language and complete corresponding tasks based on language commands remains a significant challenge.

[0005] There are currently no effective solutions to the problems in the relevant technologies. Summary of the Invention

[0006] To address the problems in related technologies, this invention proposes a robot motion skill learning method that integrates text instructions and motion information, in order to overcome the aforementioned technical problems existing in the current related technologies.

[0007] Therefore, the specific technical solution adopted by the present invention is as follows:

[0008] A method for learning robot motor skills by integrating text instructions and motion information, the method comprising the following steps:

[0009] S1. Obtain the action sequence during the robot's movement, merge it into a dataset, preprocess it, extract the keyframes of the action sequence, and obtain the text information corresponding to the task.

[0010] S2. Extract the visual and textual features of the task respectively;

[0011] S3. Integrate textual features with visual features. By learning from textual information, continuously update and optimize the dependency relationship between text and task, enabling the robot to better understand the task through text and predict future robot actions.

[0012] S4. By learning from text information, we can further understand the dependency between text information and actions, and optimize the action sequence.

[0013] Furthermore, the action sequences during the robot's movement are acquired, merged into a dataset, and preprocessed to extract keyframes from the action sequences. Simultaneously, the text information corresponding to the task is obtained, including the following steps:

[0014] S11. Through teaching and learning, control the robot to complete the set task, obtain visual and motion information in continuous motion state, and merge the visual and motion information into the dataset of the task.

[0015] S12. Sample the continuous action sequence during robot operation and extract keyframes of visual information from the dataset, including the start frame, intermediate frames where the end effector speed tends to stop, and intermediate frames where the end effector state changes. The aim is to limit the task sequence length to within 10 macrosteps.

[0016] S13. Obtain the set of text information corresponding to the task. The purpose is to learn different texts so that the robot can perform the task after receiving different text information.

[0017] Furthermore, extracting the visual and textual features of the task separately includes the following steps:

[0018] S21. First, the visual information is encoded using a global feature encoder and a local feature encoder, respectively. Then, a visual information fusion encoder is constructed to achieve multi-scale feature information fusion.

[0019] S22. Extract text features from text information using a text encoder based on the CLIP pre-trained model.

[0020] Furthermore, firstly, a global feature encoder and a local feature encoder are used to encode the visual information, and then a visual information fusion encoder is constructed to achieve multi-scale feature information fusion, including the following steps:

[0021] S211. Treat the overall environment composed of all viewpoints in the visual information as multi-view image information, and treat the local environment composed of a single viewpoint in the visual information as local image information.

[0022] S212. Construct a global feature module and extract global feature information from multi-view image information using projection and coordinate transformation.

[0023] S213. Construct a local feature module and extract local feature information from local image information using feature mapping.

[0024] S214. A visual information fusion encoder is formed by combining global feature modules and local feature modules. Then, by combining global feature information and local feature information, the encoding of visual information is achieved by using a multi-scale fusion method of global and local information.

[0025] Furthermore, a global feature module is constructed, and global feature information is extracted from multi-view image information using projection and coordinate transformation, including the following steps:

[0026] S2121. Extract multi-scale features from multi-view image information using the backbone of a convolutional neural network;

[0027] S2122. Initialize the bird's-eye view feature, which is uniformly divided into a grid, with each bird's-eye view grid corresponding to a three-dimensional coordinate P. i = (x i y i (z) and query vector z represents the predefined height of the bird's-eye view plane shared by all query vectors;

[0028] S2123. Using geometric prior knowledge to guide the attention model to focus on the distinguishable areas in the bird's-eye view, using camera parameters, the three-dimensional coordinates of each bird's-eye view grid are projected to a set of floating two-dimensional coordinates, and then the two-dimensional coordinates are rounded to the global feature coordinates of the bird's-eye view.

[0029] S2124. Expand a Kh×Kw kernel region around the prior global feature coordinates of the bird's-eye view, where Kh and Kw represent the variable height and length parameters, respectively. If the kernel region exceeds the image boundary of the multi-view image information, the excess part is set to zero. Then, use the attention model to interact with the query embedding of each bird's-eye view and the corresponding multi-view image information feature expansion kernel feature to generate a bird's-eye view representation.

[0030] S2125. To enhance the robustness of phase bias, camera bias is decomposed into translation bias and rotation bias, and random noise is added to all x, y, z dimensions and all cameras.

[0031] The expression for the translational deviation is:

[0032] ;

[0033] The expression for rotational deviation is:

[0034] ;

[0035] In the formula, Tdevi Indicates translational deviation; R devi Represents rotational deviation; Δx, Δy, and Δz represent translational noise along the x, y, and z axes, respectively; R θx R θy and R θz These represent the rotational deviation matrices for the x-axis, y-axis, and z-axis, respectively.

[0036] Furthermore, constructing a local feature module and extracting local feature information from local image information using feature mapping includes the following steps:

[0037] S2131. Obtain the RGB image data, point cloud data, and focus coordinate data corresponding to the local image information captured by each camera at time t.

[0038] S2132. Connect the RGB image data, focus coordinate data and global feature coordinates of the bird's-eye view in the channel dimension, and then process them through a fully convolutional network encoder to obtain the feature map;

[0039] S2133. The feature map is concatenated with the point cloud data in the channel dimension to indicate the spatial location of each patch in the feature map. Simultaneously, mean pooling is applied to the point cloud data to match the size of the point cloud data with the feature map. Finally, the feature map encoding is calculated. The expression for the feature map encoding is:

[0040] ;

[0041] In the formula, Indicates feature mapping encoding; Represents RGB image features; Indicates the characteristics of the focal coordinate graph; Represents the global feature coordinates of the bird's-eye view; This represents point cloud data;

[0042] S2134. Treat the patches belonging to the feature mapping encoding as separate visual markers, and then use the embedding of camera ID feature encoding, the embedding of stride ID feature encoding, and the embedding of patch position to represent the visual features of the visual markers. The expression for the operation of the visual features is as follows:

[0043] ;

[0044] In the formula, Visual features representing visual markers; W f The matrix that a convolutional neural network can learn; This represents a patch in feature mapping encoding; This represents the camera ID feature encoding; This represents the step size ID feature encoding; Indicates the location of the patch; Indicates visual markers;

[0045] S2135. Record the visual markers of each camera at different step sizes, and concatenate the encoded tokens of all cameras into a single set of visual markers.

[0046] Furthermore, a text information encoder is constructed based on the CLIP pre-trained model to extract text features from the text information, and the sentence instructions in the text information are marked and encoded, including the following steps:

[0047] S221. Construct a text information encoder based on the language encoder in the CLIP pre-trained model, and use the linear layer of the CLIP pre-trained model to obtain the word embedding of each word tag in the text information. The expression for the word embedding is:

[0048] ;

[0049] In the formula, Indicates word embedding; LN represents the first embedding output of the text information encoder; W represents layer normalization. x Represents the projection matrix; This indicates a type of embedding that distinguishes between instructions and visual observations;

[0050] S222. Extract text features based on word tags in the text information, and mark and encode sentence instructions in the text information.

[0051] Furthermore, by fusing textual and visual features and continuously updating and optimizing the dependency relationship between text and task through learning from textual information, the robot can better understand the task through text and predict future actions. This includes the following steps:

[0052] S31. Using the self-attention mechanism of the Transformer model, learn the relationships between multiple camera views, the relationship between the current observation and the command, and the relationship between the current observation and historical observations. The expression for the self-attention mechanism is:

[0053] ;

[0054] In the formula, Q, K, and V represent three sets of vectors for the self-attention mechanism of the Transformer model, and W... Q W K W v All represent learnable parameters;

[0055] S32. Using a cross-attention layer to learn the intermodal relationships between the visual label set and its conditional context, the expression for learning the intermodal relationships across attention layers is:

[0056] ;

[0057] In the formula, G represents the relationship between modes; t Represents a set of visual markers; C t Indicates the conditional context;

[0058] S33. The modal relationships between patch tokens obtained from views of multiple cameras are learned using a self-attention layer. The expression for the modal relationships learned by the self-attention layer is:

[0059] ;

[0060] In the formula, This indicates the modal relationship between patch tokens;

[0061] S34. A feedforward network consisting of two linear layers is used to connect the output embedding of the Transformer model with the embedding of the original encoded visual representation in the channel dimension, and the flat sequence is reshaped into a predicted feature map.

[0062] S35. Based on the predictive feature mapping and convolutional neural network, predict the future state of the gripper and output the robot's predicted action sequence.

[0063] Furthermore, based on the predicted feature map and convolutional neural network, the future state of the gripper is predicted, and the predicted action sequence of the robot is output, including the following steps:

[0064] S351. The predicted feature map is fed back to the decoding network of the convolutional neural network, and the linear layer regression vector is used to predict the robot's rotation angle. The prediction result is [q]. t+1 c t+1 ];

[0065] S352. Decompose the predicted position of the gripper into the expected point coordinates and offset coordinates on the point cloud. The decomposition expression for the predicted position is:

[0066] ;

[0067] In the formula, Indicates the predicted location; Indicates the coordinates of the expected point; Indicates the offset coordinates;

[0068] S353. For each camera, the attention map on the point cloud coordinates is predicted using the upsampling layer of the convolutional neural network. Each value on the attention map corresponds to the probability that the end effector reaches that point cloud coordinate. Then, the expected point coordinates are calculated. The expression for the expected point coordinates is:

[0069] ;

[0070] In the formula, This represents the numerical value in the attention map; This represents the point cloud coordinates corresponding to the attention map;

[0071] S354. Predict the task sequence ID from the instructions, and then calculate the offset coordinate based on the current step ID. The expression for calculating the offset coordinate is:

[0072] ;

[0073] In the formula, Indicates the offset coordinates; The sequence ID represents the prediction task; m represents the feature vector; E0 represents the learnable embedding; t represents the current time step; t+1 represents the next time step.

[0074] S355. Combining the predictions of rotation angle and gripper state, output the predicted sequence of future robot actions a. t+1 =[p t+1 ;q t+1 ;c t+1 ].

[0075] Furthermore, by learning from the text information, and further learning the dependencies between the text information and actions, the action sequence is optimized through the following steps:

[0076] S41. Based on the overall learning task, the robot interacts with the surrounding environment, predicts the actions for each environmental state, and the tools needed to complete the actions.

[0077] S42. Learning densely embedded robot environments, and using a knowledge corpus to train existing semantic representations, allows the robot to generalize predictions to new environments filled with new objects.

[0078] S43. Train the model using a human demonstration plan from a corpus containing common-sense knowledge of task ranking tools and interactions.

[0079] The beneficial effects of this invention are as follows:

[0080] 1. By combining textual and visual information, and using textual information, along with the robot's past multi-view visual information and previous action information, the next action can be predicted. This method can be applied to different tasks. Among them, the network that integrates textual instructions and motion information has a higher action prediction accuracy, which provides a guarantee for robot motion planning.

[0081] 2. By adopting a global and local fusion approach, the visual information around the robot is projected onto a bird's-eye view as global information, while the robot's main view is used as local information. Then, the global and local features are fused to obtain more granular visual features, thereby effectively improving segmentation accuracy and speed. Secondly, through conditional action decoding, historical visual and action information is learned and combined with contextual conditions to predict the robot's next movement. Compared with other motion planning networks, it has better prediction performance and generalization ability. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of a robot motion skill learning method that integrates text instructions and motion information according to an embodiment of the present invention;

[0084] Figure 2 This is a schematic diagram of a global and local feature representation method in a robot motion skill learning method that integrates text instructions and motion information according to an embodiment of the present invention. Detailed Implementation

[0085] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0086] According to an embodiment of the present invention, a method for learning robot motor skills that integrates text instructions and motion information is provided.

[0087] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1As shown, according to an embodiment of the present invention, a robot motion skill learning method that integrates text instructions and motion information includes the following steps:

[0088] S1. Obtain the action sequence during the robot's movement (the action sequence includes visual information and action information), merge them into a dataset, perform preprocessing, extract keyframes from the action sequence, and simultaneously obtain the text information corresponding to the task.

[0089] In the description of this invention, the process of acquiring the action sequence during robot movement, merging it into a dataset, preprocessing it, extracting keyframes from the action sequence, and simultaneously acquiring the text information corresponding to the task includes the following steps:

[0090] S11. Through teaching and learning, control the robot to complete the set task, obtain visual and motion information in continuous motion state, and merge the visual and motion information into the dataset of the task.

[0091] S12. Sample the continuous action sequence during robot operation and extract keyframes of visual information from the dataset, including the start frame, intermediate frames where the end effector speed tends to stop, and intermediate frames where the end effector state changes (open / close). The aim is to limit the task sequence length to within 10 macrosteps.

[0092] In particular, due to the large number of continuously sampled data samples, the neural network model cannot learn the skill well, resulting in a low task completion rate. Therefore, in order to improve the performance of the system, the length of the task sequence needs to be reduced from hundreds of small steps to usually less than 10 macro steps.

[0093] S13. Obtain the set of text information corresponding to the task. The purpose is to learn different texts and ensure that the robot can perform the task after receiving different text information.

[0094] In daily life, there are different ways to describe the same task. For example, in the task of shooting a basketball, "put the ball in the hoop," "play basketball," "pick up the basketball and put it in the hoop," and "throw the basketball through the hoop" can all be used as text information for the task of shooting a basketball. Therefore, this invention needs to learn different text information so that the robot can complete the task when it receives different text information.

[0095] S2. Extract the visual and textual features of the task respectively.

[0096] In the description of this invention, the extraction of visual features and textual features of a task includes the following steps:

[0097] S21. First, the visual information is encoded using a global feature encoder and a local feature encoder, respectively. Then, a visual information fusion encoder is constructed to achieve multi-scale feature information fusion.

[0098] In the description of this invention, as Figure 2 As shown, visual information is first encoded using a global feature encoder and a local feature encoder, respectively. Then, a visual information fusion encoder is constructed to achieve multi-scale feature information fusion, including the following steps:

[0099] S211. The overall environment composed of all perspectives in visual information is regarded as multi-perspective image information, and the local environment composed of a single perspective in visual information is regarded as local image information.

[0100] S212. Construct a global feature module and extract global feature information from multi-view image information by means of projection and coordinate transformation.

[0101] In the description of this invention, constructing a global feature module and extracting global feature information from multi-view image information using projection and coordinate transformation includes the following steps:

[0102] S2121, Utilizing the backbone of a convolutional neural network to extract multi-view image information I={I k Extracting multi-scale features F img ={F k}

[0103] S2122. Initialize the bird's-eye view feature, which is uniformly divided into a grid, with each bird's-eye view grid corresponding to a three-dimensional coordinate P. i = (x i y i (z) and query vector z represents the predefined height of the bird's-eye view plane shared by all query vectors;

[0104] S2123. Using geometric prior knowledge to guide the attention model to focus on distinguishable regions in the bird's-eye view, and using camera parameters, project the 3D coordinates of each bird's-eye view grid onto a set of floating-point 2D coordinates. (For different views and scales), then change the two-dimensional coordinates. Rounded to the nearest global feature coordinates in the bird's-eye view. The transformation expression for two-dimensional coordinates is:

[0105] ;

[0106] ;

[0107] In the formula, M represents the learnable projection matrix, Rt represents the transition matrix from camera coordinates to world coordinates, and k represents the number of cameras.

[0108] S2124. Expand around the global feature coordinates of the prior bird's-eye view (expansion is a change in the feature dimension) K h ×K w Core region, K h With K w These represent the variable height and length parameters, respectively. If the kernel region exceeds the image boundary of the multi-view image information, the excess portion is set to zero. Then, an attention model is used to expand the kernel features of the query embedding for each bird's-eye view with the corresponding multi-view image information features. Interact with the system (interaction refers to using the Transformer model to calculate the relationship between camera coordinates projected onto the BEV bird's-eye view, which is a parameter that the network model can learn) to generate a bird's-eye view representation.

[0109] S2125. Obtain the camera parameters of the robot, decompose the camera deviation during the action shooting process into translation deviation and rotation deviation, and add random noise to all x, y, z dimensions and all cameras.

[0110] The expression for the translational deviation is:

[0111] ;

[0112] The expression for rotational deviation is:

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] In the formula, T devi R represents the translation deviation. devi Represents rotational deviation, Δx, Δy, and Δz represent translational noise along the x, y, and z axes, respectively, and R θx R θy and R θz Let θ represent the rotational deviation matrices for the x-axis, y-axis, and z-axis, respectively. x θ y θ z These represent the rotational noise corresponding to the x-axis, y-axis, and z-axis of the camera coordinate system, respectively.

[0118] In addition, Δx, Δy, Δz, θx θ y θ z Both are random variables that follow a normal distribution, represented as:

[0119] ;

[0120] .

[0121] In this invention, robustness to camera (camera) deviation is required. For a robot's motion operation, the external environment is complex, and the camera will deviate from its calibrated position. Therefore, it is necessary to decompose the camera deviation into translational deviation and rotational deviation, and add random noise to all x, y, z dimensions and all cameras.

[0122] After introducing noise bias, the transformation expression between 2D coordinates and global feature coordinates of the bird's-eye view changes to:

[0123] ;

[0124] .

[0125] S213. Construct a local feature module and use feature mapping to extract local feature information from local image information.

[0126] In the description of this invention, constructing a local feature module and extracting local feature information from local image information using feature mapping includes the following steps:

[0127] S2131. Obtain the binary attention mapping corresponding to the local image information captured by each camera at time t. Point cloud data and proprioceptive data .

[0128] in, It is a binary attention map used for gripper p t The position is encoded, with a value of 1 at the center of the gripper and a value of 0 at other positions.

[0129] S2132. Connect the attention mapping in the channel dimension, the proprioceptive data and the global feature coordinates of the bird's-eye view, and then process them through a fully convolutional network (UNet) encoder to obtain the feature mapping.

[0130] Wherein, the feature mapping is represented as , where H v W v d v These represent the height, width, and number of channels of the feature map, respectively.

[0131] S2133. The feature map is concatenated with the point cloud data in the channel dimension to indicate the spatial location of each patch in the feature map. Simultaneously, mean pooling is applied to the point cloud data to match the size of the point cloud data with the feature map. Finally, the feature map encoding is calculated. The expression for the feature map encoding is:

[0132] ;

[0133] In the formula, Represents feature mapping encoding, Represents RGB image features, Indicates the characteristics of the focal coordinate graph. Represents the global feature coordinates of the bird's-eye view. This represents point cloud data.

[0134] S2134. Treat the patches belonging to the feature mapping encoding as separate visual markers, and then use the embedding of camera ID feature encoding, the embedding of stride ID feature encoding, and the embedding of patch position to represent the visual features of the visual markers. The expression for the operation of the visual features is as follows:

[0135] ;

[0136] In the formula, The visual characteristics that represent visual markers, W f This represents the matrix that a convolutional neural network can learn. This represents a patch in feature mapping encoding. This represents the camera ID feature encoding. This represents the step size ID feature encoding. Indicates the patch location. Indicates visual markers.

[0137] S2135. Record the visual markers of each camera at different step sizes, and concatenate the coded tokens of all cameras into a unified set of visual markers. .

[0138] The encoded visual label of the k-th camera at step t is represented as .

[0139] S214. A visual information fusion encoder is formed by combining global feature modules and local feature modules. Then, by combining global feature information and local feature information, the encoding of visual information is achieved by using a multi-scale fusion method of global and local information.

[0140] S22. Construct a text information encoder based on a contrastive language image pre-training model, extract text features from the text information, and mark and encode sentence instructions in the text information.

[0141] In the description of this invention, the process of constructing a text information encoder based on a CLIP pre-trained model, extracting text features from the text information, and marking and encoding sentence instructions in the text information includes the following steps:

[0142] S221. Based on the language encoder in the contrastive language image pre-trained model (CLIP model) (due to its visual and language pre-training, it performs better in distinguishing visually related semantics compared to pure language pre-trained models such as BERT), a text information encoder is constructed, and the word embedding of each word tag in the text information is obtained using the linear layer of the contrastive language image pre-trained model. The expression for the word embedding is:

[0143] ;

[0144] In the formula, Indicates word embedding, LN represents the first embedding output of the text information encoder, and W represents the layer normalization. x Represents the projection matrix. This indicates the type embedding that distinguishes between instructions and visual observations.

[0145] S222. Extract text features based on word tags in the text information, and mark and encode sentence instructions in the text information.

[0146] S3. By fusing textual and visual features and learning from textual information, the dependency between text and task is continuously updated and optimized, enabling the robot to better understand the task through text and predict future robot actions.

[0147] Given the encoded token at the current macrostep t, the text and vision multimodal Transformer model aims to obtain G. t The context representation, conditional on text-encoded instructions. and historical action records This allows us to learn the relationships between views from multiple cameras, the relationship between the current observation and instructions, and the relationship between the current observation and history.

[0148] In the description of this invention, embedding text features into visual features and updating and optimizing them using text information learning to predict the robot's future actions includes the following steps:

[0149] S31. Using the self-attention mechanism of the Transformer model, learn the relationships between multiple camera views, the relationship between the current observation and the command, and the relationship between the current observation and historical observations. The expression for the self-attention mechanism is:

[0150] ;

[0151] In the formula, Q, K, and V represent three sets of vectors for the self-attention mechanism of the Transformer model, and W... Q W K W v All of these represent learnable parameters.

[0152] In the Transformer model, Q, K, and V refer to Query, Key, and Value, respectively. This is the key part of the Transformer model used to implement the self-attention mechanism.

[0153] In a self-attention mechanism, the input sequence is mapped to Query, Key, and Value vectors, respectively. Then, by calculating the similarity between them, a weight distribution is obtained, which is used to perform a weighted summation of the Values. This weight distribution can represent the degree of association between the Query and the Key.

[0154] S32. Using a cross-attention layer to learn the intermodal relationships between the visual label set and its conditional context, the expression for learning the intermodal relationships across attention layers is:

[0155] ;

[0156] In the formula, G represents the relationship between modes. t C represents the set of visual markers. t Indicates the conditional context.

[0157] S33. The modal relationships between patch tokens obtained from views of multiple cameras are learned using a self-attention layer. The expression for the modal relationships learned by the self-attention layer is:

[0158] ;

[0159] In the formula, This indicates the modal relationship between patch tokens.

[0160] S34. A feedforward network consisting of two linear layers is used to connect the output embedding of the Transformer model with the embedding of the original encoded visual representation along the channel dimension, and the flat sequence is reshaped into a predicted feature map, which is as follows: .

[0161] The expression for the feedforward network is: .

[0162] S35. Based on the predictive feature mapping and convolutional neural network, predict the future state of the gripper and output the robot's predicted action sequence.

[0163] In the description of this invention, predicting the future state of the gripper based on predictive feature mapping and a convolutional neural network, and outputting a predicted sequence of robot actions includes the following steps:

[0164] S351. The predicted feature map is fed back to the decoding network of the convolutional neural network, and the linear layer regression vector is used to predict the robot's rotation angle. The prediction result is [q]. t+1 c t+1 ].

[0165] S352. Decompose the predicted position of the gripper into the expected point coordinates and offset coordinates on the point cloud. The decomposition expression for the predicted position is:

[0166] ;

[0167] In the formula, Indicates the predicted location. Indicates the coordinates of the expected point. This indicates the offset coordinate.

[0168] S353. For each camera, predict the point cloud coordinates using the upsampling layer of a convolutional neural network. Attention map on H and W represent the height and length of the visual field, respectively, and each value on the attention map... The corresponding end effector reaches the point cloud coordinates The probability is then used to calculate the expected point coordinates. The expression for calculating the expected point coordinates is:

[0169] ;

[0170] In the formula, This represents the numerical value in the attention map. This represents the point cloud coordinates corresponding to the attention map.

[0171] S354. Predict the task sequence ID from the instructions, and then calculate the offset coordinate based on the current step ID. The expression for calculating the offset coordinate is:

[0172] ;

[0173] In the formula, Indicates the offset coordinates. Let represent the prediction task sequence ID, m represent the feature vector, and E0 represent the learnable embedding. , This represents the total number of tasks, and T represents the maximum segment length. , This represents the predicted probability, t represents the current time step, and t+1 represents the next time step.

[0174] S355. Combining the predictions of rotation angle and gripper state, output the predicted sequence of future robot actions a. t+1 =[p t+1 ;q t+1 ;c t+1 ].

[0175] S4. By learning from text information, further learn the dependency relationship between text information and actions, and optimize the action sequence.

[0176] The objective of this invention is to predict the next robot action given a world state s. t Expected target g t The history of actions taken by robots in the past η t This invention considers learning to predict multi-step plans, which requires complex interactive objects as tools, and may involve new environments that are not visible during the training process.

[0177] In the description of this invention, the optimization of action sequences through learning and understanding textual information, sorting out and mastering the dependencies between textual information and actions, includes the following steps:

[0178] S41. Based on the overall learning task, the robot interacts with the surrounding environment, predicts the actions for each environmental state, and the tools needed to complete the actions.

[0179] S42. Learning densely embedded robot environments, and using a knowledge corpus to train existing semantic representations, allows the robot to generalize predictions to new environments filled with new objects.

[0180] S43. Train the model using a human demonstration plan from a corpus containing common-sense knowledge of task ranking tools and interactions.

[0181] In form, this invention introduces a mimicry learner, implemented as a hyperparameter function (neural network model), denoted as f. θ As shown below:

[0182] .

[0183] In summary, by combining textual and visual information, and using the robot's past multi-view visual information and previous action information to predict the next action, this method can be applied to various tasks. The instruction-driven robot history perception network exhibits higher action prediction accuracy, providing assurance for robot motion planning. By employing a global and local fusion approach, visual information surrounding the robot is projected onto a bird's-eye view as global information, while the robot's main view is used as local information. The global and local features are then fused to obtain finer-grained visual features, effectively improving segmentation accuracy and speed. Furthermore, through conditional action decoding, historical visual and action information is learned and combined with contextual conditions to predict the robot's next motion, exhibiting better prediction performance and generalization ability compared to other motion planning networks.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for learning robot motor skills by integrating text instructions and motion information, characterized in that, include: S1. Obtain the action sequence during the robot's movement, merge it into a dataset, preprocess it, extract the keyframes of the action sequence, and obtain the text information corresponding to the task. S2. Extract the visual and textual features of the task respectively; S3. Integrate textual features with visual features. By learning from textual information, continuously update and optimize the dependency relationship between text and task, enabling the robot to better understand the task through text and predict future robot actions. S4. By learning from text information, further learn the dependency relationship between text information and actions, and optimize the action sequence; The process of acquiring the robot's motion sequence, merging it into a dataset, preprocessing it, extracting keyframes from the motion sequence, and simultaneously acquiring the text information corresponding to the task includes the following steps: S11. By teaching and learning, the robot is controlled to complete the set task, obtain visual information and motion information in continuous motion state, and merge the visual information and motion information into the dataset of the task. S12. Sample the continuous action sequence during the robot operation process and extract the key frames of the visual information in the dataset, including the start frame, the intermediate frame where the end gripper speed tends to stop and the intermediate frame where the end gripper state changes. The purpose is to limit the length of the task sequence to within 10 macrosteps. S13. Obtain the set of text information corresponding to the task. The purpose is to learn different texts so that the robot can perform the task after receiving different text information. The process of fusing textual and visual features, and continuously updating and optimizing the dependency relationship between text and task through learning textual information, enables the robot to better understand the task through text and predict future robot actions, including: S31. Using the self-attention mechanism of the Transformer model, learn the relationships between multiple camera views, the relationship between the current observation and the command, and the relationship between the current observation and historical observations. The expression for the self-attention mechanism is: ； In the formula, Q, K, and V represent the three sets of vectors of the self-attention mechanism of the Transformer model; W Q , W K , W v each represent a learnable parameter; S32. The intermodal relationship between the visual tag set and its conditional context is learned using a cross-attention layer. The expression for learning the intermodal relationship across the cross-attention layer is: ； In the formula, Indicates the relationship between modes; G t represents a set of visual markers; C t represents a conditional context; S33. The modal relationship between patch tokens obtained from views of multiple cameras is learned using a self-attention layer. The expression for the modal relationship learned by the self-attention layer is: ； In the formula, This indicates the modal relationship between patch tokens; S34. A feedforward network consisting of two linear layers is used to connect the output embedding of the Transformer model with the embedding of the original encoded visual representation in the channel dimension, and the flat sequence is reshaped into a predicted feature map. S35. Based on the predicted feature mapping and convolutional neural network, predict the future state of the gripper and output the robot's predicted action sequence.

2. The robot motion skill learning method integrating text instructions and motion information according to claim 1, characterized in that, The extraction of visual and textual features for the task includes the following steps: S21. First, the visual information is encoded using a global feature encoder and a local feature encoder respectively. Then, a visual information fusion encoder is constructed to achieve multi-scale feature information fusion. S22. Based on the text encoder in the CLIP pre-trained model, extract the text features of the text information.

3. The robot motion skill learning method integrating text instructions and motion information according to claim 2, characterized in that, The process of first encoding visual information using a global feature encoder and a local feature encoder, and then constructing a visual information fusion encoder to achieve multi-scale feature information fusion includes the following steps: S211. The overall environment composed of all perspectives in the visual information is taken as multi-view image information, and the local environment composed of a single perspective in the visual information is taken as local image information. S212. Construct a global feature module and extract global feature information from the multi-view image information using projection and coordinate transformation. S213. Construct a local feature module and extract local feature information from the local image information using feature mapping; S214. A visual information fusion encoder is formed by combining the global feature module and the local feature module. The visual information is then encoded by combining the global feature information and the local feature information in a multi-scale fusion manner using global and local information.

4. The robot motion skill learning method integrating text instructions and motion information according to claim 3, characterized in that, The construction of the global feature module, which extracts global feature information from the multi-view image information using projection and coordinate transformation, includes the following steps: S2121. Extract multi-scale features from multi-view image information using the backbone of a convolutional neural network; S2122. Initialize the bird's-eye view feature, which is uniformly divided into a grid, with each bird's-eye view grid corresponding to a three-dimensional coordinate P. i = (x i y i (z) and query vector z represents the predefined height of the bird's-eye view plane shared by all query vectors; S2123. Using geometric prior knowledge to guide the attention model to focus on the distinguishable areas in the bird's-eye view, using camera parameters, projecting the three-dimensional coordinates of each bird's-eye view grid to a set of floating two-dimensional coordinates, and then rounding the two-dimensional coordinates to the global feature coordinates of the bird's-eye view. S2124, K is expanded around the prior global feature coordinates of the bird's eye view h ×K w core region, K h with K w respectively represent variable high and long parameters, if the core region exceeds the image boundary of the multi-view image information, the exceeding part is set to zero, and then the query embedding of each bird's eye view is interacted with the corresponding multi-view image information feature expansion core feature using an attention model to generate a bird's eye view representation. S2125. To enhance the robustness of camera bias, camera bias is decomposed into translation bias and rotation bias, and random noise is added to all x, y, z dimensions and all cameras. The expression for the translational deviation is as follows: ； The expression for the rotational deviation is: ； In the formula, T devi represents the translation bias; R devi Indicates rotational deviation; Δx, Δy, and Δz represent the translation noise corresponding to the x-axis, y-axis, and z-axis, respectively. R θx R θy and R θz These represent the rotational deviation matrices for the x-axis, y-axis, and z-axis, respectively.

5. A robot motion skill learning method integrating text instructions and motion information according to claim 4, characterized in that, The construction of the local feature module, which extracts local feature information from the local image information using feature mapping, includes the following steps: S2131. Obtain the RGB image data, point cloud data, and focus coordinate map data corresponding to the local image information captured by each camera at time t. S2132. The RGB image data, the focal coordinate map data, and the global feature coordinates of the bird's-eye view in the channel dimension are connected, and then processed by a fully convolutional network encoder to obtain the feature map; S2133. The feature map is concatenated with the point cloud data in the channel dimension to indicate the spatial location of each patch in the feature map. Simultaneously, mean pooling is applied to the point cloud data to achieve size matching between the point cloud data and the feature map. Finally, the feature map encoding is calculated, and the expression for the feature map encoding is: ； In the formula, Indicates feature mapping encoding; Represents RGB image features; Indicates the characteristics of the focal coordinate graph; Represents the global feature coordinates of the bird's-eye view; This represents point cloud data; S2134. The patch belonging to the feature mapping encoding is taken as a separate visual marker, and the visual features of the visual marker are represented by the embedding of the camera ID feature encoding, the embedding of the stride ID feature encoding, and the embedding of the patch position. The operation expression of the visual features is: ； In the formula, Visual features that indicate visual markers; W f The matrix that a convolutional neural network can learn; This represents a patch in feature mapping encoding; This represents the camera ID feature encoding; This represents the step size ID feature encoding; Indicates the location of the patch; Indicates visual markers; S2135. Record the visual markers of each camera at different step sizes, and concatenate the encoded tokens of all cameras into a single set of visual markers.

6. The robot motion skill learning method integrating text instructions and motion information according to claim 5, characterized in that, The text information encoder built based on the CLIP pre-trained model, which extracts text features from the text information and marks and encodes sentence instructions in the text information, includes the following steps: S221. Construct a text information encoder based on the language encoder in the CLIP pre-trained model, and obtain the word embedding of each word tag in the text information using the linear layer of the CLIP pre-trained model. The expression for the word embedding is: ； In the formula, Indicates word embedding; This represents the first embedded output of the text information encoder; LN indicates layer normalization; W x Represents the projection matrix; This indicates a type of embedding that distinguishes between instructions and visual observations; S222. Extract text features based on word tags in the text information, and mark and encode sentence instructions in the text information.

7. A robot motion skill learning method integrating text instructions and motion information according to claim 6, characterized in that, The step of predicting the future state of the gripper based on the predicted feature map and convolutional neural network, and outputting the robot's predicted action sequence, includes the following steps: S351. The predicted feature map is fed back to the decoding network of the convolutional neural network, and the rotation angle of the robot is predicted using the linear layer regression vector. The prediction result is [q]. t+1 c t+1 ]; S352. The predicted position of the gripper is decomposed into the expected point coordinates and offset coordinates on the point cloud. The decomposition expression of the predicted position is: ； In the formula, Indicates the predicted location; Indicates the coordinates of the expected point; Indicates the offset coordinates; S353. For each camera, the attention map on the point cloud coordinates is predicted using the upsampling layer of the convolutional neural network. Each value on the attention map corresponds to the probability that the end effector reaches that point cloud coordinate. Then, the expected point coordinates are calculated. The expression for the expected point coordinates is: ； In the formula, This represents the numerical value in the attention map; This represents the point cloud coordinates corresponding to the attention map; S354. Predict the task sequence ID from the instruction, and then calculate the offset coordinate based on the current step ID. The calculation expression for the offset coordinate is as follows: ； In the formula, Indicates the offset coordinates; Indicates the prediction task sequence ID; m represents the eigenvector; E0 represents a learnable embedding; t represents the current time step; t+1 represents the next time step; S355. Combining the predicted rotation angle and gripper state, output the predicted future action sequence a of the robot. t+1 =[p t+1 ;q t+1 c t+1 ].

Images