Robot motion prediction method using displacement vector traction and fused historical information

Abstract

The application discloses a robot action prediction method using displacement vector traction and fusing historical information, which comprises the following steps: collecting training data in a simulation environment, including a text description of a task performed by a mechanical arm and grayscale images of the end of the mechanical arm and the head; performing multi-modal fusion coding of the text description and the grayscale images by using a multi-modal fusion encoder, and outputting multi-modal fusion coding information; encoding the multi-modal fusion coding information and historical joint posture information by using a historical information encoder to obtain historical coding information; calculating joint displacement vectors and end effector displacement vectors; inputting the multi-modal fusion coding information, the displacement vectors, robot joint posture information, end effector pose and the historical coding information into a transformer architecture, adopting action block division and time integration technology in ACT, predicting end poses of the robot at multiple future time points, and obtaining robot joint angles at the multiple future time points through inverse kinematics calculation.

Description

Technical Field

[0001] This invention belongs to the field of robot motion prediction technology, and relates to a robot motion prediction method that utilizes displacement vector traction and integrates historical information. Background Technology

[0002] In recent years, large-scale artificial intelligence models have developed rapidly and have been applied to many fields such as robotics. However, existing methods for predicting robot motions based on large-scale artificial intelligence models are not yet mature and have many problems.

[0003] One of the most advanced technologies currently available is Stanford University's Robot Action Block Generative Model (ACT). This end-to-end robot task learning model has a low overall task success rate and lacks historical information memory during task execution. It relies solely on the current frame to judge and predict the subsequent action sequence. If the target object is occluded during the operation, the task cannot be completed. In contrast, the existing BeT algorithm, which utilizes historical information, only uses image information and is insufficient in describing the current environmental state.

[0004] To address this issue, the imitation problem in end-to-end autonomous driving models becomes a hurdle. Directly incorporating historical information into the Transformer backbone model can significantly degrade model performance. End-to-end robot task execution models tend to predict the state of subsequent frames based on the state of previous frames. If there are pauses during task execution, the robot may get stuck during model inference and prediction because it has lazily learned this strategy and fails to perform the correct actual operations or handle unexpected or environmental changes, resulting in the robotic arm remaining in a fixed position without changing its posture. Summary of the Invention

[0005] This invention provides a robot motion prediction method that utilizes displacement vector traction and integrates historical information. This solves the problem that existing models do not use historical information or only use historical image information, which may cause the robot to predict motion changes within a very small range under certain input conditions, resulting in the robot getting stuck.

[0006] This invention provides a method for predicting robot motions using displacement vector traction and fusing historical information, comprising:

[0007] Step 1: Collect training data in a simulation environment, including textual descriptions of the tasks performed by the robotic arm, and grayscale images of the robotic arm's end effector and head.

[0008] Step 2: Use a multimodal fusion encoder to perform multimodal fusion encoding of the text description and grayscale image, and output the multimodal fusion encoding information;

[0009] Step 3: Use a historical information encoder to encode the multimodal fusion coding information and historical joint pose information to obtain historical coding information;

[0010] Step 4: Calculate the joint displacement vector and the end effector displacement vector;

[0011] Step 5: Input the multimodal fusion encoding information, joint displacement vector, end effector displacement vector, robot joint pose information, end effector pose and historical encoding information into the transformer architecture. Use motion segmentation and time integration technology in ACT to predict the robot's end effector pose at multiple future moments, and obtain the robot joint angles at multiple future moments through inverse kinematics conversion.

[0012] The robot motion prediction method of the present invention, which utilizes displacement vector traction and fuses historical information, has the following beneficial effects:

[0013] This invention collects training data in a simulation environment and generates multimodal fusion encoded information using a multimodal fusion encoder. Simultaneously, a historical information encoder is used to obtain historical encoded information. The historical encoded information, historical multimodal fusion encoded information, joint displacement vectors, and end effector displacement vectors are input into a transformer architecture. Employing motion segmentation and time integration techniques from Actions in Transformers (ACT), the end effector pose of the robot at multiple future moments is predicted. Furthermore, inverse kinematics is used to calculate the robot's joint angles at these future moments, improving task success rate and resolving the problem of the robot getting stuck. Attached Figure Description

[0014] Figure 1 This is a flowchart of the robot motion prediction method of the present invention, which utilizes displacement vector traction and integrates historical information. Detailed Implementation

[0015] like Figure 1 As shown, the present invention provides a robot motion prediction method that utilizes displacement vector traction and fuses historical information, comprising:

[0016] Step 1: Collect training data in a simulation environment, including textual descriptions of the tasks performed by the robotic arm and grayscale images of the robotic arm's end effector and head.

[0017] In practice, a training dataset was obtained through the Coppeliasim simulation environment. A grasping task was designed, and text commands, camera images, and robot pose information were collected during the simulation demonstration. By executing the task multiple times, a dataset of a certain size was collected. The image size was (160, 120), and the robot pose information included joint angles and end effector pose.

[0018] Step 2: Use a multimodal fusion encoder to perform multimodal fusion encoding of the text description and the grayscale image, and output the multimodal fusion encoding information, specifically:

[0019] Step 2.1: Establish a multimodal fusion encoder, including a BERT model that has undergone knowledge distillation, a multilayer perceptron, and a visual encoder, wherein the visual encoder consists of multiple residual blocks.

[0020] Step 2.2: Use the BERT model that has undergone knowledge distillation to encode the text description of the task and obtain the text encoding information.

[0021] Step 2.3: Input the text encoding information into the multilayer perceptron and learn two coefficient tensors through two linear scaling.

[0022] Step 2.4: Input the two grayscale images into the visual encoder and obtain the visual encoding information through residual block extraction.

[0023] Step 2.5: Multiply the text encoding information with the two coefficient tensors and then add it to the visual encoding information, and input it into the next residual block; perform the same operation in each residual block in the visual encoder, and finally output a 3-dimensional tensor (512, 4, 5) as multimodal fusion encoding information.

[0024] Step 3: Use a historical information encoder to encode the multimodal fusion coding information and historical joint pose information to obtain historical coding information, specifically:

[0025] Step 3.1: Establish a historical information encoder, including a compression module, a linear layer, and a transformer encoder.

[0026] Step 3.2: Input the multimodal fusion coding information into the compression module, perform linear stretching on the second and third dimensions to obtain the two-dimensional multimodal fusion coding information (512, 20), and compress the second dimension by averaging to obtain the compressed multimodal fusion coding information (512, 1).

[0027] Step 3.3: Transform all joint pose information up to the current time step into 512 dimensions using a linear layer.

[0028] Step 3.4: The joint pose information at k time points, the compressed multimodal fusion coding information, and an all-zero tensor (1, 512) are concatenated and input into the transformer encoder. The coding output of the all-zero tensor is taken and converted into a tensor (1, 1024) through a linear layer. Its variance and mean are predicted and then reparameterized and output as a tensor (1, 512) as historical coding information.

[0029] Step 4: Calculate the joint displacement vector and the end effector displacement vector, specifically:

[0030] The difference between the joint pose information of the robot at the previous moment and the joint pose information of the robot at the initial position is taken as the joint displacement vector, and the difference between the pose of the end effector at the previous moment and the pose of the robot at the initial position is taken as the end effector displacement vector.

[0031] Step 5: Input the multimodal fusion encoded information, joint displacement vectors, end effector displacement vectors, robot joint pose information, end effector pose, and historical encoded information into the transformer architecture. Using motion segmentation and temporal integration techniques from ACT, predict the robot's end effector pose at multiple future moments. Then, calculate the robot's joint angles at these future moments using inverse kinematics. Specifically:

[0032] Step 5.1: Input the multimodal fusion encoding information, displacement vector, robot joint posture information, end effector pose and historical encoding information into the transformer architecture, and use the motion block and time integration technology in ACT to predict the robot's end effector pose at multiple future moments.

[0033] Step 5.2: Calculate the final end pose at a future moment by weighted summation of the predicted end poses from multiple historical moments. Specifically:

[0034] The algorithm employs action segmentation and temporal ensemble techniques from ACT, predicting the end-effector pose for x future time steps each time, with the prediction time steps set to be the same as the historical encoding time steps (x = k). This effectively avoids robot stuttering caused by long model computation times. An exponential weighting scheme is used to perform a weighted average of the predicted action sequences.

[0035]

[0036] in, The weight of the action at time t is calculated based on the information at time i. m = 0.01 is an adjustable parameter. The larger m is, the lower the weight of actions at later times will be.

[0037] The final end pose at a future moment is calculated by weighting and summing the end poses predicted from multiple historical moments:

[0038]

[0039] in, Let be the action predicted at time t at time j.

[0040] Step 5.3: Calculate the robot joint angles at multiple future moments by using inverse kinematics to obtain the final end pose at multiple future moments.

[0041] In practice, a composite loss function is used to train the model. One part is to calculate the KL divergence between the mean and variance obtained from the encoding of action history information and the commonly found Gaussian distribution in real life, so that the distribution obtained from the encoding of history information is close to the Gaussian distribution; the other part is to calculate the MSE loss by comparing the predicted action sequence with the actual action sequence.

[0042]

[0043] in It is the end effector pose predicted by the model k time steps after t, a t,t+k It represents the end effector pose k time steps after the actual value t in the training data, where β is an adjustable parameter. It is the historical information encoder based on the currently observed environmental state information. and historical action information a t-k,t The z-distribution formed by the mean and variance obtained after encoding, N(0,1) is a Gaussian distribution.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot motion prediction method using displacement vectors to pull and fuse historical information, characterized by, include: Step 1: Collect training data in a simulation environment, including textual descriptions of the tasks performed by the robotic arm, and grayscale images of the robotic arm's end effector and head. Step 2: Use a multimodal fusion encoder to perform multimodal fusion encoding of the text description and grayscale image, and output the multimodal fusion encoding information; Step 3: Use a historical information encoder to encode the multimodal fusion coding information and historical joint pose information to obtain historical coding information; Step 4: Calculate the joint displacement vector and the end effector displacement vector; Step 5: Input the multimodal fusion coding information, joint displacement vector, end effector displacement vector, robot joint posture information, end effector pose and historical coding information into the transformer architecture, use the motion block and time integration technology in ACT to predict the robot's end effector pose at multiple future moments, and obtain the robot joint angles at multiple future moments through inverse kinematics conversion. Step 2 specifically involves: Step 2.1: Establish a multimodal fusion encoder, including a BERT model that has undergone knowledge distillation, a multilayer perceptron, and a visual encoder, wherein the visual encoder is composed of multiple residual blocks; Step 2.2: Encode the text description of the task using a knowledge-distilled BERT model to obtain text encoding information; Step 2.3: Input the text encoding information into the multilayer perceptron and learn two coefficient tensors through two linear scaling steps; Step 2.4: Input the two grayscale images into the visual encoder and obtain the visual coding information through residual block extraction; Step 2.5: Multiply the text encoding information by the two coefficient tensors and then add it to the visual encoding information, and input it into the next residual block; perform the same operation in each residual block in the visual encoder, and finally output a 3-dimensional tensor (512, 4, 5) as multimodal fusion encoding information; Step 3 specifically involves: Step 3.1: Establish a historical information encoder, including a compression module, a linear layer, and a transformer encoder; Step 3.2: Input the multimodal fusion coding information into the compression module, linearly stretch the second and third dimensions to obtain the two-dimensional multimodal fusion coding information (512, 20), and compress the second dimension by averaging to obtain the compressed multimodal fusion coding information (512, 1). Step 3.3: Transform all joint pose information up to the current time step into 512 dimensions using a linear layer; Step 3.4: The joint pose information at k time points, the compressed multimodal fusion coding information, and an all-zero tensor (1, 512) are concatenated and input into the transformer encoder. The coding output of the all-zero tensor is taken and converted into a tensor (1, 1024) through a linear layer. Its variance and mean are predicted and then reparameterized and output as a tensor (1, 512) as historical coding information. 2.The robot motion prediction method using displacement vectors and fusion of history information according to claim 1, wherein, Step 4 specifically involves: The difference between the joint pose information of the robot at the previous moment and the joint pose information of the robot at the initial position is taken as the joint displacement vector, and the difference between the pose of the end effector at the previous moment and the pose of the robot at the initial position is taken as the end effector displacement vector. 3.The robot motion prediction method using displacement vectors and fusion of history information according to claim 1, wherein, Step 5 specifically involves: Step 5.1: Input the multimodal fusion encoding information, joint displacement vector, end effector displacement vector, robot joint pose information, end effector pose and historical encoding information into the transformer architecture, and use the motion block and time integration technology in ACT to predict the robot's end effector pose at multiple future moments; Step 5.2: Calculate the final end pose at a future moment by weighted summation of the predicted end poses from multiple historical moments; Step 5.3: Calculate the robot joint angles at multiple future moments by using inverse kinematics to obtain the final end pose at multiple future moments.

4. The robot motion prediction method using displacement vectors and fusion of history information according to claim 3, wherein, Step 5.2 specifically involves: The algorithm employs action segmentation and temporal ensemble techniques from ACT, predicting the end pose for x future time steps each time, with the prediction time steps set to be the same as the historical encoding time steps (x=k). This effectively avoids robot stuttering caused by long model computation times. An exponential weighting scheme is used to perform a weighted average of the predicted action sequences. wherein, is the weight of the action at time t solved according to the information at time i, m = 0.01 is an adjustable parameter, the greater m is, the lower the weight of the action at a later time. The final end pose at a future moment is calculated by weighting and summing the end poses predicted from multiple historical moments: in, Let be the action predicted at time t at time j.

Images