Robot action prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a robot action prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting multiple pieces of visual information of a robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, outputting key point features of the operated object grabbed by the robot; wherein different visual information comes from different mechanical arms of the robot; and performing action prediction processing on current mechanical arm state information, multiple pieces of visual information, task execution instruction information and the key point features based on a second network layer of the robot action prediction model, and outputting an action sequence of the robot. The key point features of the clamping jaw on the mechanical arm of the robot and the operated object are accurately determined through the robot action prediction model, and then action prediction processing is conducted through the key point features to improve the accuracy of robot action sequence determination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent decision-making, in particular to a robot action prediction method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the prior art, the perception and decision-making capabilities of embodied intelligent robots facing complex scenarios mainly rely on traditional perception and decision-making algorithms. These methods usually analyze through static data models, which are difficult to effectively cope with the dynamic environment and real-time decision-making requirements in complex scenarios. The shortcomings of the prior art include insufficient response capability to dynamic changes, poor real-time performance, limited processing capability, and especially when facing a high-speed changing environment, decision lag or insufficient precision problems are prone to occur. In addition, these technologies usually lack comprehensive processing capability for multi-modal information, making it difficult to accurately capture and understand complex perception information, resulting in limited adaptability and decision-making accuracy of robots in complex scenarios. Therefore, how to improve the accuracy of robot action prediction has become a technical problem that cannot be ignored. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a robot action prediction method and device, electronic equipment and storage medium, which accurately determines the key point features of the gripper on the robot's mechanical arm and the object being operated by using a robot action prediction model, and then uses the key point features for action prediction processing to improve the accuracy of robot action sequence determination.

[0004] The robot action prediction method provided by the present application comprises: inputting multiple visual information of the robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, and outputting key point features of the object being operated by the robot; wherein different visual information is derived from different mechanical arms of the robot; performing action prediction processing on the current mechanical arm state information, multiple visual information, task execution instruction information and the key point features based on a second network layer of the robot action prediction model, and outputting an action sequence of the robot.

[0005] In one possible implementation, the inputting multiple visual information of the robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, and outputting key point features of the object being operated by the robot comprises: performing cross-attention processing and multi-layer perception processing on multiple visual information and task execution instruction information based on the first network layer, and outputting multiple key point heat maps of the object being operated. The key point heatmap is fused with the corresponding visual information to output multiple key point features of the robot grasping the manipulated object; wherein, the key point features include key point features of different robotic arms.

[0006] In one possible implementation, for any of the aforementioned visual information, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information based on the first network layer to output a heatmap of multiple key points of the manipulated object, including: Visual features are determined by feature extraction from the visual information, and multiple text features and multiple entity features are extracted from the task execution instruction information. The visual features and multiple text features are processed using a multilayer perceptron and a cross-attention mechanism to generate a first feature; The multiple entity features are multiplied by their corresponding weight parameters and then input into the multilayer perceptron and activation function for processing to determine the second feature; The second feature is merged with the first feature to determine the third feature. The third feature is then subjected to convolution, activation function processing, and normalization processing to output the heat map of the key points of the object grasped by the robotic arm corresponding to the visual information.

[0007] In one possible implementation, the second network layer based on the robot motion prediction model performs motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features, and outputs the robot's motion sequence, including: Temporal attention and cross-attention processing are performed on multiple visual information to determine multiple target features; The state features of the current robotic arm state information are extracted, the state features and the target features are multiplied, and the multiplied features are fused with the key point features to generate fused features; The fused features are processed using a transformer network layer to predict actions, and the robot's action sequence is output.

[0008] In one possible implementation, the step of performing temporal attention processing and cross-attention processing on the multiple visual information to determine multiple target features includes: The region of the manipulated object related to the task execution instruction information in each visual information is segmented by masking to determine the mask image in each visual information. The mask image in each of the visual information is subjected to convolution processing, activation function processing, and normalization processing to determine the visual features of each visual image; Convolutional processing is performed on the visual features of each visual information to obtain the convolutional features of each visual information at each time step, and global average pooling processing is performed on the visual features of each visual information to obtain the pooling features of each visual information at each time step; Cross-attention processing and merging processing are performed on the convolutional features and pooling features of each visual information at each time step to output the target feature.

[0009] In one possible implementation, the process of performing cross-attention processing and merging processing on the convolutional features and pooling features of each visual information at each time step to output the target feature includes: Cross-attention processing is performed on the convolutional features of each visual information at any two adjacent time points, and the convolutional features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a third feature; Cross-attention processing is performed on the pooled features of each visual information at any two adjacent time points, and the pooled features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a fourth feature. The target feature is determined by multiplying the third feature and the fourth feature.

[0010] In one possible implementation, the robot motion prediction model is determined through the following steps: Multiple sample visual information of the robot, the current state information of the robotic arm of the sample, and the sample task execution instruction information are input into the supervised training model for key point prediction processing and action prediction processing, and the predicted key point features and predicted action sequence of the robot grasping the manipulated object are output. The supervised training model is iteratively trained based on the cross-entropy loss value between the actual key point features and the predicted key point features of the object grasped by the robot, and the cross-entropy loss value between the predicted action sequence and the actual action sequence, to determine the robot action prediction model.

[0011] This application embodiment also provides a robot motion prediction device, the robot motion prediction device comprising: The key point determination module is used to input multiple visual information and task execution instruction information of the robot into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot. The motion determination module is used to perform motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information and key point features based on the second network layer of the robot motion prediction model, and output the robot's motion sequence.

[0012] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the robot motion prediction method described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot motion prediction method described above.

[0014] This application provides a robot motion prediction method, apparatus, electronic device, and storage medium. The robot motion prediction method includes: inputting multiple visual information and task execution instruction information of the robot into a first network layer of a pre-trained robot motion prediction model for key point prediction processing, and outputting key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot; and performing motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features based on a second network layer of the robot motion prediction model, and outputting the robot's motion sequence. The robot motion prediction model accurately determines the key point features of the gripper on the robot's robotic arm and the manipulated object, and then uses the key point features for motion prediction processing to improve the accuracy of robot motion sequence determination.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a robot motion prediction method provided in an embodiment of this application; Figure 2 This is one of the structural schematic diagrams of a robot motion prediction device provided in the embodiments of this application; Figure 3 This is a second schematic diagram of the structure of a robot motion prediction device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable application scenarios of this application will be introduced. This application can be applied to the field of intelligent decision-making technology.

[0020] Research has revealed that current technologies for embodied intelligent robots in complex scenarios primarily rely on traditional perception and decision-making algorithms. These methods typically analyze data using static models, making them ill-suited to effectively address the changing environments and real-time decision-making demands of complex scenarios. The shortcomings of existing technologies include insufficient responsiveness to dynamic changes, poor real-time performance, and limited processing capabilities. Particularly in rapidly changing environments, they are prone to decision lag or insufficient accuracy. Furthermore, these technologies often lack the ability to comprehensively process multimodal information, making it difficult to accurately capture and understand complex perceptual information, thus limiting the robot's adaptability and decision-making accuracy in complex scenarios. Therefore, improving the accuracy of robot motion prediction has become a significant technical challenge.

[0021] Based on this, this application provides a robot motion prediction method, which accurately determines the key point features of the gripper on the robot's robotic arm and the manipulated object through a robot motion prediction model, and then uses the key point features to perform motion prediction processing to improve the accuracy of robot motion sequence determination.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a robot motion prediction method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the robot motion prediction includes: S101: Input multiple visual information and task execution instruction information of the robot into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot.

[0023] In this step, multiple visual information and task execution instruction information of the robot are input into the first network layer of the robot motion prediction model for key point prediction processing, and the key point features of the robot grasping the manipulated object are output.

[0024] The different visual information comes from the different robotic arms of the robot.

[0025] It should be noted that the different visual information was collected by the robot's left robotic arm, right robotic arm, and head.

[0026] Here, the task execution instruction information includes the name of the object being manipulated, the color of the object being manipulated, the shape of the object being manipulated, the size of the object being manipulated, and the grasping method, etc.

[0027] Among them, the key features of the robot grasping the manipulated object include the key feature points of the manipulated object corresponding to different robotic arms.

[0028] In one possible implementation, the step of inputting multiple visual information and task execution instruction information of the robot into the first network layer of a pre-trained robot motion prediction model for key point prediction processing, and outputting the key point features of the robot grasping the manipulated object, includes: A: Based on the first network layer, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information to output a heat map of multiple key points of the manipulated object.

[0029] Here, the first network layer is used to perform cross-attention processing and multilayer perceptron processing on multiple visual information and task execution instruction information, and outputs a heat map of multiple key points of the manipulated object.

[0030] In one possible implementation, for any of the aforementioned visual information, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information based on the first network layer to output a heatmap of multiple key points of the manipulated object, including: (1): Visual features are determined by feature extraction of the visual information, and multiple text features and multiple entity features are extracted from the task execution instruction information.

[0031] Here, RGBD image information from the robot's left arm, right arm, and head cameras is acquired, and the RGB image information is input into the DINOv2 image encoder to obtain visual features. Simultaneously, the task instructions are input into GPT4, which outputs text entity information. This includes "object name," "object color," "object shape," "object size," and "grasping method," etc. The acquired information is then input into the CLIP text encoder, which outputs the entity features. Simultaneously, the task instructions are input into the T5 text encoder to directly obtain text features.

[0032] (2): The visual features and multiple text features are processed based on the multilayer perceptron and cross-attention mechanism to generate the first feature.

[0033] Here, visual features are first input into the MLP module to obtain k and v features respectively, and text features are input into the MLP module to obtain q features. Then, cross-attention is performed on q, k, and v respectively to obtain the first feature.

[0034] (3): Multiply the multiple entity features by their corresponding weight parameters and input them into the multilayer perceptron and activation function for processing to determine the second feature.

[0035] Here, learnable attention parameters of the same shape and size are set simultaneously. After obtaining the weights of each feature through MLP and softmax parameters, the weights are multiplied with the entity features to obtain the second feature.

[0036] (4): The second feature and the first feature are merged to determine the third feature. The third feature is then subjected to convolution, activation function processing and normalization processing to output the key point heat map of the robotic arm grasping the manipulated object corresponding to the visual information.

[0037] Here, the second feature is merged with the first feature to determine the third feature. This feature is then input into the convolution module, which first performs channel-based convolution and then spatial convolution, while simultaneously performing ReLU activation and BN normalization operations. The resulting feature is then input into the upsampling module, which consists of multiple deconvolutions, unpooling, activation functions, and BN normalization operations, resulting in the final feature. Figure 1 The size of the sample predicts Gaussian heatmap information.

[0038] B: The key point heatmap is fused with the corresponding visual information to output multiple key point features of the robot grasping the manipulated object; wherein, the key point features include key point features of different robotic arms.

[0039] Here, the key point heatmap is fused with the corresponding visual information to output multiple key point features of the robot grasping the object being manipulated.

[0040] S102: Based on the second network layer of the robot motion prediction model, the current robotic arm state information, multiple visual information, task execution instruction information and the key point features are processed for motion prediction, and the robot's motion sequence is output.

[0041] In this step, the robot's motion prediction model performs motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and key point features based on the second network layer of the robot motion prediction model, and outputs the robot's motion sequence. In one possible implementation, the second network layer based on the robot motion prediction model performs motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features, and outputs the robot's motion sequence, including: a: Perform temporal attention processing and cross-attention processing on multiple visual information to determine multiple target features.

[0042] Here, multiple visual information are processed by temporal attention and cross-attention to determine multiple target features.

[0043] In one possible implementation, the step of performing temporal attention processing and cross-attention processing on the multiple visual information to determine multiple target features includes: I: Perform mask segmentation on the region of the manipulated object related to the task execution instruction information in each of the visual information to determine the mask image in each of the visual information; perform convolution processing, activation function processing and normalization processing on the mask image in each of the visual information to determine the visual features of each visual image.

[0044] Here, visual information and task instructions acquired by the cameras of the robot's left arm, head, and right arm at different times are input into the model. Based on this model, the area of ​​the manipulated object involved in the task can be segmented using a mask. This mask information is then input into a convolution module, which consists of a series of dilated convolutions, ReLU activation functions, and BN normalization operations to obtain the visual features segmented by the left arm, head, and right arm cameras at different times.

[0045] II: Perform convolution processing on the visual features of each visual information to obtain the convolutional features of each visual information at each time step, and perform global average pooling processing on the visual features of each visual information to obtain the pooling features of each visual information at each time step.

[0046] Here, the visual features of each visual information are convolved to obtain the convolutional features of each visual information at each time step, and the visual features of each visual information are convolved and global average pooled to obtain the pooled features of each visual information at each time step.

[0047] III: Perform cross-attention processing and merging processing on the convolutional features and pooling features of each visual information at each time step, and output the target feature.

[0048] Here, cross-attention processing is performed on the convolutional features of each visual information at each time step, and cross-attention processing is performed on the pooling features of each visual information at each time step. The features after the two cross-attention processing are merged to determine the target features.

[0049] In one possible implementation, the process of performing cross-attention processing and merging processing on the convolutional features and pooling features of each visual information at each time step to output the target feature includes: i: Perform cross-attention processing on the convolutional features of each visual information at any two adjacent time points, and then perform merging, multilayer perceptron processing, and activation function processing on the cross-processed convolutional features to output the third feature.

[0050] Here, the convolutional features output by the convolutional module at time t are subtracted from the convolutional features output at time t+1, and the convolutional features output by the convolutional module at time t+1 are subtracted from the convolutional features output by the convolutional module at time t+2. Then, the two are merged, processed by a multilayer perceptron, and processed by an activation function to output the third feature.

[0051] ii: Perform cross-attention processing on the pooled features of each visual information at any two adjacent time points, and then perform merging, multilayer perceptron processing, and activation function processing on the pooled features after cross-processing to output the fourth feature.

[0052] Here, cross-attention operations are performed on the pooling features at time t and time t+1 respectively, and cross-attention operations are performed on the pooling features at time t+1 and time t+2 respectively. The features output by the cross-attention operations are then merged and input into the MLP and the Sigmoid activation function to obtain the fourth feature.

[0053] iii: Multiply the third feature and the fourth feature to determine the target feature.

[0054] b: Extract the state features of the current robotic arm state information, multiply the state features and the target features, and fuse the multiplied features with the key point features to generate fused features.

[0055] Here, the state features of the current robotic arm state information are extracted, the state features and target features are multiplied, and the multiplied features are fused with the key point features to generate fused features.

[0056] In this process, the previously obtained key point features, state features, and target features are fully integrated to obtain more prior information.

[0057] c: Based on the transformer network layer, the fused features are processed for action prediction, and the robot's action sequence is output.

[0058] The robot motion prediction model is determined through the following steps: Multiple sample visual information and sample task execution instruction information of the robot are input into the supervised training model for key point prediction and action prediction processing, and the predicted key point features and predicted action sequence of the robot grasping the manipulated object are output. The supervised training model is iteratively trained based on the cross-entropy loss value between the actual key point features and the predicted key point features of the object grasped by the robot, and the cross-entropy loss value between the predicted action sequence and the actual action sequence, to determine the robot action prediction model.

[0059] In one possible implementation, the robot motion prediction model is determined through the following steps: Multiple sample visual information of the robot, the current state information of the robotic arm, and the sample task execution instruction information are input into the supervised training model for key point prediction and motion prediction processing. The model outputs the predicted key point features and predicted motion sequence of the robot grasping the manipulated object. Based on the cross-entropy loss value between the actual key point features and the predicted key point features of the robot grasping the manipulated object, and the cross-entropy loss value between the predicted motion sequence and the actual motion sequence, the supervised training model is iteratively trained to determine the robot motion prediction model.

[0060] Here, the network parameters of the supervised training model are modified based on the cross-entropy loss value between the actual key point features and the predicted key point features, as well as the cross-entropy loss value between the predicted action sequence and the actual action sequence. The supervised training model with modified network parameters is then iteratively trained to determine the robot action prediction model.

[0061] During model training, we divided the dataset into training and validation sets. We used the training set for training and the validation set to observe the model's convergence. The weights with the lowest decrease in the validation set loss function were selected as the final weights for the model. During training, we continuously changed the background information, added Gaussian noise to videos from different perspectives, and performed data augmentation on images, including color jittering and random masking. During inference, we added interpolation operations to the robotic arm's movements to ensure smooth motion.

[0062] In this application, the robot action prediction model can still generalize in training environments with few or even zero samples. Furthermore, the inclusion of segmentation and keypoint information effectively provides the model with excellent prior attention, improving the accuracy of action execution. The use of a temporal attention module allows the model to better perceive the sensitivity of prediction results to environmental changes, improving the accuracy of predicting the next action sequence.

[0063] This application provides a robot motion prediction method, which includes: inputting multiple visual information and task execution instruction information of the robot into a first network layer of a pre-trained robot motion prediction model for key point prediction processing, and outputting key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot; and performing motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features based on a second network layer of the robot motion prediction model, and outputting the robot's motion sequence. The robot motion prediction model accurately determines the key point features of the gripper on the robot's robotic arm and the manipulated object, and then uses the key point features for motion prediction processing to improve the accuracy of robot motion sequence determination.

[0064] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a robot motion prediction device provided in the embodiments of this application; Figure 3 This is a second schematic diagram of a robot motion prediction device provided in an embodiment of this application. Figure 3 As shown, the robot motion prediction device 300 includes: The key point determination module 210 is used to input multiple visual information and task execution instruction information of the robot into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot. The motion determination module 220 is used to perform motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information and key point features based on the second network layer of the robot motion prediction model, and output the robot's motion sequence.

[0065] Furthermore, the key point determination module 210 is used to input multiple visual information and task execution instruction information of the robot into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object: Based on the first network layer, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information to output a heat map of multiple key points of the manipulated object. The key point heatmap is fused with the corresponding visual information to output multiple key point features of the robot grasping the manipulated object; wherein, the key point features include key point features of different robotic arms.

[0066] Furthermore, the key point determination module 210 is used to perform cross-attention processing and multilayer perceptron processing on multiple visual information and task execution instruction information based on the first network layer for any of the aforementioned visual information, and output a heatmap of multiple key points of the manipulated object: Visual features are determined by feature extraction from the visual information, and multiple text features and multiple entity features are extracted from the task execution instruction information. The visual features and multiple text features are processed using a multilayer perceptron and a cross-attention mechanism to generate a first feature; The multiple entity features are multiplied by their corresponding weight parameters and then input into the multilayer perceptron and activation function for processing to determine the second feature; The second feature is merged with the first feature to determine the third feature. The third feature is then subjected to convolution, activation function processing, and normalization processing to output the heat map of the key points of the object grasped by the robotic arm corresponding to the visual information.

[0067] Furthermore, the motion determination module 220 is used in the second network layer based on the robot motion prediction model to perform motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features, and outputs the robot's motion sequence: Temporal attention and cross-attention processing are performed on multiple visual information to determine multiple target features; The state features of the current robotic arm state information are extracted, the state features and the target features are multiplied, and the multiplied features are fused with the key point features to generate fused features; The fused features are processed using a transformer network layer to predict actions, and the robot's action sequence is output.

[0068] Furthermore, the action determination module 220 is used to perform temporal attention processing and cross-attention processing on the multiple visual information pieces to determine multiple target features: The region of the manipulated object related to the task execution instruction information in each visual information is segmented by masking to determine the mask image in each visual information. The mask image in each of the visual information is subjected to convolution processing, activation function processing, and normalization processing to determine the visual features of each visual image; Convolutional processing is performed on the visual features of each visual information to obtain the convolutional features of each visual information at each time step, and global average pooling processing is performed on the visual features of each visual information to obtain the pooling features of each visual information at each time step; Cross-attention processing and merging processing are performed on the convolutional features and pooling features of each visual information at each time step to output the target feature.

[0069] Furthermore, the action determination module 220 is used to perform cross-attention processing and merging processing on the convolutional features and pooling features of each visual information at each time step, and output the target features: Cross-attention processing is performed on the convolutional features of each visual information at any two adjacent time points, and the convolutional features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a third feature; Cross-attention processing is performed on the pooled features of each visual information at any two adjacent time points, and the pooled features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a fourth feature. The target feature is determined by multiplying the third feature and the fourth feature.

[0070] Furthermore, such as Figure 3 As shown, the robot motion prediction 200 also includes a model training module 230, which determines the robot motion prediction model through the following steps: Multiple sample visual information of the robot, the current state information of the robotic arm of the sample, and the sample task execution instruction information are input into the supervised training model for key point prediction processing and action prediction processing, and the predicted key point features and predicted action sequence of the robot grasping the manipulated object are output. The supervised training model is iteratively trained based on the cross-entropy loss value between the actual key point features and the predicted key point features of the object grasped by the robot, and the cross-entropy loss value between the predicted action sequence and the actual action sequence, to determine the robot action prediction model.

[0071] This application provides a robot motion prediction device, comprising: a key point determination module, used to input multiple visual information and task execution instruction information of the robot into the first network layer of a pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot; and a motion determination module, used to perform motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features based on the second network layer of the robot motion prediction model, and output the robot's motion sequence. By accurately determining the key point features of the gripper on the robot's robotic arm and the manipulated object through the robot motion prediction model, the accuracy of robot motion sequence determination is improved by utilizing the key point features for motion prediction processing.

[0072] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0073] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the robot motion prediction method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0074] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the robot motion prediction method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0075] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0079] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting robot actions, characterized in that, The robot motion prediction method includes: Multiple visual information and task execution instruction information of the robot are input into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and the key point features of the robot grasping the manipulated object are output; wherein, different visual information comes from different robotic arms of the robot. The second network layer based on the robot motion prediction model performs motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and the key point features, and outputs the robot's motion sequence.

2. The robot motion prediction method according to claim 1, characterized in that, The process of inputting multiple visual information and task execution command information of the robot into the first network layer of a pre-trained robot motion prediction model for key point prediction processing, and outputting the key point features of the robot grasping the manipulated object, includes: Based on the first network layer, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information to output a heat map of multiple key points of the manipulated object. The key point heatmap is fused with the corresponding visual information to output multiple key point features of the robot grasping the manipulated object; wherein, the key point features include key point features of different robotic arms.

3. The robot motion prediction method according to claim 2, characterized in that, For any of the aforementioned visual information, based on the first network layer, cross-attention processing and multilayer perceptron processing are performed on multiple visual information and task execution instruction information to output a heatmap of multiple key points of the manipulated object, including: Visual features are determined by feature extraction from the visual information, and multiple text features and multiple entity features are extracted from the task execution instruction information. The visual features and multiple text features are processed using a multilayer perceptron and a cross-attention mechanism to generate a first feature; The multiple entity features are multiplied by their corresponding weight parameters and then input into the multilayer perceptron and activation function for processing to determine the second feature; The second feature is merged with the first feature to determine the third feature. The third feature is then subjected to convolution, activation function processing, and normalization processing to output the heat map of the key points of the object grasped by the robotic arm corresponding to the visual information.

4. The robot motion prediction method according to claim 1, characterized in that, The second network layer based on the robot motion prediction model performs motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information, and key point features, and outputs the robot's motion sequence, including: Temporal attention and cross-attention processing are performed on multiple visual information to determine multiple target features; The state features of the current robotic arm state information are extracted, the state features and the target features are multiplied, and the multiplied features are fused with the key point features to generate fused features; The fused features are processed using a transformer network layer to predict actions, and the robot's action sequence is output.

5. The robot motion prediction method according to claim 4, characterized in that, The process of performing temporal attention processing and cross-attention processing on multiple visual information pieces determines multiple target features, including: The region of the manipulated object related to the task execution instruction information in each visual information is segmented by masking to determine the mask image in each visual information. The mask image in each of the visual information is subjected to convolution processing, activation function processing, and normalization processing to determine the visual features of each visual image; Convolutional processing is performed on the visual features of each visual information to obtain the convolutional features of each visual information at each time step, and global average pooling processing is performed on the visual features of each visual information to obtain the pooling features of each visual information at each time step; Cross-attention processing and merging processing are performed on the convolutional features and pooling features of each visual information at each time step to output the target feature.

6. The robot motion prediction method according to claim 5, characterized in that, The process of performing cross-attention processing and merging processing on the convolutional features and pooling features of each visual information at each time step to output the target feature includes: Cross-attention processing is performed on the convolutional features of each visual information at any two adjacent time points, and the convolutional features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a third feature; Cross-attention processing is performed on the pooled features of each visual information at any two adjacent time points, and the pooled features after cross-processing are merged, processed by a multilayer perceptron, and processed by an activation function to output a fourth feature. The target feature is determined by multiplying the third feature and the fourth feature.

7. The robot motion prediction method according to claim 1, characterized in that, The robot motion prediction model is determined through the following steps: Multiple sample visual information of the robot, the current state information of the robotic arm of the sample, and the sample task execution instruction information are input into the supervised training model for key point prediction processing and action prediction processing, and the predicted key point features and predicted action sequence of the robot grasping the manipulated object are output. The supervised training model is iteratively trained based on the cross-entropy loss value between the actual key point features and the predicted key point features of the object grasped by the robot, and the cross-entropy loss value between the predicted action sequence and the actual action sequence, to determine the robot action prediction model.

8. A robot motion prediction device, characterized in that, The robot motion prediction device includes: The key point determination module is used to input multiple visual information and task execution instruction information of the robot into the first network layer of the pre-trained robot motion prediction model for key point prediction processing, and output the key point features of the robot grasping the manipulated object; wherein, different visual information comes from different robotic arms of the robot. The motion determination module is used to perform motion prediction processing on the current robotic arm state information, multiple visual information, task execution instruction information and key point features based on the second network layer of the robot motion prediction model, and output the robot's motion sequence.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the robot motion prediction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the robot motion prediction method as described in any one of claims 1 to 7.

Images