A generalizable robotic manipulation method and system based on segmentation mask representation

Abstract

The application discloses a generalizable robot operation method and system based on segmentation mask representation, and belongs to the field of desktop scene robot operation. A large number of virtual environment desktop scene layouts are automatically generated, and robot operation trajectory data for each layout is generated. Each trajectory data generates diversified text instructions. Robot visual angle images, robot state data and a text instruction at each operation step are collected as a training sample. A pre-trained multi-modal large model is used to locate a target object and a target area indicated by the text instruction in each training sample, so that a target object mask and a target area mask are obtained. The target object mask, the target area mask and the training sample corresponding to a plurality of historical operation steps are input into a robot operation strategy network, and an action instruction of a next operation step of the robot is predicted. The application effectively combines visual, language and spatial information, and improves the generalization ability of the robot strategy in a variable task scene.

Description

Technical Field

[0001] This invention relates to the field of desktop robot operation, and in particular to a generalizable robot operation method and system based on segmentation mask representation. Background Technology

[0002] Currently, robot manipulation has become an important research direction in artificial intelligence and robotics, especially robot manipulation in complex desktop scenarios, which has attracted widespread attention. This field aims to endow robots with efficient object manipulation capabilities by combining visual, language, and motion data, covering applications such as object grasping, precise placement, and task execution in complex scenes. Achieving universal operation of robots in diverse tasks and scenarios is key to further advancing robotics technology.

[0003] Existing robot manipulation technologies can be broadly categorized into three types: 1) imitation learning models, 2) vision-target-based policy models, and 3) vision-language-action (VLA) models.

[0004] Imitation learning models, trained on large amounts of artificial demonstration data, focus on specific tasks or scenarios and can achieve good performance under certain conditions. However, their generalization ability is poor due to the diversity of data and the limitations of the model. Vision-based policy models typically use target images or predefined rules to provide target guidance for robots. These models perform well in spatial localization and operational accuracy, but rely on high-quality data and specific task settings, making it difficult to adapt to more complex dynamic scenarios. Visual Language Algorithm (VLA) models, by combining large-scale robot training data with pre-trained Vision-Language Models (VLMs), demonstrate strong task reasoning capabilities and broad operational adaptability, supporting diverse robot operation tasks. However, these methods still face the challenge of insufficient generalization ability when faced with entirely new scenarios or tasks, usually requiring large-scale datasets and additional model fine-tuning, which is not only costly but also limits the scalability of the models in practical applications. Summary of the Invention

[0005] To address the lack of generalization capability in existing robot operation technologies, this invention provides a generalizable robot operation method and system based on segmentation mask representation, aiming to achieve the universality of robot operation strategies in diverse scenarios and tasks.

[0006] The specific technical solution adopted in this invention is as follows:

[0007] In a first aspect, the present invention provides a generalizable robot operation method based on segmentation mask representation, wherein the robot consists of a gripper and a robotic arm, and includes the following steps:

[0008] (1) Construct a 3D object library and a desktop scene library, and randomly generate a large number of desktop scene layouts in a virtual environment based on the data in the library;

[0009] (2) Given a target object and a target area, generate robot operation trajectory data for the target object in the desktop scene; for each robot operation trajectory data, generate diverse text instructions by combining the appearance, spatial position relationship and common sense knowledge of all objects in the desktop scene.

[0010] (3) For each robot operation trajectory data, collect the robot's perspective image, robot state data and a text command under each operation step as a training sample to construct a training sample set; the robot state data includes the robot's gripper switch and the joint angle of the robot arm.

[0011] (4) Use the pre-trained multimodal large model to locate the target object and target region indicated by the text instruction in each training sample, and obtain the target object mask and target region mask;

[0012] (5) Input the target object mask, target region mask and training samples corresponding to several historical operation steps into the robot operation strategy network, extract robot view image features, robot state features and text command features, fuse the target object mask and target region mask with the robot view image features, and based on the fusion result and the robot state features, text command features and action features corresponding to learnable action tokens, predict the action command of the robot's next operation step, and calculate the loss based on the robot's real state and predicted action command in the next operation step to train the robot operation strategy network.

[0013] (6) Use the pre-trained multimodal large model and the trained robot operation policy network to complete the given instructions in the actual desktop scenario.

[0014] Furthermore, the desktop scene layout in the virtual environment includes the desktop scene, a target object, and several interfering objects.

[0015] Furthermore, the objects in the 3D object library contain attribute information, including category, color, shape, and material.

[0016] Furthermore, in step (2), each robot operation trajectory data corresponds to several text instructions.

[0017] Furthermore, the text instructions are implemented by a large language model, and the input of the large language model includes the target object, the target region, the robot's initial view image, the attribute information of all objects in the desktop scene, and relevant prompt words.

[0018] Furthermore, the pre-trained multimodal large model described in step (4) includes an image encoder, a multilayer perceptron, a large language model, and a localization module;

[0019] Step (4) specifically includes:

[0020] (4-1) Use an image encoder to obtain the robot's initial viewpoint image x from the training samples. v,0 Encode features and project them onto the embedding feature space of a large language model through a multilayer perceptron;

[0021] (4-2) A localization module is initialized by a pre-trained SAM model. The localization module includes a pre-trained image encoder and a pre-trained image decoder. The localization module is connected after the large language model.

[0022] (4-3) Add special tokens to the vocabulary of the large language model <seg>Given prompts from a large language model, the model is required to locate the target object and target region indicated in the text instruction based on the input image encoding features and text instruction, and generate text descriptions accordingly. Where CLIP(·) represents the image encoder, f v (·) denotes a multilayer perceptron, x t Indicates text instructions. Representing a large language model, y t,1 This represents the text description of the target object specified in the text command, y t,2 The text description indicating the location of the target area specified in the text instruction;

[0023] (4-4) When the text output of a large language model contains tokens <seg>At that time, the feature vector of the last hidden layer before text output is input into the localization module to obtain the corresponding segmentation mask. Where M o M is the mask for the target object. p The target region mask is defined by two segmentation masks, both of which are 0-1 mask matrices of the same size as the robot's view image; ε(·) represents the pre-trained image encoder in the localization module. This refers to the pre-trained image decoder in the localization module. Indicates y t,1 Output the feature vector of the last hidden layer. Indicates y t,2 Output the feature vector of the last hidden layer.

[0024] Furthermore, the robot operation strategy network includes a pre-trained image encoder, a pre-trained text encoder, a multilayer perceptron, a localization perceptron, and a Transformer decoder, wherein the localization perceptron is composed of several attention layers.

[0025] Step (5) specifically includes:

[0026] (5-1) For the robot's view image at each operation step, the robot's view image features are obtained using the pre-trained image encoder in the robot operation policy network. Includes a global image feature and a local image feature Among them, D v Feature dimensions encoded for an image;

[0027] (5-2) Each localization sensor initializes a global query vector. A target object query vector A target region query vector Where D p The initial feature dimension;

[0028] (5-3) In the first attention layer of the localization perceptron, the three vectors are concatenated and projected onto the hidden layer space to obtain the query vector. Where d represents the feature dimension of the hidden layer; robot's perspective image features After projection through different projection matrices, the vectors are concatenated with the query vector Q to obtain the key vectors. Sum value vector Calculate the attention matrix based on the query vector Q and the key vector K.

[0029] (5-4) Mask the target object M o and target area mask M p Map each feature map to a size of 14×14, then convert it into a one-dimensional vector of size 1×196, and then M... o The corresponding one-dimensional vector is applied to A [1,:196] M p The corresponding one-dimensional vector is applied to A [2,:196] This replaces the attention values ​​in the masked region with the maximum value of the current matrix, resulting in the updated attention matrix A. ′ Next, calculate A. ′ The softmax function is multiplied by the value vector V, and then passed through the feedforward network FFN(·) to obtain the output O of the attention layer: O = FFN(softmax(A) ′ )×V);

[0030] (5-5) Return to step (5-3) and use the output O of the previous attention layer as the query vector for the next attention layer. Until the output of the last attention layer is obtained, the final output image features fused with mask information are denoted as...

[0031] (5-6) Extract robot state features Z using a pre-trained text encoder and a multilayer perceptron, respectively. t Text instruction features Z s And, initialize a learnable action token. <act>And extract action features Z a ; Obtain the input sequence corresponding to each operation step

[0032] (5-7) Input the above input sequence corresponding to N historical operation step data into the Transformer decoder to predict the next action instruction, which includes gripper action and robotic arm action.

[0033] Furthermore, step (6) specifically includes:

[0034] (6-1) In a real desktop scenario, given a text instruction, the robot is required to move the target object to a specified target area; the robot's initial view image and the given text instruction are input into a pre-trained multimodal large model to generate a target object mask and a target area mask;

[0035] (6-2) Maintain an input sequence consisting of N historical operation steps. When there are fewer than N operation steps in the historical data, the latest historical operation steps are copied as a supplement. The input sequence corresponding to each operation step is obtained in the same way as in the training phase. The robot operation policy network after training generates the next action instruction. This process is repeated until the robot completes the task given the text instruction.

[0036] Furthermore, the pre-trained image encoder in the robot operation strategy network adopts the ViTMAE image encoder.

[0037] Secondly, this invention proposes a generalizable robot operating system based on segmentation mask representation, used to implement the aforementioned generalizable robot operation method based on segmentation mask representation.

[0038] Compared with the prior art, the beneficial effects of this invention are:

[0039] This invention is a generalizable robot operation method and system based on segmentation mask representation. In its implementation, this invention utilizes a multimodal large model to generate a segmentation mask, and uses this as an intermediate representation to guide the training and execution of robot operation strategies.

[0040] (1) By using a multimodal large model, this invention can jointly extract semantic information and visual features from robot-perspective images and text commands to generate segmentation masks for target objects and placement areas. As a pre-trained general knowledge system, the multimodal large model can effectively adapt to different tasks and scenarios, providing high-precision object localization and region labeling, and providing reliable input for subsequent operation strategies.

[0041] (2) By using segmentation mask representation, this invention combines the target object and scene mask generated by the multimodal large model with the robot's perspective image and robot state information, providing precise spatial and semantic guidance for operation strategies. The segmentation mask not only clearly defines the position and shape of the target object, but also significantly enhances the robot's operational accuracy and scene adaptability in complex tasks.

[0042] (3) This invention can automatically generate a large-scale training set containing diverse objects, complex scenes, and rich task instructions to improve the generalization ability of the model. By combining the segmentation mask generated by the multimodal large model with this diverse data, the adaptability of the operation strategy to unknown tasks and scenes is effectively enhanced.

[0043] In summary, this invention significantly improves the robot's generalization ability in multi-task and multi-scenario environments by generating a segmentation mask using a multimodal large model and using it as an intermediate representation to guide the learning and execution of robot operation strategies. This method improves operational accuracy while adapting to diverse task instructions and complex scenarios, providing a novel technical solution for achieving efficient and generalizable robot operation. Attached Figure Description

[0044] Figure 1 This is a flowchart of the generalizable robot operation method based on segmentation mask representation of the present invention;

[0045] Figure 2 This is a schematic diagram of a multimodal large model;

[0046] Figure 3 This is a schematic diagram of a robot operation strategy network. Detailed Implementation

[0047] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the present invention provides a generalizable robot operation method based on segmentation mask representation, comprising the following steps:

[0049] Step 1: Obtain diverse 3D objects and their desktop scene layouts. Randomly select target objects and interference objects from the 3D objects and place them in the target desktop scene. Record the robot's initial view image. Here, the robot's initial view image includes the target objects.

[0050] An automated trajectory generation algorithm is used to generate robot operation trajectory data for target objects in a target desktop scene. Combined with the appearance, spatial relationship and common sense of all objects in the target desktop scene, a variety of text commands are generated. Here, each robot operation trajectory data corresponds to several text commands.

[0051] Step 2: For each robot operation trajectory data, collect the robot's perspective image, robot state data, and one text command for each operation step as a training sample to construct a training set;

[0052] Step 3: Use a pre-trained multimodal large model to locate the target object and target region indicated by the text instruction in each training sample, and obtain the target object mask and target region mask;

[0053] Step 4: Input the target object mask, target region mask, and training samples into the downstream robot operation strategy network. Use the pre-trained image encoder to extract robot view image features. Use the attention mechanism to fuse the target object mask, target region mask, and robot view image features. Use the Transformer decoder to predict the robot's next action command. The action command includes gripper action and robotic arm action.

[0054] Step 5: For the predicted action commands, calculate the L1 loss of the gripper action and the binary cross-entropy loss of the robotic arm action respectively. Use this loss to jointly train the robot operation policy network, thereby improving its adaptability in multiple tasks and multiple scenarios.

[0055] The robot manipulation strategy network of this invention achieves a generalizable robot manipulation method by performing cross-modal joint training on automatically generated diverse datasets, utilizing the generalization ability of a multimodal large model and the precise spatial guidance of segmentation masks. This method can maintain high performance in complex scenarios and diverse tasks.

[0056] Step 6: Combine the pre-trained multimodal large model and the trained robot operation policy network to complete the given instructions in the actual desktop scenario.

[0057] In step 1 above, an automated robot operation data generation process is used, as detailed below:

[0058] 1.1) Collecting high-quality 3D object assets: In this embodiment, for the 760K 3D objects in the public dataset Objaverse, render their multi-view images, use the GPT-4o large model to generate detailed description information of the objects, including attribute information such as category, color, shape and material, and filter out 3526 objects suitable for desktop scene robot operation through rules.

[0059] 1.2) The open-source robot simulation simulator RoboCasa is used to generate diverse desktop scene layouts. Simultaneously, a target object A and 3-10 interfering objects B are randomly selected from the 3D object assets and placed in the scene. RoboCasa automatically annotates some reference trajectories, which represent the robot's gripper movements and robotic arm joint movements when an object is placed in a certain area. These reference trajectories are used to provide guidance for the downstream MimicGen. It should be noted that the reference trajectories do not need to be specific to target object A. If other methods are used to generate the desktop scene layout, some reference trajectories can be manually annotated.

[0060] 1.3) Using the open-source automated trajectory generation solution MimicGen, robot operation trajectory data of the target object in the target desktop scene is generated based on the reference trajectory. In the process of automated trajectory generation, the target area can be given. By updating the desktop scene layout, target object and interference object in step 1.2), a large amount of robot operation trajectory data can be generated.

[0061] Here, generating robot operation trajectory data can also be achieved using other automated methods.

[0062] 1.4) For each robot operation trajectory data, the GPT-4o large model is used to generate text instructions for appearance, spatial position relationship and common sense knowledge based on the target object, target area, robot initial view image and all object attribute information in the desktop scene; the diverse text instructions help improve the model's generalization ability for different instructions.

[0063] For example, for a direct instruction: "Bring the water cup to the seat", a text instruction based on appearance can be generated: "Bring the transparent object with the handle to the seat", a text instruction based on spatial relationship can be generated: "Bring the object to the left of the kettle to the seat", and a text instruction based on common sense knowledge: "I want to drink water".

[0064] In step 2 above, for each robot operation trajectory data, the robot's state within each unit of time can be obtained. This state refers to the robot's gripper switch and the joint angles of the robotic arm. Taking the robot used in this embodiment as an example, the robot state includes the gripper switch with one degree of freedom and the joint angles of the robotic arm with seven degrees of freedom. Additionally, a robot perspective image can be obtained within each unit of time. The robot perspective images and robot state during the operation process are recorded. The robot perspective image, robot state data, and one text command for each operation step are collected as a training sample to construct a training set.

[0065] In step 3 above, such as Figure 2 As shown, the specific steps for obtaining the segmentation mask corresponding to the target object and the target region are as follows:

[0066] 3.1) Extract the robot's initial viewpoint image x from the training data. v and text command x t Input a pre-trained multimodal large model, which includes a CLIP image encoder, a multilayer perceptron, a large language model, and a localization module;

[0067] 3.2) Obtaining image coding features F using the CLIP image encoder v = CLIP(x v Using a multilayer perceptron f v (·) Project it onto the embedding feature space of the large language model to obtain Where D t For embedding feature dimensions;

[0068] 3.3) To achieve localization of target objects or regions, an additional localization module is added after the large language model, consisting of a pre-trained image encoder ε(·) and a pre-trained image decoder. In this embodiment, the components are a pre-trained image encoder ε(·) and a pre-trained image decoder. Initialized from a pre-trained SAM model;

[0069] 3.4) Add special tokens to the vocabulary of the large language model <seg>Given prompts, the large language model is required to locate the target object and target region indicated by the text command and generate text descriptions accordingly. Under the prompts, the large language model... Based on image encoding features and text instruction x t Input, get text output When the text output of a large language model contains a token <seg>When, the corresponding last layer feature vector F seg Input localization module; since the model outputs two separate texts for the target object and the target region, it is necessary to determine whether each text contains a token. <seg>and two different feature vectors The corresponding segmentation mask is obtained by inputting the positioning module. Where M o M is the mask for the target object. p The target region mask is defined by two segmentation masks, both of which are 0-1 mask matrices of the same size as the robot's viewpoint image.

[0070] In step 4 above, such as Figure 3 As shown, the robot operation policy network includes a pre-trained image encoder, a pre-trained text encoder, a multilayer perceptron, a localization sensor, and a Transformer decoder. The localization sensor is composed of several attention layers. An optional implementation of the robot operation policy network is as follows:

[0071] 4.1) For each input operation step, the robot's view image x v Image coding features are obtained using a pre-trained image encoder ViTMAE. Where D v This is the feature dimension for image encoding, which contains the global CLS image feature vector. and the local 14×14 feature map embedding vector

[0072]

[0073] 4.2) Initialize a global query vector for each localization sensor. A target object query vector A target region query vector Where D p The initial feature dimension;

[0074] 4.3) To parallelize the processing, in the first attention layer, the three vectors are concatenated and projected onto the hidden layer space to obtain the query vector. Where d is the hidden layer feature dimension, and d is the feature map embedding vector of the robot's view image. The key vector is obtained by concatenating the projection with the query vector Q. Sum value vector Calculate the attention matrix based on the query vector Q and the key vector K.

[0075] 4.4) To introduce segmentation mask information into the attention matrix, the segmentation masks for the target object and the region are mapped to feature maps of size 14×14, and then converted into one-dimensional vectors of size 1×196. Then, M... o The corresponding one-dimensional vector is applied to A [1,:196] M p The corresponding one-dimensional vector is applied to A [2,:196] This replaces the attention values ​​in the masked region with the maximum value of the current matrix, resulting in the updated attention matrix A. ′ Next, calculate A. ′ The softmax function is multiplied by the value vector V, and then passed through the feedforward network FFN(·) to obtain the output O of the attention layer: O = FFN(softmax(A) ′ )×V);

[0076] 4.5) The localization perceptron consists of L attention layers as described above. Return to step 4.3), and use the output O of the previous attention layer as the query vector for the next attention layer. Until the output of the last attention layer is obtained, the final output image features fused with mask information are denoted as...

[0077] 4.6) For text instruction x t The text feature vector Z is extracted using the pre-trained text encoder CLIP. t ;

[0078] For robot state information x s The state feature vector Z is obtained by projection using a multilayer perceptron (MLP). s ;

[0079] For action commands, initialize a learnable action token. <act>Its corresponding action feature vector is Z. a ;

[0080] Therefore, the input sequence at the current time is denoted as

[0081] 4.7) The complete input, i.e. the above input sequence containing N historical operation step data, is used to predict the next action instruction using the Transformer decoder. The action instruction includes a gripper action with 1 degree of freedom and a robotic arm action with 7 degrees of freedom.

[0082] In step 5 above, the loss function for jointly training the robot operation strategy network includes gripper action loss and robotic arm action loss.

[0083] In one specific embodiment of the present invention, taking the robot used as an example, the loss function includes the gripper motion loss of one degree of freedom, which is calculated using a binary cross-entropy function. In addition, the motion loss of the robotic arm with 7 degrees of freedom is calculated using the L1 function. Total loss is

[0084] Using this loss function, the gradient descent learning method is employed to train the learnable parameters involved in the model. Since the training data contains diverse operational tasks, joint training on multiple different operational tasks can be achieved, enabling the model to complete multiple robot operation tasks simultaneously.

[0085] In step 6 above, the pre-trained multimodal large model and the trained robot operation policy network are combined to complete the given instructions in the actual desktop scenario.

[0086] In a real-world desktop scenario, given a text instruction, the robot is instructed to move target object A to a designated target area. The robot's initial viewpoint image and the given text instruction are input into a pre-trained multimodal large model to generate a target object mask and a target area mask.

[0087] Maintain an input sequence consisting of N historical operation steps. When there are fewer than N operation steps in the historical data, the latest historical operation step data is copied as a supplement. The input sequence corresponding to each operation step data is obtained in the same way as in the training phase. The robot operation policy network after training generates the next action instruction. This process is repeated until the robot completes the task given the text instruction.

[0088] The above method will be applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0089] This invention utilizes the RoboCasa simulator to conduct simulation experiments on a pick and place task. To objectively evaluate the performance of this invention, 400 test data points were randomly generated, and the success rate of task completion was used for evaluation. The results were then compared with the following prior art models:

[0090] In contrast, the ACT model is a Transformer-based policy network trained using a conditional variational autoencoder and employing an action block mechanism for action prediction. While this improves prediction efficiency, the block-based approach limits its performance in fine-grained tasks and is susceptible to pattern collapse, resulting in insufficient diversity in action generation.

[0091] In contrast, the BC-Transformer model is a multimodal fusion model based on behavioral cloning, which uses CLIP coding language as input and combines visual features. Although it has strong fusion capabilities, the model is highly dependent on high-quality demonstration data, has poor adaptability to unseen scenarios, and its generalization ability is limited in complex tasks.

[0092] In contrast, the GR-1 model is a GPT-style robot manipulation model that predicts actions by combining visual and linguistic information. While its design can predict future actions and scenarios, the model relies heavily on large-scale pre-training, resulting in significant computational overhead and data requirements.

[0093] The experimental results obtained by following the steps described in the specific implementation method are shown in Tables 1 and 2.

[0094] Table 1: Success rate of robot operation test in RoboCasa simulation scenario according to the present invention

[0095]

[0096] Table 2: Success rate of robot grasping and placement operations in new object scenarios according to the present invention

[0097]

[0098] As shown in Table 1, the success rate of this invention in robot operation tasks under various types of text instructions is significantly better than that of existing state-of-the-art methods. For grasping and placement tasks, the success rate of this invention in difficult task settings involving appearance, spatial relationships, and common-sense reasoning is improved from 13.8 to 30.5 (an improvement of 121%), from 16.3 to 33.5 (an improvement of 105%), and from 11.5 to 30.0 (an improvement of 160%), respectively, demonstrating significant performance improvements. These improvements are attributed to the invention's use of a pre-trained multimodal large model to generate segmentation masks for target objects and regions, fully utilizing the powerful reasoning and localization capabilities of the multimodal large model, thereby significantly enhancing the overall performance of robot operation.

[0099] As shown in Table 2, without segmentation mask information, the success rate of this invention in testing new object categories is relatively limited. However, with the addition of segmentation mask information, the success rate is significantly improved. This indicates that segmentation mask is an effective intermediate representation that can greatly improve its performance in manipulating unseen objects and its task generalization ability, thereby achieving an efficient and generalizable robot manipulation strategy.

[0100] This embodiment also provides a generalizable robot operating system based on segmentation mask representation, which is used to implement the above embodiments. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that implement a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.

[0101] A generalizable robot operating system based on segmentation mask representation, comprising:

[0102] The training sample generation module is used to construct a 3D object library and a desktop scene library, and randomly generate a large number of desktop scene layouts in virtual environments based on the data in the library; given a target object and a target area, it generates robot operation trajectory data for the target object in the desktop scene; for each robot operation trajectory data, it generates diverse text instructions by combining the appearance, spatial position relationship and common sense knowledge of all objects in the desktop scene; for each robot operation trajectory data, it collects the robot's perspective image, robot state data and a text instruction at each operation step as a training sample to construct a training sample set; the robot state data includes the robot's gripper switch and the joint angle of the robotic arm;

[0103] The multimodal large model module uses a pre-trained multimodal large model to locate the target object and target region indicated by the text instruction in each training sample, and obtains the target object mask and the target region mask.

[0104] The operation strategy network module takes the target object mask, target region mask, and training samples corresponding to several historical operation steps as input, extracts robot view image features, robot state features, and text command features, fuses the target object mask and target region mask with the robot view image features, and predicts the action command for the robot's next operation step based on the fused result and the robot state features, text command features, and action features corresponding to learnable action tokens.

[0105] The operation strategy network training module is used to calculate the loss based on the robot's actual state and the predicted action command for the next operation step in order to train the robot's operation strategy network.

[0106] The robot module is used to execute given instructions in real-world desktop scenarios using a pre-trained multimodal large model and a trained robot operation policy network.

[0107] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0109] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.< / act> < / seg> < / seg> < / seg> < / act> < / seg> < / seg>

Claims

1. A generalizable robot operation method based on segmentation mask representation, the robot is composed of a gripper and a robot arm, characterized in that, The method comprises the following steps: (1) constructing a three-dimensional object library and a desktop scene library, and randomly generating a large number of desktop scene layouts in a virtual environment according to the data in the library; (2) generating robot operation trajectory data for a target object in the desktop scene according to a given target object and target area; for each piece of robot operation trajectory data, diversified text instructions are generated in combination with the appearance, spatial position relationship and common sense knowledge of all objects in the desktop scene; (3) for each piece of robot operation trajectory data, a robot visual angle image, robot state data and a text instruction at each operation step are collected as a training sample to construct a training sample set; the robot state data includes a gripper switch and joint angle of a robot arm; (4) using a pre-trained multi-modal large model to locate a target object and a target area indicated by the text instruction in each training sample to obtain a target object mask and a target area mask; (5) inputting the target object mask, the target area mask and the training sample corresponding to a plurality of historical operation steps into a robot operation strategy network to extract robot visual angle image features, robot state features and text instruction features, fusing the target object mask and the target area mask with the robot visual angle image features, and based on the fused results, the robot state features, the text instruction features and action features corresponding to learnable action tokens, predicting an action instruction of a next operation step of the robot, and calculating a loss according to a real state of the robot at the next operation step and the predicted action instruction to train the robot operation strategy network; Step (5) specifically comprises: (5-1) For each robot perspective image of the operation step, the robot perspective image features are obtained by using the pre-trained image encoder in the robot operation strategy network wherein, is the feature dimension of the image encoding. (5-2) The positioning sensor initializes a global query vector, a target object query vector, and a target region query vector, respectively wherein is the initial feature dimension;​​ (5-3) In the first attention layer of the positioning sensor, the three vectors are connected together and projected to the hidden layer space to obtain the query vector wherein is the hidden layer feature dimension; the robot perspective image feature is projected through different projection matrices and combined with the query vector to obtain the key vector and the value vector , and the attention matrix is calculated according to the query vector , the key vector ; (5-4) mapping the target object mask and the target region mask to the feature map size of respectively, and converting them into one-dimensional vectors with size , then applying the one-dimensional vector corresponding to to , and applying the one-dimensional vector corresponding to to , so that the attention value of the mask region is replaced by the maximum value of the current matrix, obtaining the updated attention matrix ; next, calculating the , multiplying the value vector , and obtaining the output of the attention layer through the feedforward network ; (5-5) Return to step (5-3) and concatenate the output of the previous attention layer query vector computed as the next attention layer , until the output of the last attention layer is obtained, and the final output image feature fused with the mask information is denoted as ; (5-6) Extract robot state features using pre-trained text encoder, multi-layer perceptron, respectively , text instruction features , and initialize a learnable action token <act>And extract action features ; get the input sequence corresponding to each operation step ;< / act> (5-7) inputting the above-mentioned input sequences corresponding to N pieces of historical operation step data into a Transformer decoder to predict an action instruction of the next step, wherein the action instruction includes a gripper action and a robot arm action; (6) using the pre-trained multi-modal large model and the trained robot operation strategy network to complete a given instruction in an actual desktop scene.

2. The segmentation mask representation based generalizable robot manipulation method of claim 1, wherein, In step (1), the desktop scene layout in the virtual environment includes a desktop scene, a target object and a plurality of interference objects.

3. The generative robot operation method based on segmentation mask representation according to claim 1 or 2, characterized in that, The objects in the three-dimensional object library contain attribute information, and the attribute information includes category, color, shape and material.

4. The segmentation mask representation based generalizable robot manipulation method of claim 1, wherein, In step (2), each piece of robot operation trajectory data corresponds to a plurality of text instructions.

5. The segmentation mask representation based generalizable robot manipulation method of claim 4, wherein, The text instruction is realized by a large language model, and the input of the large language model includes a target object, a target area, a robot initial visual angle image, attribute information of all objects in the desktop scene, and related prompt words.

6. The segmentation mask representation based generalizable robot manipulation method of claim 1, wherein, The pre-trained multi-modal large model in step (4) includes an image encoder, a multi-layer perception, a large language model and a positioning module. Step (4) specifically comprises: (4-1) acquiring the initial view image of the robot in the training sample by using an image encoder encode the features and project them to the embedding feature space of the large language model through a multi-layer perception. (4-2) initializing a positioning module from a pre-trained SAM model, wherein the positioning module includes a pre-trained image encoder and a pre-trained image decoder, and the positioning module is connected after the large language model; (4-3) Adding special tokens to the vocabulary of large language models <seg>and given a large language model prompt, requiring the large language model to locate the target object and the target region referred to in the text instruction according to the input image encoding features and the text instruction, and generate a literal description ; wherein, denotes an image encoder, denotes a multi-layer perception, denotes a text instruction, denotes a large language model, denotes a literal description of the target object referred to in the text instruction, denotes a literal description of the target region referred to in the text instruction;< / seg> (4-4) When the token <seg>The last layer hidden layer feature vector before the text output is input into the positioning module to obtain a corresponding segmentation mask 、 , wherein is a target object mask, is a target region mask, both of which are 0-1 mask matrices with the same size as the robot perspective image; represents a pre-trained image encoder in the positioning module, represents a pre-trained image decoder in the positioning module, represents the corresponding last layer hidden layer feature vector before output, represents the corresponding last layer hidden layer feature vector before output.< / seg> 7. The segmentation mask representation based generalizable robot manipulation method of claim 1, wherein, The robot operation strategy network comprises a pre-trained image encoder, a pre-trained text encoder, a multi-layer perception, a positioning sensor, and a Transformer decoder, wherein the positioning sensor is composed of a plurality of attention layers.

8. The segmentation mask representation based generalizable robot manipulation method of claim 7, wherein, The step (6) specifically comprises: (6-1) in an actual desktop scene, a text instruction is given, and the robot is required to take a target object to a specified target area; an initial visual image of the robot and the given text instruction are input into a pre-trained multi-modal large model to generate a target object mask and a target area mask; (6-2) maintain an input sequence composed of N historical operation step data When the historical data is less than N operation steps, the latest historical operation step is copied as a supplement; the input sequence corresponding to each operation step is obtained in the same way as in the training phase, and the next action instruction is generated by the trained robot operation strategy network, and the process is repeated until the robot completes the task of the given text instruction.

9. The segmentation mask representation based generalizable robot manipulation method of claim 7, wherein, The pre-trained image encoder in the robot operation strategy network adopts a ViTMAE image encoder.

10. A generalizable robot operating system based on segmentation mask representation, for implementing the method of claim 1; characterized in that, The system comprises: A training sample generation module is configured to construct a three-dimensional object library and a desktop scene library, randomly generate a large number of desktop scene layouts in virtual environments according to the data in the libraries, generate robot operation trajectory data for a target object in a desktop scene by giving the target object and a target area, generate diversified text instructions for each robot operation trajectory data in combination with the appearance, spatial position relationship and common sense knowledge of all objects in the desktop scene, collect robot visual images, robot state data and a text instruction at each operation step as a training sample for each robot operation trajectory data to construct a training sample set, and the robot state data comprises a gripper switch of the robot and a joint angle of the robot arm; A multi-modal large model module is configured to locate a target object and a target area indicated by a text instruction in each training sample by using a pre-trained multi-modal large model to obtain a target object mask and a target area mask; An operation strategy network module is configured to input a target object mask, a target area mask and a training sample corresponding to a plurality of historical operation steps, extract robot visual image features, robot state features and text instruction features, fuse the target object mask and the target area mask with the robot visual image features, and predict an action instruction of a next operation step of the robot based on the fused results, the robot state features, the text instruction features and action features corresponding to learnable action tokens. An operation strategy network training module is configured to calculate a loss according to a real state of the robot at the next operation step and the predicted action instruction to train the robot operation strategy network. A robot module is configured to complete a given instruction in an actual desktop scene by using the pre-trained multi-modal large model and the trained robot operation strategy network.

Images