A method for cluttered scene object grasping based on visual-linguistic-action joint modeling

Abstract

The application discloses a method for cluttered scene target object grasping based on visual-language-action joint modeling. The application uses object-centered representation to realize a method for cluttered scene target object grasping based on visual-language-action joint modeling, processes the object-centered representation through a pre-trained visual-language model and a grasping model, obtains visual-language features and grasping features of each bounding box, and uses a transformer to implement cross-attention mechanisms among visual-language-action multimodality, generates visual-language-action cross-attention features, and then generates decisions and executes, so that higher sample utilization is realized, and additional data collection and training in the simulation-physical migration process are avoided; compared with a two-stage strategy, visual attributes and planner screening rules for language-visual matching need not be artificially designed, so that more flexible language instructions can be adapted, and better task generalization is achieved.

Description

Technical Field

[0001] This invention relates to a method for grasping target objects in cluttered scenes, specifically, a method for grasping target objects in cluttered scenes based on vision-language-action joint modeling. Background Technology

[0002] Currently, object grasping in cluttered environments is a crucial task in robotics and a key technology for operating service robots. Common methods for specifying target objects include providing images of the objects or collecting demonstration data on object grasping. However, users often struggle to provide such information, especially in open-scene applications such as organizing an open desktop.

[0003] One intuitive idea is to use natural language instead of visual commands to specify target objects. Recent works have utilized pre-trained large language models or multimodal models to learn robot manipulation strategies from raw images. However, learning these image-based strategies requires extensive simulation data for training and additional data to achieve simulation-to-physical transfer of the strategy, which becomes even more difficult in cluttered scenes. In contrast, object-centered representations provide the robot with information about objects in the scene, accelerating scene understanding and improving grasping efficiency. Based on this idea, another class of works decouples language-based target object grasping into two stages: language-visual matching to find the target object and planning and executing the grasping action. These works generate object-centered representations using bounding boxes, perform language-visual matching based on object category and visual attributes, and finally use a rule-based planner to select the target object. However, these hand-designed attributes and rules limit the generalization of language commands. Furthermore, errors in language-visual matching and interference from other objects in cluttered scenes severely affect the successful grasping of the target object. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the present invention aims to provide a method for object grasping in cluttered scenes based on joint vision-language-action modeling. This invention proposes a method for predicting grasping pose based on joint modeling of vision, language, and action using object-centered representations.

[0005] Specifically, this method uses a pre-trained visual-language model to encode visual-language features from images and language commands within the object's bounding box. A pre-trained grasping network generates a set of grasping poses, which are then encoded into spatial features by a grasping encoding network. The method uses a transformer to generate visual-language-action cross-attention features, which the policy network uses to select the grasping action. The entire system is trained in a simulation environment using a model-free reinforcement learning approach. Compared to previous work, this method achieves higher sample utilization, requires no additional data for simulation-to-real-object transfer, and achieves better task performance and language generalization.

[0006] This invention is achieved through the following technical solution:

[0007] A method for object grasping in cluttered scenes based on vision-language-action joint modeling includes:

[0008] Step 1: Given the scenario and language instructions;

[0009] Step 2: Obtain the RGB-D image of the scene, obtain the bounding boxes of the detected objects in the scene using the open set object detection algorithm, and crop out the RGB image within the bounding box as the representation of the object within the bounding box;

[0010] Step 3: Obtain the linguistic features of the language instructions in Step 1 and the visual features of the bounding boxes of all objects in Step 2 through a pre-trained vision-language algorithm model;

[0011] Step 4: Based on the acquired scene RGB-D image, predict the set of grasping poses in the scene using a pre-trained grasping algorithm model. Each grasping pose is encoded into grasping action features through a grasping encoding network.

[0012] Step 5: Based on the language features obtained in Step 3, the visual features of all object bounding boxes, and the grasping action features obtained in Step 4, use the transformer to implement a cross-attention mechanism between multimodal features to generate visual-language-action cross-attention features.

[0013] Step 6: Based on the visual-language-action cross-attention features obtained in Step 5, generate probability values ​​for all grasping postures through the policy network and generate evaluation values ​​for all grasping postures through the value network.

[0014] Step 7: Select a grasping action based on the probability value of the grasping posture obtained in Step 6. The robot executes the corresponding grasping posture and moves to the fixed placement position.

[0015] Step 8: Repeat steps 2-7 until the target object specified by the language is successfully captured to complete the task. The number of attempts should be within the specified number.

[0016] Furthermore, in step 1, the language instruction specifies the attributes of the target object, including object category (banana), visual attributes (red, round), or functional attributes (drink).

[0017] Further, in step 2, a real-time RGB-D image of the desktop scene is acquired using a RealSense 515 depth camera. The open set object detection algorithm is ORE, which takes the RGB image of the scene as input and outputs a set of bounding boxes for the objects. The image representation of each bounding box is obtained by cropping the original RGB image sequentially, denoted as {o i} i＝1,..,n .

[0018] Further, in step 3, the pre-trained vision-language algorithm is CLIP. This algorithm is trained using millions of image-text data points, mapping vision and language to the same feature space, and then calculating cosine similarity to represent the similarity between images and text. The algorithm includes an image encoder and a text encoder; the image encoder is ViT, and the text encoder is a transformer. The image encoder sequentially encodes {o} i} i＝1,..,n Obtain its corresponding visual features The text encoder encodes language instructions to obtain language features f l .

[0019] Furthermore, in step 4, the pre-trained grasping algorithm is Graspnet-baseline. This algorithm takes the scene point cloud as input and outputs a set of grasping poses in that scene {g i} i＝1,..,k Then, a grasping encoding network (three-layer MLP) is used to encode the grasping pose into grasping action features.

[0020] Furthermore, step 5, generating visual-language-action cross-attention features, includes the following steps:

[0021] Step 51: Based on the camera extrinsic parameters and depth map, calculate the bounding boxes {o} of all objects sequentially. i} i＝1,..,n The 3D position coordinates of the center pixel are non-linearly mapped using trigonometric functions, and then encoded into the position features of the object bounding box using a position encoding network (three-layer MLP).

[0022] Step 52: Add the visual features and positional features of each object's bounding box to obtain the visual-positional features of each object's bounding box, denoted as . The visual-linguistic features of each object bounding box are obtained by element-wise multiplication of the visual and linguistic features of each object bounding box.

[0023] Step 53: Calculate visual-linguistic-motor features using the transformer cross-attention mechanism. Specifically, the attention mechanism can be represented as:

[0024] Attention(Q,K,V) = softmax(QK) T V;

[0025] Where Q, K, and V are query, key, and value, respectively. The Attention operator first calculates the similarity between query and key to obtain weights, then normalizes the weights using softmax to obtain usable weights, and finally sums the weights and value using weighted summation.

[0026] Based on the attention mechanism, grasp the action features As a query, the visual-positional features of the bounding box As the key, the visual-linguistic features of the bounding box As the value, the cross-attention features are calculated, resulting in visual-language-action cross-attention features.

[0027] Furthermore, in step 6, based on the visual-language-action cross-attention features obtained in step 5... The probability values ​​logits of all grasping poses are generated through a policy network (three-layer MLP), with a size of 1*k. The evaluation values ​​value of all grasping poses are generated through a value network, with a size of 1*k.

[0028] Furthermore, in step 7, the grasping action is selected based on the probability value logits of the grasping pose obtained in step 6. During the training process, the action is sampled with logits as the probability. During the testing process, the grasping action with the highest probability value is directly selected. Then, the robot executes the corresponding grasping pose and moves to the fixed placement position.

[0029] Further, step 8 repeats steps 2-7 until the target object specified by the language is successfully captured, with a maximum of 5 attempts to complete the task.

[0030] The beneficial effects of this invention are as follows:

[0031] This invention utilizes object-centered representation to implement a method for object grasping in cluttered scenes based on joint vision-language-action modeling. By processing the object-centered representation through a pre-trained vision-language model and a grasping model, visual-language features and grasping features of each bounding box are obtained. A transformer is then used to implement a cross-attention mechanism between the vision-language-action multimodals, generating vision-language-action cross-attention features, which in turn generate and execute decisions. Compared to strategies based on raw images, this method achieves higher sample utilization and avoids the additional data collection and training required for simulation-to-physical transfer. Compared to two-stage strategies, it eliminates the need for manually designing visual attributes for language-visual matching and filtering rules for the planner, thus adapting to more flexible language instructions and achieving better task generalization. Attached Figure Description

[0032] Figure 1 This is a system framework diagram of the present invention;

[0033] Figure 2 This is a data flow diagram of a specific implementation of the present invention; Detailed Implementation

[0034] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0035] The purpose of this invention is to address the shortcomings of existing language-based target object grasping methods in cluttered scenes, such as low sample utilization and poor generalization. This invention proposes a method for target object grasping in cluttered scenes based on vision-language-action joint modeling. The device for implementing this method includes a camera, a robotic arm and grippers, and a computer controller. Figure 1 This is a system framework diagram of the present invention. Figure 2 This is a data flow diagram of a specific implementation of the present invention; the specific implementation method of the present invention is as follows:

[0036] A method for object grasping in cluttered scenes based on vision-language-action joint modeling includes:

[0037] Step 1: Given the scenario and language instructions;

[0038] Step 2: Obtain the RGB-D image of the scene, and use the open set object detection algorithm to obtain the bounding boxes of objects in the scene;

[0039] Step 3: Obtain the linguistic features of the language instructions and the visual features of the bounding boxes of all objects through a pre-trained vision-language algorithm model;

[0040] Step 4: Based on the acquired scene RGB-D image, predict the set of grasping poses in the scene using a pre-trained grasping algorithm model. Each grasping pose is encoded into grasping action features through a grasping encoding network.

[0041] Step 5: Based on the language features obtained in Step 3, the visual features of all object bounding boxes, and the grasping action features obtained in Step 4, use the transformer to implement a cross-attention mechanism between multimodal features to generate visual-language-action cross-attention features.

[0042] Step 6: Based on the visual-language-action cross-attention features obtained in Step 5, generate probability values ​​for all grasping poses through the policy network and generate evaluation values ​​for all grasping poses through the value network.

[0043] Step 7: Select a grasping action based on the probability value of the grasping pose, and the robot executes the corresponding grasping pose and moves to the fixed placement position;

[0044] Step 8: Repeat steps 2-7 until the target object specified by the language is successfully captured until the task is completed, with the number of attempts within the specified number.

[0045] The specific operation flow of this invention can be described as follows: Real-time RGB-D images of the desktop scene are acquired using a Realsense515 depth camera; open-set object detection algorithms are used to detect bounding boxes; a visual-language model is used to generate linguistic features of language instructions and visual features of all bounding boxes; and a grasping algorithm is used to generate a set of grasping poses in the scene. Based on the visual-language features and grasping action features, a cross-attention mechanism between multiple modalities is implemented using a transformer to generate visual-language-action cross-attention features, which are then used to generate real-time decisions.

[0046] It should be noted that other methods, processes, or operating procedures obtained without creative effort based on this invention, without departing from the principles and spirit of this invention, are also included within the scope of protection of this invention.

[0047] In one specific embodiment of the present invention, in step 1, the language instruction specifies the attributes of the target object, such as object category (banana), visual attributes (red, round), or functional attributes (drink). Specifically, the language instruction can be: "Give me a banana.", "I need a round object.", or "I want something to drink."

[0048] In one specific embodiment of the present invention, in step 2, the open set object detection algorithm is ORE. This algorithm takes an RGB image of the scene as input and outputs a set of object bounding boxes. The original RGB image is cropped sequentially to obtain the image representation of each object bounding box, denoted as {o i} i＝1,..,n .

[0049] In step 3, the pre-trained vision-language algorithm is CLIP. This algorithm is trained using millions of image-text data points, mapping vision and language to the same feature space, and then calculating cosine similarity to represent the similarity between images and text. The algorithm includes an image encoder and a text encoder; specifically, the image encoder is ViT, and the text encoder is a transformer. The image encoder sequentially encodes {o} i} i＝1,..,n Obtain its corresponding visual features The text encoder encodes language instructions to obtain language features f l .

[0050] In step 4, the pre-trained grasping algorithm is Graspnet-baseline. This algorithm takes a scene point cloud as input and outputs a set of grasping poses for that scene. Specifically, this method first generates a scene point cloud from an RGB-D image of the scene using Open3D's create_point_cloud_from_rgbd_image interface. Then, it uses the Graspnet-1 Billion algorithm to generate a set of scene grasping poses {g i} i＝1,..,k Specifically, each grasping pose is represented as a 7-bit vector, including a 3-dimensional position vector and a 4-dimensional quaternion direction vector. Finally, a grasping encoding network (a three-layer MLP) is used to map the grasping pose to a high-dimensional space to obtain grasping action features.

[0051] In a specific embodiment of the present invention, step 5, based on the visual-language features obtained in step 3 and the grasping action features obtained in step 4, utilizes a transformer to implement a cross-attention mechanism between multimodalities to generate visual-language-action cross-attention features. Further, generating visual-language-action cross-attention features in step 5 includes the following steps:

[0052] Step 51: Based on the camera extrinsic parameters and depth map, calculate the bounding boxes {o} of all objects sequentially. i} i＝1,..,n The 3D position coordinates of the center pixel are non-linearly mapped using trigonometric functions, and then encoded into the position features of the object bounding box using a position encoding network (three-layer MLP).

[0053] Step 52: Add the visual features and positional features of each object's bounding box to obtain the visual-positional features of each object's bounding box, denoted as . The visual-linguistic features of each object bounding box are obtained by element-wise multiplication of the visual and linguistic features of each object bounding box.

[0054] Step 53: Calculate visual-linguistic-motor features using the transformer cross-attention mechanism. Specifically, the attention mechanism can be represented as:

[0055] Attention(Q,K,V) = softmax(QK) T V

[0056] Where Q, K, and V are query, key, and value, respectively. The Attention operator first calculates the similarity between query and key to obtain weights, then normalizes the weights using softmax to obtain usable weights, and finally sums the weights and value using weighted summation.

[0057] Based on the attention mechanism, grasp the action features As a query, the visual-positional features of the bounding box As the key, the visual-linguistic features of the bounding box As the value, the cross-attention features are calculated; specifically, the visual-language-action cross-attention features are obtained.

[0058] In one specific embodiment of the present invention, step 6 is based on the visual-language-action cross-attention features obtained in step 5. The probability values ​​logits of all grasping poses are generated through a policy network (three-layer MLP), with a size of 1*k. The evaluation values ​​value of all grasping poses are generated through a value network, with a size of 1*k.

[0059] In one specific embodiment of the present invention, step 7 selects a grasping action based on the probability value logits of the grasping pose obtained in step 6. Specifically, during training, action sampling is performed with logits as the probability; during testing, the grasping action with the highest probability value is directly selected. Then, the robot executes the corresponding grasping pose and moves to a fixed placement position.

[0060] In one specific embodiment of the invention, step 8 repeats steps 2-7 until the target object specified by the language is successfully captured, indicating that the task is complete. Specifically, the maximum number of attempts is 5.

[0061] This method employs model-free reinforcement learning for training. Specifically, it uses the Discrete SAC algorithm for data collection and training on the Pybullet simulator. During training, the network parameters of the vision-language model CLIP and the grasping model Graspnet-1 Billion are fixed. In each round, the scene contains 15 randomly stacked objects. A language instruction is given, which can specify the object's category, color, shape, or even function, such as "give me a container".

[0062] The above description is not intended to limit the present invention. It should be noted that, for those skilled in the art, various changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention, and these improvements and refinements should also be considered within the scope of protection of the present invention.

Claims

1. A method for grasping target objects in cluttered scenes based on vision-language-action joint modeling, characterized in that, include: Step 1: Given the scenario and language instructions; Step 2: Obtain the RGB-D image of the scene, and use the open set object detection algorithm to obtain the bounding boxes of objects in the scene; Step 3: Obtain the linguistic features of the language instructions in Step 1 and the visual features of the bounding boxes of all objects in Step 2 through a pre-trained vision-language algorithm model; Step 4: Based on the acquired scene RGB-D image, predict the set of grasping poses in the scene using a pre-trained grasping algorithm model. Each grasping pose is encoded into grasping action features through a grasping encoding network. Step 5: Based on the language features obtained in Step 3, the visual features of all object bounding boxes, and the grasping action features obtained in Step 4, use the transformer to implement a cross-attention mechanism between multimodal features to generate visual-language-action cross-attention features. Step 6: Based on the visual-language-action cross-attention features obtained in Step 5, generate probability values ​​for all grasping poses through a policy network; Step 7: Select a grasping action based on the probability value of the grasping pose obtained in Step 6. The robot executes the corresponding grasping pose and moves to the fixed placement position. Step 8: Repeat steps 2-7 until the target object specified by the language is successfully captured until the task is completed, with the number of attempts within the specified number; Step 5, generating visual-language-action cross-attention features, includes the following steps: Step 51: Calculate the bounding boxes of all objects sequentially based on the camera extrinsic parameters and depth map. The 3D position coordinates of the center pixel are non-linearly mapped using trigonometric functions, and then encoded into the position features of the object bounding box using a position encoding network. The location coding network is a three-layer MLP. Step 52: Add the visual features and positional features of each object's bounding box to obtain the visual-positional features of each object's bounding box, denoted as . The visual-linguistic features of each object's bounding box are obtained by element-wise multiplication of the visual and linguistic features. ; Step 53: Calculate visual-linguistic-motor features using the transformer cross-attention mechanism. Specifically, the attention mechanism can be represented as: ； in The query, key, and value are respectively. The Attention operator first calculates the similarity between the query and key to obtain the weights, then uses softmax to normalize the weights to obtain usable weights, and finally sums the weights and value. Based on the attention mechanism, grasp the action features As a query, the visual-positional features of the bounding box As the key, the visual-linguistic features of the bounding box As the value, the cross-attention features are calculated, resulting in visual-language-action cross-attention features. .

2. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 1, characterized in that, In step 1, the language instruction specifies the attributes of the target object, including object category, visual attributes, or functional attributes.

3. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 2, characterized in that, In step 2, a real-time RGB-D image of the desktop scene is acquired using a RealSense 515 depth camera. The open set object detection algorithm is ORE, which takes the RGB image of the scene as input and outputs a set of bounding boxes for the objects. The image representations of each bounding box are obtained by cropping the original RGB image sequentially, denoted as . .

4. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 3, characterized in that, In step 3, the pre-trained vision-language algorithm is CLIP. This algorithm is trained using millions of image-text data points, mapping vision and language to the same feature space, and then calculating cosine similarity to represent the similarity between images and text. The algorithm includes an image encoder and a text encoder; the image encoder is ViT, and the text encoder is a transformer. The image encoder encodes sequentially... Obtain its corresponding visual features The text encoder encodes language instructions to obtain language features. .

5. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 4, characterized in that, In step 4, the pre-trained grasping algorithm is Graspnet-baseline. This algorithm takes scene point clouds as input and outputs a set of grasping poses in that scene. Then, a grasping encoding network is used to encode the grasping pose into grasping action features. The aforementioned capture encoding network is a three-layer MLP.

6. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 5, characterized in that, In step 6, based on the visual-language-action cross-attention features obtained in step 5... The probability values ​​logits of all grasping poses are generated through a policy network, with a size of 1*k; the policy network is a three-layer MLP.

7. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 6, characterized in that, In step 7, the grasping action is selected based on the probability value logits of the grasping pose obtained in step 6. During training, the action is sampled with logits as the probability. During testing, the grasping action with the highest probability value is selected directly. Then, the robot executes the corresponding grasping pose and moves to the fixed placement position.

8. The method for grasping target objects in cluttered scenes based on vision-language-action joint modeling according to claim 7, characterized in that, Step 8: Repeat steps 2-7 until the target object specified by the language is successfully captured. The maximum number of attempts is 5.

Images