Robot grasping pose prediction method and device, equipment and storage medium

Abstract

The application discloses a robot grasping posture prediction method and device, equipment and storage medium, and relates to the field of robot vision. The method comprises the following steps: training an intention reasoning model by taking an object image and a text instruction with implicit intention as input, and taking a text instruction with explicit intention as expected output; training a region segmentation model by taking the object image and the text instruction with explicit intention as input, and taking a region segmentation image as expected output; performing intention reasoning on a to-be-processed image and a to-be-processed text instruction by using the trained intention reasoning model; segmenting the to-be-processed image by using the trained region segmentation model according to the to-be-processed text instruction with explicit intention; performing region-level marking in the to-be-processed image according to a to-be-processed region segmentation image, and generating a region-level grasping posture. The application improves the understanding ability of human intention and the accuracy of region segmentation, and improves the grasping posture prediction precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision, and in particular to a region-level robot grasping pose prediction method and device based on intention understanding, equipment and a storage medium. BACKGROUND

[0002] Existing robot grasping pose prediction methods mainly rely on analysis techniques, which focus on analyzing the geometric properties of objects or evaluating contact forces to determine the most stable grasping manner. However, these analysis techniques often struggle to effectively extend and maintain stability when faced with unfamiliar objects, especially irregularly shaped objects.

[0003] In recent years, data-driven methods have been used to train deep learning models based on convolutional neural architectures using large amounts of labeled data to predict robot grasping poses. However, this approach often suffers from overfitting problems and is difficult to generalize to unfamiliar objects, greatly limiting their effectiveness in practical applications. More critically, most existing methods rely solely on visual information to predict robot grasping poses, failing to implement reasoning based on input containing human intentions (such as text, audio, etc.).

[0004] With the development of multi-modal large models, more and more research has begun to apply these models to robot grasping pose prediction to improve the interaction capabilities of robots and humans. For example, some research work uses the understanding and fusion capabilities of multi-modal large models for text and images to achieve detection and positioning of grasping targets in cluttered scenes. However, these existing works can only predict grasping poses based on simple and explicit text instructions, lacking the ability to understand and reason about implicit human intentions in text instructions. In addition, in terms of vision, existing methods can only achieve positioning of grasping targets, failing to further generate grasping poses at a more granular region level. SUMMARY

[0005] The present application provides a robot grasping pose prediction method, device, equipment and storage medium to solve the problem that existing methods fail to implement human intention reasoning or lack understanding of human intentions, resulting in inaccurate grasping of targets.

[0006] In a first aspect, the present application provides a robot grasping pose prediction method, comprising:

[0007] constructing a sample data set; wherein the samples in the sample data set include object images and region name texts in the object images, region segmentation images, implicit intention text instructions and explicit intention text instructions;

[0008] constructing an intention reasoning model, taking the object image and the text instruction with the implied intention in the sample data set as input, and taking the text instruction with the corresponding explicit intention as expected output, training the intention reasoning model;

[0009] constructing a region segmentation model, taking the object image and the text instruction with the explicit intention in the sample data set as input, and taking the corresponding region segmentation image as expected output, training the region segmentation model;

[0010] obtaining a to-be-processed image and a to-be-processed text instruction with an implied intention, using the trained intention reasoning model to perform intention reasoning on the to-be-processed image and the to-be-processed text instruction with the implied intention, and obtaining a to-be-processed text instruction with an explicit intention;

[0011] According to the to-be-processed text instruction with the explicit intention, the trained region segmentation model is used to segment the to-be-processed image, and a to-be-processed region segmentation image is obtained.

[0012] According to the to-be-processed region segmentation image, region-level labeling is performed in the to-be-processed image, and a region-level grasping pose is generated.

[0013] Further, the sample data set is constructed, comprising:

[0014] According to the object category grasped by the robot, an object image is collected;

[0015] According to the object name in the object image, a generative model is used to perform grasping reasoning on the object image, generate a region name text and a corresponding text instruction with an implied intention;

[0016] According to the region name text, an expert model is used to segment the object image, and a corresponding region segmentation image is generated;

[0017] A generative model is used to extract the intention of the text instruction with the implied intention, and an explicit intention text instruction is obtained.

[0018] Further, the intention reasoning model is a pre-trained multi-modal visual language large model, and the multi-modal visual language large model comprises a visual processing module, a language processing module, a fusion module and an output module.

[0019] Further, the pre-trained multi-modal visual language large model is fine-tuned by using a LoRA supervision fine-tuning method, comprising:

[0020] A LoRA module is inserted into the language processing module of the pre-trained multi-modal visual language large model;

[0021] The visual processing module is used for feature extraction on the input object image to obtain an image feature vector; the language processing module is used for feature extraction on the input text instruction of the implicit intent to obtain a text feature vector; and the LoRA module is used for adjusting the text feature vector;

[0022] The fusion module is used for fusing the image feature vector and the adjusted text feature vector to obtain a fusion feature;

[0023] The output module is used for generating a predicted text instruction of an explicit intent according to the fusion feature;

[0024] A loss value is calculated according to the predicted text instruction of the explicit intent and the text instruction of the explicit intent, and the parameters of the LoRA module are adjusted according to the loss value.

[0025] Further, the region segmentation model comprises an image encoder, an image adapter, a text encoder, a text adapter, a transformer module and a pixel-level decoder, the image encoder is connected with the image adapter, the text encoder is connected with the text adapter, the image adapter and the text adapter are connected with the transformer module, and the transformer module is connected with the pixel-level decoder.

[0026] Further, the image encoder and the text encoder select a contrastive language-image pre-training model; during training of the region segmentation model, only the image adapter, the text adapter, the transformer module and the pixel-level decoder are trained.

[0027] Further, the to-be-processed image comprises an RGB image and a depth image; according to the to-be-processed region segmentation image, region-level marking is performed on the to-be-processed image, specifically comprising:

[0028] According to the to-be-processed region segmentation image, region-level marking is performed on the RGB image to obtain a 2D grasping region;

[0029] Based on the correspondence between the pixels in the RGB image and the depths in the depth image, based on the 2D grasping region, the depth values outside the grasping region in the depth image are filtered out to obtain a 3D grasping region.

[0030] In a second aspect, the present application further provides a robot grasping pose prediction device, comprising:

[0031] A data set construction unit is configured to construct a sample data set; wherein the samples in the sample data set comprise an object image, a region name text in the object image, a region segmentation image, a text instruction of an implicit intent and a text instruction of an explicit intent;

[0032] The intention reasoning model construction and training unit is configured to construct an intention reasoning model, input object images and text instructions with implied intentions in the sample data set, and text instructions with corresponding explicit intentions as expected output, and train the intention reasoning model.

[0033] The region segmentation model construction and training unit is configured to construct a region segmentation model, input object images and text instructions with explicit intentions in the sample data set, and corresponding region segmentation images as expected output, and train the region segmentation model.

[0034] The first acquisition unit is configured to acquire a to-be-processed image and a text instruction with an implied intention, perform intention reasoning on the to-be-processed image and the text instruction with the implied intention by using the trained intention reasoning model, and obtain a text instruction with an explicit intention.

[0035] The second acquisition unit is configured to segment the to-be-processed image by using the trained region segmentation model according to the text instruction with the explicit intention, and obtain a to-be-processed region segmentation image.

[0036] The prediction unit is configured to perform region-level labeling in the to-be-processed image according to the to-be-processed region segmentation image, and generate a region-level grasping pose.

[0037] In a third aspect, the present application further provides an electronic device, which comprises a memory, a processor, and a computer program / instruction stored in the memory, and the processor executes the computer program / instruction to implement the robot grasping pose prediction method of the present application.

[0038] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the robot grasping pose prediction method of the present application.

[0039] The present application has the following beneficial effects:

[0040] The present application can better understand and reason the human intention implied in the text instruction through the multi-modal visual language large model, improves the processing capability of complex text instructions, and improves the grasping accuracy and human-machine cooperation efficiency; the present application performs region segmentation in the image according to the text instruction with the explicit intention, further improves the grasping accuracy and adaptability; through the fine-tuning of the multi-modal visual language large model and the training of the region segmentation model, the present application can better generalize to unfamiliar objects and environments, and enhance the application potential in diversified scenes; the present application solves the problems of poor human-computer interaction, poor generalization, and low grasping success rate of the traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the following embodiment description will be briefly introduced. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0042] Figure 1 A flow chart of the robot grasping pose prediction method in the embodiment of the present application is shown.

[0043] Figure 2 A flow chart of the robot grasping pose prediction method in the embodiment of the present application is shown.

[0044] Figure 3 A structure block diagram of the robot grasping pose prediction device in the embodiment of the present application is shown.

[0045] Figure 4 An electronic device structure schematic diagram in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The technical solutions in the present application will be described clearly and completely in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present application.

[0047] The technical solutions of the present application will be described in detail in combination with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0048] Embodiment one

[0049] Figure 1 A flow chart of the robot grasping pose prediction method provided by the present application is shown. As shown in the figure, Figure 1 The robot grasping pose prediction method comprises the following steps:

[0050] Step S1: Construct a sample data set.

[0051] The samples in the sample data set include object images, region name texts in the object images, region segmentation images, text instructions with implicit intentions and text instructions with explicit intentions. One object image can correspond to multiple region name texts, one region name text corresponds to one region segmentation image and one text instruction with explicit intention, and one text instruction with explicit intention can correspond to multiple text instructions with implicit intention.

[0052] As Figure 2As shown, taking a teapot as an example, the teapot image corresponds to two region name texts, i.e., a handle and a lid; the handle corresponds to an explicit intention text instruction of grabbing the handle, and the lid corresponds to an explicit intention text instruction of grabbing the lid; the grabbing of the handle corresponds to an implicit intention text instruction of please pour tea into a cup, and the grabbing of the lid corresponds to an implicit intention text instruction of please add tea leaves to the teapot.

[0053] In the specific embodiments of the present application, constructing a sample data set comprises:

[0054] Step S1.1: integrating object categories suitable for robot grasping, and collecting corresponding object images, for example, collecting object images from PhraseCut, COCONut-PanCap, etc. data sets;

[0055] Step S1.2: according to the object name in the object image, using a generative model to perform grasping inference on the object image, generating region name text and corresponding implicit intention text instructions;

[0056] Step S1.3: according to the region name text, using an expert model to segment the object image, generating a corresponding region segmentation image;

[0057] Step S1.4: using a generative model to extract the intention of the implicit intention text instruction, obtaining explicit intention text instructions.

[0058] In this embodiment, the object image is an RGB image, the generative model is selected as a GPT-4 model, and the expert model is selected as a Grounding-SAM model. In step S1.3, the region segmentation image generated by the expert model can also be manually audited and corrected to improve the accuracy of region segmentation and thus improve the grasping accuracy.

[0059] For example, input the teapot image into the GPT-4 model, and the GPT-4 model generates:

[0060] Handle (i.e., region name text): please pour tea into a cup (i.e., implicit intention text instruction);

[0061] Lid (i.e., region name text): please add tea leaves to the teapot (i.e., implicit intention text instruction).

[0062] The expert model performs segmentation on the teapot image according to the region name text - handle, and generates a handle region segmentation image; the expert model performs segmentation on the teapot image according to the region name text - lid, and generates a lid region segmentation image. The GPT-4 model extracts the intent of the text instruction "please pour tea into the teacup", and obtains the text instruction with an explicit intent: grab the handle; the GPT-4 model extracts the intent of the text instruction "please add tea leaves to the teapot", and obtains the text instruction with an explicit intent: grab the lid.

[0063] Step S2: constructing an intent reasoning model, and training the intent reasoning model according to a sample data set.

[0064] In the specific embodiment of the present application, the intent reasoning model is a pre-trained multi-modal visual language large model VLMs, and the multi-modal visual language large model includes a visual processing module, a language processing module, a fusion module and an output module. The function of the intent reasoning model is to analyze the input text instruction with an implicit intent, and output a text instruction with an explicit intent in combination with the current environment (i.e. the object image), so as to guide the robot to perform a region-level grabbing operation in the current environment.

[0065] In order to enable the multi-modal visual language large model to understand human intent more deeply and improve the generalization of the multi-modal visual language large model, the pre-trained multi-modal visual language large model is fine-tuned based on a sample data set. During fine-tuning of the pre-trained multi-modal visual language large model, the object image and the text instruction with an implicit intent in the sample data set are input, and the corresponding text instruction with an explicit intent is taken as the expected output.

[0066] In the specific embodiment of the present application, the pre-trained multi-modal visual language large model is fine-tuned by using a LoRA (Low-Rank Adaptation) supervised fine-tuning method, which includes:

[0067] Step S2.1: inserting a LoRA module in the language processing module of the pre-trained multi-modal visual language large model;

[0068] Step S2.2: using the visual processing module to extract features from the input object image to obtain an image feature vector; using the language processing module to extract features from the input text instruction with an implicit intent to obtain a text feature vector; and using the LoRA module to adjust the text feature vector;

[0069] Step S2.3: using the fusion module to fuse the image feature vector and the adjusted text feature vector to obtain a fusion feature;

[0070] Step S2.4: using the output module to generate a predicted text instruction with an explicit intent according to the fusion feature;

[0071] Step S2.5: Calculate the loss value based on the predicted text instruction with explicit intent and the text instruction with explicit intent, and adjust the parameters of the LoRA module based on the loss value.

[0072] like Figure 2 As shown, the trained intent reasoning model includes a pre-trained multimodal visual language model and a LoRA module. During fine-tuning of the intent reasoning model, the pre-trained multimodal visual language model is frozen, and only a small number of parameters related to the LoRA module are adjusted. The LoRA-supervised fine-tuning method achieves efficient parameter fine-tuning by introducing low-rank matrix factorization on the basis of the pre-trained weight matrix, greatly reducing the consumption of computational resources, and enabling better generalization to unfamiliar objects and environments, thus enhancing its application potential in diverse scenarios. During the fine-tuning process, optimization algorithms such as gradient descent are used to update the parameters in the LoRA module.

[0073] By using a pre-trained multimodal visual language large model and LoRA module, we can gain a deeper understanding of human intentions and obtain clear text instructions, which is beneficial for the accurate generation of grasping gestures.

[0074] Step S3: Construct a region segmentation model and train the region segmentation model based on the sample dataset.

[0075] In a specific embodiment of the present invention, such as Figure 2 As shown, the region segmentation model includes an image encoder, an image adapter, a text encoder, a text adapter, a transformer module, and a pixel decoder. The image encoder is connected to the image adapter, the text encoder is connected to the text adapter, the image adapter and the text adapter are connected to the transformer module, and the transformer module is connected to the pixel decoder. The function of the region segmentation model is to segment corresponding regions in an object image based on a clearly defined text instruction and output a segmented region image.

[0076] During region segmentation model training, the image encoder and text encoder are frozen, and only the image adapter, text adapter, transformer module, and pixel-level decoder are trained. In this embodiment, the image encoder and text encoder use a contrastive language-image pre-trained model, i.e., the CLIP pre-trained model. By training the region segmentation model, the ability to perform region segmentation in images based on text instructions is improved, which is beneficial for the accurate generation of grasping poses.

[0077] Step S4: Obtain the image to be processed and the text instructions to be processed that imply the intent.

[0078] In this embodiment, the image to be processed includes an RGB image and a depth image. The RGB image and the depth image are acquired simultaneously by the same camera, so the pixels in the RGB image and the depth in the depth image are in one-to-one correspondence.

[0079] Step S5: Use the trained intent reasoning model to perform intent reasoning on the RGB image to be processed and the text instruction to be processed with implicit intent to obtain the text instruction to be processed with explicit intent.

[0080] like Figure 2 As shown, for example, the image to be processed is an RGB image of a teapot, and the text instruction to be processed with the implicit intent is "Please pour the tea into the teacup". Then, the text instruction to be processed with the explicit intent output by the trained intent reasoning model is: "Grab the handle of the teapot".

[0081] Step S6: Based on the explicit text instruction to be processed, the trained region segmentation model is used to segment the RGB image to be processed, resulting in a segmented image of the region to be processed.

[0082] like Figure 2 As shown, for example, based on the text instruction "grab the handle", the trained region segmentation model is used to segment the RGB image of the teapot and output the handle image.

[0083] Step S7: Based on the segmented image of the region to be processed, perform region-level labeling in the RGB image and depth image to be processed, and generate region-level grasping pose.

[0084] Existing technologies cannot generate grasping poses on fine-grained target regions. To address this issue, this invention transforms the 2D grasping region into 3D space based on the relationship between RGB and depth images, thereby generating more accurate grasping poses. In a specific embodiment of this invention, the image is segmented according to the region to be processed, and region-level labeling is performed in both the RGB and depth images, specifically including:

[0085] Step S7.1: Based on the segmentation of the region to be processed, perform region-level marking in the RGB image to obtain the 2D grasping region;

[0086] Step S7.2: Based on the correspondence between pixels in the RGB image and depth in the depth image, filter out depth values ​​outside the grasping area in the depth image based on the 2D grasping area to obtain the 3D grasping area.

[0087] Based on the 3D grasping region, the GraspNet model is used as the grasping posture prediction network to predict the grasping posture in 3D space, avoiding interference from other regions and improving the accuracy of the grasping task.

[0088] Example 2

[0089] Figure 3 A structural block diagram of the robot grasping posture prediction device provided by the present invention is shown. Figure 3 As shown, the robot grasping posture prediction device includes a dataset construction unit, an intent reasoning model construction and training unit, a region segmentation model construction and training unit, a first acquisition unit, a second acquisition unit, and a prediction unit.

[0090] The dataset construction unit is used to construct a sample dataset; the samples in the sample dataset include object images and region name text in the object images, region segmentation images, text instructions with implicit intent, and text instructions with explicit intent.

[0091] The intent reasoning model construction and training unit is used to build an intent reasoning model. It takes object images and text instructions with implicit intents from the sample dataset as inputs and text instructions with explicit intents as the expected outputs to train the intent reasoning model.

[0092] The region segmentation model construction and training unit is used to build a region segmentation model. It takes object images and explicit text instructions from the sample dataset as inputs and the corresponding region segmentation image as the expected output to train the region segmentation model.

[0093] The first acquisition unit is used to acquire the image to be processed and the text instruction to be processed with implicit intent. It uses the trained intent reasoning model to perform intent reasoning on the image to be processed and the text instruction to be processed with implicit intent to obtain the text instruction to be processed with explicit intent.

[0094] When the image to be processed includes an RGB image and a depth image, the first acquisition unit is used to perform intent reasoning on the RGB image to be processed and the text instruction to be processed with implicit intent using the trained intent reasoning model, so as to obtain the text instruction to be processed with explicit intent.

[0095] The second acquisition unit is used to segment the image to be processed using a trained region segmentation model based on the text instruction to be processed with a clear intent, so as to obtain a segmented image of the region to be processed.

[0096] When the image to be processed includes an RGB image and a depth image, the second acquisition unit is used to segment the RGB image to be processed using the trained region segmentation model to obtain the region segmentation image to be processed.

[0097] The prediction unit is used to segment the image based on the region to be processed, perform region-level labeling in the image to be processed, and generate region-level grasping pose.

[0098] When the image to be processed includes an RGB image and a depth image, the prediction unit is configured to segment the image according to the region to be processed, perform region-level labeling in the RGB image and the depth image to be processed, and generate a region-level grasping pose.

[0099] In some embodiments of the present application, the robot grasping pose prediction device can combine the technical features in the robot grasping pose prediction method in Embodiment One of the present application, and vice versa.

[0100] Embodiment Three

[0101] The present application also provides an electronic device, as shown in the figure, which includes a memory, a processor and a computer program / instruction stored in the memory, and the processor executes the computer program / instruction to implement the robot grasping pose prediction method in Embodiment One of the present application. Figure 4

[0102] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or loaded from a storage section into a random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), etc. In the RAM, various programs and data required for device operation are also stored. The processor, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0103] The above processor and memory are used together to execute programs / instructions stored in the memory, which when executed by a computer can implement the methods, steps or functions described in the above embodiments.

[0104] Although not shown, the present application also provides a computer-readable storage medium having stored thereon a computer program / instruction, which when executed by a processor implements the robot grasping pose prediction method in Embodiment One of the present application.

[0105] ​Read only memory (ROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information which can be accessed by a computing device. According to the definition provided herein, computer readable medium does not include transitory computer readable medium, such as a modulated data signal and a carrier wave.

[0106] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those skilled in the art once they learn of the basic inventive concepts. Such additional variations and modifications should be considered as within the scope of the present application as defined by the claims appended hereto. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method for predicting the posture of a robot grasping object, characterized in that, The prediction method includes: Construct a sample dataset; wherein, the samples in the sample dataset include object images and region name text in the object images, region segmentation images, text instructions with implicit intent, and text instructions with explicit intent; An intent reasoning model is constructed, taking object images and text instructions with implicit intents from the sample dataset as inputs and text instructions with explicit intents as the expected outputs, and the intent reasoning model is trained. A region segmentation model is constructed by taking object images and text commands with explicit intent from the sample dataset as inputs and the corresponding region segmentation images as the desired outputs, and training the region segmentation model. The process involves acquiring the image to be processed and the text instruction with implicit intent, then using the trained intent reasoning model to perform intent reasoning on the image to be processed and the text instruction with implicit intent to obtain the text instruction with explicit intent. Based on the explicit intent of the text instruction to be processed, the trained region segmentation model is used to segment the image to be processed, resulting in a segmented image of the region to be processed. Based on the segmented image of the region to be processed, region-level labeling is performed in the image of the region to be processed, and region-level grasping pose is generated.

2. The robot grasping posture prediction method according to claim 1, characterized in that, The construction of the sample dataset includes: Collect object images based on the category of the object grasped by the robot; Based on the object names in the object image, a generative model is used to perform crawling and reasoning on the object image to generate region name text and corresponding implicit intent text instructions. Based on the region name text, the object image is segmented using an expert model to generate a corresponding region segmentation image; Generative models are used to extract the intent from the text instructions with implicit intent, resulting in text instructions with explicit intent.

3. The robot grasping posture prediction method according to claim 1, characterized in that, The intent reasoning model is a pre-trained multimodal visual language large model, which includes a visual processing module, a language processing module, a fusion module, and an output module.

4. The robot grasping posture prediction method according to claim 3, characterized in that, The LoRA-supervised fine-tuning method is used to fine-tune a pre-trained multimodal visual language model, including: Insert the LoRA module into the language processing module of the pre-trained multimodal visual language large model; The visual processing module extracts features from the input object image to obtain an image feature vector; the language processing module extracts features from the input text command with implicit intent to obtain a text feature vector; and the LoRA module adjusts the text feature vector. The image feature vector and the adjusted text feature vector are fused using the fusion module to obtain the fused features; Based on the fusion features, the output module is used to generate predicted text instructions with clear intent; The loss value is calculated based on the predicted text instruction with explicit intent and the text instruction with explicit intent, and the parameters of the LoRA module are adjusted based on the loss value.

5. The robot grasping posture prediction method according to claim 1, characterized in that, The region segmentation model includes an image encoder, an image adapter, a text encoder, a text adapter, a transformer module, and a pixel-level decoder. The image encoder is connected to the image adapter, the text encoder is connected to the text adapter, the image adapter and the text adapter are connected to the transformer module, and the transformer module is connected to the pixel-level decoder.

6. The robot grasping posture prediction method according to claim 5, characterized in that, The image encoder and the text encoder use a contrastive language-image pre-trained model; during the training of the region segmentation model, only the image adapter, text adapter, transformer module, and pixel-level decoder are trained.

7. The robot grasping posture prediction method according to any one of claims 1 to 6, characterized in that, The image to be processed includes an RGB image and a depth image; the image is segmented according to the region to be processed, and region-level marking is performed in the image to be processed, specifically including: Based on the segmentation of the region to be processed, region-level marking is performed in the RGB image to obtain the 2D grasping region; Based on the correspondence between pixels in the RGB image and depth in the depth image, the 3D grasping region is obtained by filtering out depth values ​​outside the grasping region in the depth image based on the 2D grasping region.

8. A robot grasping posture prediction device, characterized in that, The prediction device includes: A dataset construction unit is used to construct a sample dataset; wherein, the samples in the sample dataset include object images and region name text, region segmentation images, text instructions with implicit intent, and text instructions with explicit intent in the object images; The intent reasoning model construction and training unit is used to construct an intent reasoning model, taking object images and text instructions with implicit intents in the sample dataset as inputs and text instructions with explicit intents as the expected outputs, and to train the intent reasoning model. The region segmentation model construction and training unit is used to construct a region segmentation model, taking object images and text instructions with explicit intent from the sample dataset as inputs and the corresponding region segmentation image as the expected output, and training the region segmentation model. The first acquisition unit is used to acquire the image to be processed and the text instruction to be processed with implicit intent. It uses the trained intent reasoning model to perform intent reasoning on the image to be processed and the text instruction to be processed with implicit intent to obtain the text instruction to be processed with explicit intent. The second acquisition unit is used to segment the image to be processed using a trained region segmentation model based on the text instruction to be processed with a clear intention, so as to obtain the region segmentation image to be processed. The prediction unit is used to segment the image based on the region to be processed, perform region-level labeling in the image to be processed, and generate region-level grasping pose.

9. An electronic device comprising a memory, a processor, and a computer program / instructions stored in the memory, characterized in that, The processor executes the computer program / instructions to implement the robot grasping posture prediction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the robot grasping posture prediction method as described in any one of claims 1 to 7.

Images