Language-driven object grabbing attitude prediction method, terminal and storage medium

Abstract

The invention discloses a language-driven object grabbing attitude prediction method, a terminal and a storage medium, and relates to the technical field of artificial intelligence and computer vision. The language-driven object grabbing posture prediction model provided by the invention is a model in which the language interaction capability is introduced, and interactive prediction can be performed in combination with the language cue word input by the user, so that an operator can specify a grabbing object through the language cue word, and a more accurate grabbing posture can be predicted through the model. According to the method, the interactivity and the model flexibility of an object grabbing attitude prediction model are expanded, and the method has relatively high generalization for an unstructured task scene.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a language-driven object grasping posture prediction method, terminal and storage medium. Background Art

[0002] In the manufacturing industry, computer vision focuses on creating artificial systems that can capture and analyze visual input from the physical world to assist humans in completing various production-related tasks and provide operators or control agencies with location information related to navigation, grasping, and other interactions with the environment.

[0003] For example, in an assembly line sorting scenario, an industrial robot needs the guidance of visual information to perceive objects in the scene and predict the grasping posture. The four-degree-of-freedom grasping posture is a planar grasping posture representation, including displacement along the three coordinate axes and rotation along the z-axis. This type of method represents the grasping posture of an object as A quaternary group, where is the two-dimensional grasping center; is the grasping angle, which is the angle at which the end of the robot arm rotates around the z-axis; is the grasping width, which indicates the degree of opening of the robot arm's gripper. In this posture representation, the robot arm posture is always perpendicular to the grasping plane.

[0004] However, the traditional grasping posture prediction algorithm based on a 4-DOF robot arm still has certain limitations when facing unstructured sorting scenarios: there are usually multiple targets in the sorting scenario, and the traditional algorithm can only be used for a few predefined object categories, and the prediction effect for newly emerging object categories or unseen object instances is poor. In addition, the traditional algorithm is not interactive during reasoning and does not support flexible specification of sorting target categories.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a language-driven object grasping posture prediction method, terminal and storage medium are provided, aiming to solve the problems of poor generalization and low interactivity of the existing grasping posture prediction algorithms.

[0007] The technical solution adopted by the present invention to solve the problem is as follows: In a first aspect, an embodiment of the present invention provides a language-driven object grasping posture prediction method, the method comprising: Obtain scene images and language prompts; Inputting the scene image and the language prompt word into a language-driven object grasping posture prediction model to obtain a grasping posture; Wherein, the language-driven object grasping posture prediction model includes: An image encoder, configured to output an image embedding based on the scene image; An object detector and segmenter, used for outputting a bounding box prompt and a point prompt of a target to be captured according to the scene image and the language prompt word; A cue encoder, configured to output a cue embedding according to a bounding box cue and a point cue of the object to be grasped; a mask decoder for outputting a predicted grasp position mask, a grasp width mask, and a grasp angle mask based on the image embedding and the hint embedding; The grasping posture extraction module is used to output the grasping posture according to the grasping position mask, the grasping width mask and the grasping angle mask.

[0008] In one embodiment, the image encoder includes several attention encoder modules and a convolutional layer with a last channel number of 256; the attention encoder module includes several normalization layers, multi-head attention layers and a multi-layer perceptron.

[0009] In one embodiment, the hint encoder includes a point hint encoder and a bounding box hint encoder; the point hint encoder and the bounding box hint encoder include a plurality of convolutional layers, normalization layers, and activation function layers; outputting hint embedding according to the bounding box hint and the point hint of the target to be grasped includes: Inputting the bounding box prompt of the target to be captured into the bounding box prompt encoder to obtain a first prompt embedding; Inputting the point cue into the point cue encoder to obtain a second cue embedding; The concatenated data of the first hint embedding and the second hint embedding is used as the hint embedding.

[0010] In one embodiment, the mask decoder includes a composite attention module, an upsampling convolution module, and a composite perceptron module; outputting a predicted grasp position mask, a grasp width mask, and a grasp angle mask according to the image embedding and the hint embedding, including: Inputting the image embedding and the corresponding position embedding into the composite attention module to obtain segmentation features and mask features; Inputting the segmentation features into the upsampling convolution module to obtain low-level position mask features, low-level angle mask features and low-level width mask features; Input the mask feature into the composite perceptron module to obtain a position token, an angle token and a width token; multiplying the low-level position mask feature with the position token to obtain a grasp position mask; multiplying the low-level angle mask feature with the angle token to obtain a grasped angle mask; The low-level width mask feature is multiplied by the width token to obtain a grabbed width mask.

[0011] In one implementation, outputting a grasping posture according to the grasping position mask, the grasping width mask, and the grasping angle mask includes: Determine a grasping position peak value according to the predicted grasping position mask, and use the grasping position peak value as a grasping center; Determine an angle index with a maximum confidence at each pixel position of the predicted grasping angle mask, and construct a single-channel grasping angle mask according to the angle index at each pixel position; Acquire a grasping angle index corresponding to the grasping center according to the single-channel grasping angle mask, and convert the grasping angle index into a corresponding grasping angle; Acquire a grasping width corresponding to the grasping center according to the predicted grasping width mask; The grasping posture is determined according to the position of the grasping center, the grasping angle, and the grasping width.

[0012] In one embodiment, the language-driven object grasping posture prediction model is pre-trained, and a method for generating training samples includes: Acquire a plurality of image data of a preset resolution, wherein each of the image data contains objects to be grasped within a preset number range, and the number and position of the objects to be grasped are randomly determined; For each of the image data, a tuple is generated according to the position and grasping posture of each object to be grasped in the image data, a grasping label in mask form is generated according to the tuple set corresponding to the image data, and a training sample is generated according to the image data and the grasping label.

[0013] In one implementation, generating a capture label in the form of a mask according to a set of tuples corresponding to the image data includes: For each grasping posture in the set of tuples, determining a neighborhood of the grasping posture; Divide the full angle range into several intervals, and for each grasping angle in the tuple set, calculate the angle mask index of the grasping angle according to the grasping angle and the angle range covered by each interval; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture, and generate a grasping position mask label according to the determination result of each pixel coordinate; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture and has a corresponding angle mask index, and generate a grasping angle mask label according to the determination result of each pixel coordinate; For each pixel coordinate, find all width sets covering the pixel coordinate, determine whether the width set is an empty set, and generate a captured width mask label according to the determination result of each pixel coordinate; The grasping label is generated according to the grasping position mask label, the grasping angle mask label, and the grasping width mask label.

[0014] In one embodiment, the training method of the language-driven object grasping posture prediction model includes: Initializing network weights of the image encoder, the object detector and segmenter, and the cue encoder according to pre-trained weights of a visual macro model and a visual language model; Inputting the image data in the training sample into the initial language-driven object grasping posture prediction model to obtain the grasping position mask, grasping angle mask and grasping width mask predicted during training; According to the grasping position mask, grasping angle mask and grasping width mask predicted during training and the grasping labels in the training samples, the model error is calculated through a binary cross entropy loss function, and the parameters of the image encoder and the mask decoder are updated according to the model error.

[0015] In a second aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-mentioned language-driven object grasping posture prediction methods; and the processor is used to execute the program.

[0016] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-mentioned language-driven object grasping posture prediction methods.

[0017] Beneficial effects of the present invention: The language-driven object grasping posture prediction model provided by the embodiment of the present invention is a model that introduces language interaction capabilities, and can perform interactive predictions in combination with language prompts input by the user, so that the operator can specify the grasping object through the language prompts, and the model predicts a more accurate grasping posture. The present invention expands the interactivity and model flexibility of the object grasping posture prediction model, and has strong generalization for unstructured task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a language-driven object grasping posture prediction method provided in an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the algorithm framework of the language-driven object grasping posture prediction model provided by an embodiment of the present invention.

[0021] Figure 3 It is a schematic diagram of the structure of an image encoder provided by an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of a prompt encoder provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of a mask decoder provided by an embodiment of the present invention.

[0024] Figure 6 FIG. 4 is a functional block diagram of a terminal according to an implementation example of the present invention. DETAILED DESCRIPTION

[0025] The present invention discloses a language-driven object grasping posture prediction method, terminal and storage medium. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0028] In view of the above-mentioned defects of the prior art, the present invention provides a language-driven object grasping posture prediction method, the method acquires a scene image and a language prompt word; inputs the scene image and the language prompt word into a language-driven object grasping posture prediction model to obtain a grasping posture; wherein the language-driven object grasping posture prediction model includes: an image encoder, used to output an image embedding according to the scene image; a target detector and segmenter, used to output a bounding box prompt and a point prompt of a target to be grasped according to the scene image and the language prompt word; a prompt encoder, used to output a prompt embedding according to the bounding box prompt and the point prompt of the target to be grasped; a mask decoder, used to output a predicted grasping position mask, a grasping width mask and a grasping angle mask according to the image embedding and the prompt embedding; a grasping posture extraction module, used to output a grasping posture according to the grasping position mask, the grasping width mask and the grasping angle mask. The language-driven object grasping posture prediction model provided by the present invention is a model that introduces language interaction capabilities and can perform interactive predictions in combination with language prompts input by the user, so that the operator can specify the grasping object through the language prompts, and the model predicts a more accurate grasping posture. The present invention expands the interactivity and model flexibility of the object grasping posture prediction model and has strong generalization for unstructured task scenarios.

[0029] like Figure 1 As shown, the method includes: Step S100, obtaining scene images and language prompt words; Step S200, inputting the scene image and the language prompt word into a language-driven object grasping posture prediction model to obtain a grasping posture; Wherein, the language-driven object grasping posture prediction model includes: An image encoder, configured to output an image embedding based on the scene image; An object detector and segmenter, used for outputting a bounding box prompt and a point prompt of a target to be captured according to the scene image and the language prompt word; A cue encoder, configured to output a cue embedding according to a bounding box cue and a point cue of the object to be grasped; a mask decoder for outputting a predicted grasp position mask, a grasp width mask, and a grasp angle mask based on the image embedding and the hint embedding; The grasping posture extraction module is used to output the grasping posture according to the grasping position mask, the grasping width mask and the grasping angle mask.

[0030] Specifically, this embodiment pre-constructs a language-driven object grasping posture prediction model, which takes scene images and language prompt words as input data, and predicts the grasping posture of objects matching the language prompt words. The network structure of the model mainly includes an image encoder, a prompt encoder, a mask decoder, and a target detector and segmenter, wherein the target detector and segmenter can use a plug-and-play Grounding SAM target detector (Grounding SAM is a model that can identify and segment objects in a picture). In practical applications, first obtain a scene image, which can be obtained by an RGB video stream captured by an industrial camera. Then give a language prompt word, which can be a word or phrase input by the user based on the scene image, and the language prompt word can be a description of the appearance and function of the target object in the scene, which is used to specify the grasping object. The scene image acquired in real time is input into the image encoder to obtain an image embedding. The scene image acquired in real time and the language prompt word are input into the target detector and segmenter to obtain a two-dimensional bounding box of the target to be grasped. The two-dimensional bounding box of the target to be grasped is input into the prompt encoder to obtain a prompt embedding. The image embedding and the hint embedding are input into the mask decoder to obtain the predicted grasp position mask, grasp width mask and grasp angle mask. The grasp pose extraction algorithm is used to extract the grasp pose from the predicted grasp position mask, grasp width mask and grasp angle mask.

[0031] For example, Figure 2 As shown in Figure 1, the image encoder processes the scene image into an image embedding. The Grounding SAM object detector processes the scene image and the user's language prompt words into bounding box prompts and point prompts. The prompt encoder processes the bounding box prompts and point prompts into prompt embeddings. The mask decoder processes the image embedding and the prompt embedding into position masks, width masks, and angle masks.

[0032] In one implementation, the image encoder includes several attention encoder modules and a convolutional layer with a last channel number of 256; the attention encoder module includes several normalization layers, multi-head attention layers and a multi-layer perceptron.

[0033] Specifically, the image encoder consists of multiple attention encoder modules (Transformer encoder modules) and a convolutional layer with a last channel number of 256. The number of attention encoder modules can be selected as 12. The attention encoder module includes several normalization layers, multi-head attention layers, and multi-layer perceptrons. For example, it can be composed of a stack of normalization layers, multi-head attention layers, normalization layers, and multi-layer perceptrons. The data image encoder takes training sample images or scene images acquired in real time as input and outputs image embedding.

[0034] For example, Figure 3 is the image encoder structure. It contains an embedding layer, a random inactivation layer (i.e., Dropout layer), several stacked encoder modules, and several modules consisting of normalization layers and convolution layers. The number of encoder modules can be selected as 12, and the number of modules consisting of normalization layers and convolution layers can be selected as 2. Specifically, the embedding layer processes the image into a feature embedding of 32x32x768, which is then added to the position embedding. The position embedding is a parameter vector of the same size that is initialized to 0 and is updated during the training process. The encoder module is a standard encoder module in the attention network. The structure of each encoder module is a layer normalization operation, a multi-head attention module, a layer normalization operation, and a multi-layer perceptron. The image encoder also contains 2 normalization layers and a module consisting of a convolution with a channel number of 256.

[0035] In one implementation, the hint encoder includes a point hint encoder and a bounding box hint encoder; the point hint encoder and the bounding box hint encoder include a plurality of convolutional layers, a normalization layer, and an activation function layer; outputting hint embedding according to the bounding box hint and the point hint of the target to be grasped includes: Inputting the bounding box prompt of the target to be captured into the bounding box prompt encoder to obtain a first prompt embedding; Inputting the point cue into the point cue encoder to obtain a second cue embedding; The concatenated data of the first hint embedding and the second hint embedding is used as the hint embedding.

[0036] Specifically, the prompt encoder includes a two-dimensional point prompt encoder and a two-dimensional bounding box prompt encoder. The two-dimensional point prompt encoder and the two-dimensional bounding box prompt encoder are stacked by a number of convolutional layers, a number of normalization layers, and a number of activation function layers, wherein the number of convolutional layers is preferably 3, the number of normalization layers is preferably 2, and the number of activation function layers is preferably 2. The input data of the prompt encoder are the bounding box prompts and point prompts of the target to be captured output by the target detector and segmenter, and the output data of the prompt encoder is the prompt embedding, which is spliced ​​by the output of the two-dimensional point prompt encoder and the two-dimensional bounding box encoder.

[0037] For example, Figure 4 is the prompt encoder structure. The point prompt is represented as , and is processed by the point embedding layer into a 1x1x256 embedding vector. The two-dimensional bounding box is represented as , and processed by the bounding box embedding layer into an embedding vector of 1x2x256. Finally, after vector concatenation, the resulting hint embedding size is 1x3x256.

[0038] In one implementation, the mask decoder includes a composite attention module, an upsampling convolution module, and a composite perceptron module; outputting a predicted grasp position mask, a grasp width mask, and a grasp angle mask according to the image embedding and the hint embedding, including: Inputting the image embedding and the corresponding position embedding into the composite attention module to obtain segmentation features and mask features; Inputting the segmentation features into the upsampling convolution module to obtain low-level position mask features, low-level angle mask features and low-level width mask features; Input the mask feature into the composite perceptron module to obtain a position token, an angle token and a width token; multiplying the low-level position mask feature with the position token to obtain a grasp position mask; multiplying the low-level angle mask feature with the angle token to obtain a grasped angle mask; The low-level width mask feature is multiplied by the width token to obtain a grabbed width mask.

[0039] Specifically, the mask decoder consists of a composite attention module, an upsampling convolution module, and a composite perceptron module. The composite attention module takes the image embedding and the corresponding position embedding as input data, and the output data is the segmentation feature and the mask feature. The position embedding is a vector with the same size as the feature embedding generated in real time. The upsampling convolution module takes the segmentation feature as input data, and the output data is the low-level position mask feature, the angle position mask feature, and the low-level width mask feature. The composite perceptron module takes the mask feature as input data, and the output data is the position token, the angle token, and the width token. Finally, the low-level position mask feature is multiplied by the position token to obtain the grasped position mask; the low-level angle mask is multiplied by the angle token to obtain the grasped angle mask; the low-level width mask is multiplied by the width token to obtain the grasped width mask.

[0040] Furthermore, the composite attention module is composed of a self-attention layer, a cross-attention layer, a multi-layer perceptron, and two parallel cross-attention layers located at the output head position of the composite attention module, which output segmentation features and mask features respectively.

[0041] For example, Figure 5is the mask decoder structure. The image embedding comes from the output data of the image encoder, and the position embedding is a learnable parameter of the same size. The prompt embedding comes from the output data of the prompt encoder. The network structure includes three cross-attention structures. In the first cross-attention module after calculating the self-attention, the prompt embedding is used as the Q matrix, and the image embedding is used as the K and V matrices to calculate the attention. In the cross-attention structure with the upsampling convolution branch on the output side, the image embedding is used as the Q matrix, and the prompt embedding is used as the K and V matrices to calculate the attention. In the cross-attention structure with the multi-layer perceptron branch on the output side, the prompt embedding is used as the Q matrix, and the image embedding is used as the K and V matrices to calculate the attention. The mask decoder output contains three groups of upsampling convolutions, each of which consists of two convolution stacks with a kernel size of 2, a stride of 2, and a channel number of 128. The mask decoder output contains three groups of multi-layer perceptrons, of which the two groups of multi-layer perceptrons corresponding to the position mask and the width mask have a channel number of 1, and the multi-layer perceptron corresponding to the angle mask has a channel number of 120. The output of each group of upsampling convolutions is dot-multiplied with the output of the corresponding group of multi-layer perceptrons to obtain the predicted grasping position mask, grasping angle mask, and grasping width mask.

[0042] In one implementation, outputting a grasping posture according to the grasping position mask, the grasping width mask, and the grasping angle mask includes: Determine a grasping position peak value according to the predicted grasping position mask, and use the grasping position peak value as a grasping center; Determine an angle index with a maximum confidence at each pixel position of the predicted grasping angle mask, and construct a single-channel grasping angle mask according to the angle index at each pixel position; Acquire a grasping angle index corresponding to the grasping center according to the single-channel grasping angle mask, and convert the grasping angle index into a corresponding grasping angle; Acquire a grasping width corresponding to the grasping center according to the predicted grasping width mask; The grasping posture is determined according to the position of the grasping center, the grasping angle, and the grasping width.

[0043] Specifically, this embodiment obtains the predicted grasping position mask through the language-driven object grasping posture prediction model , grab angle mask and grab width mask Later, find the grasp position peak from the predicted grasp position mask as the grasp center: . At each pixel position in the capture angle mask At , determine the angle index with the maximum confidence and construct a single-channel grasping angle mask: In the crawl center Get the corresponding grab angle index . Then convert the grab angle index to the corresponding grab angle . Get the crawl width The final grasping posture is: .

[0044] In one implementation, the language-driven object grasping posture prediction model is pre-trained, and a method for generating training samples includes: Acquire a plurality of image data of a preset resolution, wherein each of the image data contains objects to be grasped within a preset number range, and the number and position of the objects to be grasped are randomly determined; For each of the image data, a tuple is generated according to the position and grasping posture of each object to be grasped in the image data, a grasping label in mask form is generated according to the tuple set corresponding to the image data, and a training sample is generated according to the image data and the grasping label.

[0045] Specifically, there are multiple training samples for model training of the language-driven object grasping posture prediction model. Each training sample is generated in the same way. Therefore, this embodiment takes a training sample as an example to illustrate the generation process: first, multiple image data of a specified resolution are obtained. Each image data needs to contain no more than a specified number of objects to be grasped, and the number and position are random. The processing method of each image data is the same. Taking one image data as an example, a tuple is constructed by the object and grasping posture therein, and a grasping label of the image data is generated by all the constructed tuples. The grasping label includes a grasping position mask label, a grasping angle label, and a grasping width label. Finally, the image data and the corresponding grasping label are combined into a training sample.

[0046] For example, 10k RGB images with a resolution of 500x500 containing grasped targets are obtained for model training. Each image contains no more than 10 objects to be grasped with random numbers, positions and postures. The object's The grasping posture is represented as a tuple ,in represents the pixel position of the grasp center, Indicates the grab angle, represents the grasp width. The grasp posture contained in each image can be represented as a set ,in Indicates the number of objects, Indicates The number of grasping postures of an object. The grasping labels in the training samples are represented in the form of masks. Finally, each image-label pair is used as a training sample. In practical applications, 10k training images and corresponding grasping labels can be obtained from the Jacquard public grasping dataset.

[0047] In one implementation, generating a capture label in a mask form according to a set of tuples corresponding to the image data includes: For each grasping posture in the set of tuples, determining a neighborhood of the grasping posture; Divide the full angle range into several intervals, and for each grasping angle in the tuple set, calculate the angle mask index of the grasping angle according to the grasping angle and the angle range covered by each interval; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture, and generate a grasping position mask label according to the determination result of each pixel coordinate; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture and has a corresponding angle mask index, and generate a grasping angle mask label according to the determination result of each pixel coordinate; For each pixel coordinate, find all width sets covering the pixel coordinate, determine whether the width set is an empty set, and generate a captured width mask label according to the determination result of each pixel coordinate; The grasping label is generated according to the grasping position mask label, the grasping angle mask label, and the grasping width mask label.

[0048] Specifically, the grasping label includes three types of data, and these three types of data need to be generated separately to obtain the final grasping label. First, the neighborhood of each grasping posture in the tuple set is calculated to describe a subset containing the grasping posture. Then the full angle range (i.e., 360°) is divided into multiple intervals according to the preset angle interval, and the angle range covered by each interval is the same. For each grasping angle in the tuple set, an angle mask index is assigned to the grasping angle through the grasping angle and the preset angle interval (i.e., the angle range covered by each interval). Then, the data of each pixel in different mask labels is determined pixel by pixel, and different mask labels use different judgment conditions to determine the value of the pixel in the mask label. Taking a pixel as an example, according to the coordinates of the pixel, it is judged whether it is located in the neighborhood of the grasping posture. Yes and no correspond to different values. After the pixel-by-pixel judgment is completed, the grasping position mask label can be generated according to the values ​​of all pixels. According to the coordinates of the pixel point, determine whether it is in the neighborhood of the grasping posture and has a corresponding angle mask index. Yes and no correspond to different values. After the pixel-by-pixel judgment is completed, the grasping angle mask label can be generated according to the values ​​of all pixels. According to the coordinates of the pixel point, find all the width sets that cover the coordinates of the pixel point, and determine whether the width set is an empty set. Yes and no correspond to different values. After the pixel-by-pixel judgment is completed, the grasping width mask label can be generated according to the values ​​of all pixels.

[0049] For example, according to the grasping posture The method for generating the corresponding grasping position mask, grasping angle mask and grasping width mask labels is as follows: for each grasping posture , define its neighborhood: ; The angle range from 0° to 360° is divided into 120 intervals, each covering 3°. , calculate its angle mask index: ; in, , represents the arc range covered by each interval, .

[0050] Generate grab location labels , for all pixel coordinates ,make: .

[0051] Generate grab angle label , for each angle mask and all pixel coordinates ,make: .

[0052] Generate grab width tags , for all pixel coordinates , find the set of all widths that cover that pixel: .

[0053] The grab width tag is calculated as: .

[0054] In one implementation, the training method of the language-driven object grasping posture prediction model includes: Initializing network weights of the image encoder, the object detector and segmenter, and the cue encoder according to pre-trained weights of a visual macro model and a visual language model; Inputting the image data in the training sample into the initial language-driven object grasping posture prediction model to obtain the grasping position mask, grasping angle mask and grasping width mask predicted during training; According to the grasping position mask, grasping angle mask and grasping width mask predicted during training and the grasping labels in the training samples, the model error is calculated through a binary cross entropy loss function, and the parameters of the image encoder and the mask decoder are updated according to the model error.

[0055] Specifically, the network weights of the language-driven object grasping posture prediction model are initialized by the pre-trained weights of the visual large model and the visual language model, where the weights of the target detector and segmenter and the prompt encoder are frozen. The purpose of introducing the visual large model during model training in this embodiment is to utilize its pre-trained weights on a large-scale image dataset to enhance the generalization performance of the model. This embodiment fine-tunes the network structure and network parameters based on the weights of large models such as the visual large model and the visual language model, and can effectively utilize the extensive visual knowledge of the visual large model. After initialization, the language-driven object grasping posture prediction model is iteratively updated through multiple training samples to optimize the model parameters. This embodiment defines the model when model training is not completed as an initial language-driven object grasping posture prediction model. Taking a training sample as an example, the image data in the training sample is input into the initial language-driven object grasping posture prediction model to obtain the grasping position mask, grasping angle mask and grasping width mask predicted by the initial language-driven object grasping posture prediction model. The three masks predicted by the model are compared with the three mask labels in the grasping label according to the category correspondence. The model error is calculated through the binary cross entropy loss function. The model error can reflect the gap between the model prediction and the true value. The image encoder and the mask decoder are updated with the parameters guided by the model error, which can effectively improve the model performance.

[0056] For example, the pre-trained weights of the visual large model SAM and the visual language model Grounding SAM are used to initialize the network weights of the image encoder, prompt encoder, and object detection and segmentation. The images in the training samples are input to the image encoder to obtain the image embedding. The image data and language prompt words in the training samples are input to the object detection and segmentation to obtain the two-dimensional bounding box and semantic mask of the target to be grasped, and a prompt point is randomly sampled from the semantic mask. The two-dimensional bounding box of the target to be grasped is input to the prompt encoder to obtain the prompt embedding. The image embedding and the prompt embedding are input to the mask decoder to obtain the predicted grasp position mask, grasp angle mask, and grasp width mask. Finally, the three masks and the grasp label are used to calculate the error through the binary cross entropy loss function, and the image encoder and mask decoder network parameters are updated.

[0057] In one implementation, the binary cross entropy loss function is the sum of three parts: grasping position mask loss, grasping angle mask loss, and grasping width mask loss.

[0058] For example, the loss function of the language-driven object grasping posture prediction model is expressed as a weighted binary cross entropy loss: ; in, and Indicates the length and width of the image, and Respectively represent the mask in The true value and predicted value at represents the Sigmoid activation function, Represents the weight term.

[0059] The calculation method is as follows: ; Use weighted binary cross entropy loss to calculate the grasp position mask loss , Grasp the angle mask loss and the capture width mask loss Finally, the loss of the language-driven object grasping posture prediction model is .

[0060] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 6As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a language-driven object grasping posture prediction method is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0061] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0062] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a language-driven object grasping posture prediction method.

[0063] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0064] In summary, the present invention discloses a language-driven object grasping posture prediction method, terminal and storage medium, which relates to the field of artificial intelligence and computer vision technology. The method obtains a scene image and a language prompt word; inputs the scene image and the language prompt word into a language-driven object grasping posture prediction model to obtain a grasping posture; wherein the language-driven object grasping posture prediction model includes: an image encoder, which is used to output an image embedding according to the scene image; a target detector and segmenter, which is used to output a bounding box prompt and a point prompt of a target to be grasped according to the scene image and the language prompt word; a prompt encoder, which is used to output a prompt embedding according to the bounding box prompt and the point prompt of the target to be grasped; a mask decoder, which is used to output a predicted grasping position mask, a grasping width mask and a grasping angle mask according to the image embedding and the prompt embedding; a grasping posture extraction module, which is used to output a grasping posture according to the grasping position mask, the grasping width mask and the grasping angle mask. The language-driven object grasping posture prediction model provided by the present invention is a model that introduces language interaction capabilities and can perform interactive predictions in combination with language prompts input by the user, so that the operator can specify the grasping object through the language prompts, and the model predicts a more accurate grasping posture. The present invention expands the interactivity and model flexibility of the object grasping posture prediction model and has strong generalization for unstructured task scenarios.

[0065] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A language-driven object grasping posture prediction method, characterized in that: The method comprises: Obtain scene images and language prompts; Inputting the scene image and the language prompt word into a language-driven object grasping posture prediction model to obtain a grasping posture; Wherein, the language-driven object grasping posture prediction model includes: An image encoder, configured to output an image embedding based on the scene image; An object detector and segmenter, used for outputting a bounding box prompt and a point prompt of a target to be captured according to the scene image and the language prompt word; A cue encoder, configured to output a cue embedding according to a bounding box cue and a point cue of the object to be grasped; a mask decoder for outputting a predicted grasp position mask, a grasp width mask, and a grasp angle mask based on the image embedding and the hint embedding; The grasping posture extraction module is used to output the grasping posture according to the grasping position mask, the grasping width mask and the grasping angle mask.

2. The language-driven object grasping posture prediction method according to claim 1, characterized in that: The image encoder includes several attention encoder modules and a convolutional layer with a last channel number of 256; the attention encoder module includes several normalization layers, multi-head attention layers and a multi-layer perceptron.

3. The language-driven object grasping posture prediction method according to claim 1, characterized in that: The hint encoder includes a point hint encoder and a bounding box hint encoder; the point hint encoder and the bounding box hint encoder include a plurality of convolutional layers, normalization layers and activation function layers; Outputting hint embedding according to the bounding box hint and point hint of the target to be grasped includes: Inputting the bounding box prompt of the target to be captured into the bounding box prompt encoder to obtain a first prompt embedding; Inputting the point cue into the point cue encoder to obtain a second cue embedding; The concatenated data of the first hint embedding and the second hint embedding is used as the hint embedding.

4. The language-driven object grasping posture prediction method according to claim 1, characterized in that: The mask decoder includes a composite attention module, an upsampling convolution module, and a composite perceptron module; outputs a predicted grasp position mask, a grasp width mask, and a grasp angle mask according to the image embedding and the prompt embedding, including: Inputting the image embedding and the corresponding position embedding into the composite attention module to obtain segmentation features and mask features; Inputting the segmentation features into the upsampling convolution module to obtain low-level position mask features, low-level angle mask features and low-level width mask features; Input the mask feature into the composite perceptron module to obtain a position token, an angle token and a width token; multiplying the low-level position mask feature with the position token to obtain a grasp position mask; multiplying the low-level angle mask feature with the angle token to obtain a grasped angle mask; The low-level width mask feature is multiplied by the width token to obtain a grabbed width mask.

5. The language-driven object grasping posture prediction method according to claim 1, characterized in that: Outputting a grasping posture according to the grasping position mask, the grasping width mask, and the grasping angle mask includes: Determine a grasping position peak value according to the predicted grasping position mask, and use the grasping position peak value as a grasping center; Determine an angle index with a maximum confidence at each pixel position of the predicted grasping angle mask, and construct a single-channel grasping angle mask according to the angle index at each pixel position; Acquire a grasping angle index corresponding to the grasping center according to the single-channel grasping angle mask, and convert the grasping angle index into a corresponding grasping angle; Acquire a grasping width corresponding to the grasping center according to the predicted grasping width mask; The grasping posture is determined according to the position of the grasping center, the grasping angle, and the grasping width.

6. The language-driven object grasping posture prediction method according to claim 1, characterized in that: The language-driven object grasping posture prediction model is pre-trained, and the method for generating training samples includes: Acquire a plurality of image data of a preset resolution, wherein each of the image data contains objects to be grasped within a preset number range, and the number and position of the objects to be grasped are randomly determined; For each of the image data, a tuple is generated according to the position and grasping posture of each object to be grasped in the image data, a grasping label in mask form is generated according to the tuple set corresponding to the image data, and a training sample is generated according to the image data and the grasping label.

7. The language-driven object grasping posture prediction method according to claim 6, characterized in that: Generating a capture label in the form of a mask according to a set of tuples corresponding to the image data, including: For each grasping posture in the set of tuples, determining a neighborhood of the grasping posture; Divide the full angle range into several intervals, and for each grasping angle in the tuple set, calculate the angle mask index of the grasping angle according to the grasping angle and the angle range covered by each interval; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture, and generate a grasping position mask label according to the determination result of each pixel coordinate; For each pixel coordinate, determine whether the pixel coordinate is located in the neighborhood of the grasping posture and has a corresponding angle mask index, and generate a grasping angle mask label according to the determination result of each pixel coordinate; For each pixel coordinate, find all width sets covering the pixel coordinate, determine whether the width set is an empty set, and generate a captured width mask label according to the determination result of each pixel coordinate; The grasping label is generated according to the grasping position mask label, the grasping angle mask label, and the grasping width mask label.

8. The language-driven object grasping posture prediction method according to claim 6, characterized in that: The training method of the language-driven object grasping posture prediction model includes: Initializing network weights of the image encoder, the object detector and segmenter, and the cue encoder according to pre-trained weights of a visual macro model and a visual language model; Inputting the image data in the training sample into the initial language-driven object grasping posture prediction model to obtain the grasping position mask, grasping angle mask and grasping width mask predicted during training; According to the grasping position mask, grasping angle mask and grasping width mask predicted during training and the grasping labels in the training samples, the model error is calculated through a binary cross entropy loss function, and the parameters of the image encoder and the mask decoder are updated according to the model error.

9. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the language-driven object grasping posture prediction method as described in any one of claims 1-8; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the language-driven object grasping posture prediction method as described in any one of claims 1-8.

Images