Attribute perception based robot referent segmentation and grasp detection method and system

Abstract

The application discloses a robot reference segmentation and grasping detection method and system based on attribute perception, which comprises the following steps: extracting multi-scale visual features from a scene image and extracting word-level features from a text description of a reference target; adopting a cross-modal point product attention operation on the multi-scale visual features and the word-level features after average pooling to obtain interactive features, and connecting the interactive features with the multi-scale visual features in a top-down manner to obtain multi-scale visual attribute features of the reference target; calculating a segmentation mask according to the last layer of visual attribute features; extracting grasping features of the reference target according to the last layer of visual features and the multi-scale visual attribute features under the constraint of the segmentation mask, and predicting a grasping position, a width and a grasping angle based on the grasping features to obtain an optimal grasping posture. Not only the interaction and alignment of multi-modal features are considered, but also the extraction of visual attributes of the reference target is considered to promote the refinement of the grasping posture, so that the precision of the reference segmentation and grasping detection is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of referring segmentation and grasping detection, and particularly relates to a robot referring segmentation and grasping detection method and system based on attribute perception. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the rapid development of robot technology, the language-condition interaction paradigm can guide robots out of the cage and cooperate with humans more effectively. The purpose of visual language segmentation and grasping detection is to enable robots to perceive and divide specific objects by understanding free-form natural language, so as to robustly complete the grasping task. Due to the complexity and diversity of natural language expression and object attributes in human-computer interaction, this task is more challenging than the visual-based grasping detection and closed-set segmentation in practical applications. In addition, cluttered scenes and occluded objects further increase the difficulty of description and target matching.

[0004] Vision-based grasping is a basic task of robot technology, aiming to detect high-quality grasping poses and perform operations. The grasping rectangle is usually taken as the optimization target, similar to object detection in computer vision. Most excellent algorithms are designed to regress the center, width and angle of the grasping rectangle, such as ROI-based methods, anchor-based methods and generative methods. The ROI and anchor-based methods can obtain better grasping accuracy, but are time-consuming. The generative method produces pixel-by-pixel prediction of grasping poses, rather than sampling and classifying grasping candidates, which is faster and lighter. Therefore, some advanced techniques are introduced to enhance the generative network, such as residual structure, attention mechanism and Transformer structure. However, the flexibility, practicality and interactivity of these methods in application scenarios still need to be further explored.

[0005] Compared with traditional vision-based grasping detection technology, referring segmentation and grasping detection technology involves two major fields of natural language processing and computer vision, with difficulties such as large variance of language description, small image target and unbalanced modal semantics, which seriously affect the performance of language understanding, target detection and modal interaction.

[0006] In recent studies, the target is retrieved by the similarity between the candidate region and the language embedding, but the results are greatly affected by the language and object differences. In order to eliminate the ambiguity of similar objects, the attribute words in the language are captured in the interactive grasping system, and the question based on the attribute is constructed, but the importance of the visual attribute to the grasping posture is ignored, resulting in large changes in the width and angle of the generated grasping matrix. A neural symbolic system is also defined to decompose the command into subtasks for step-by-step reasoning, but it relies heavily on the detection ability of the ROI-based method and the handcrafted rule base. For some related algorithms based on large pre-trained models, the ability to understand language and visual concepts is lacking, making it difficult to align text features and visual features, making it difficult to capture relevant visual information, and ignoring the importance of object visual attributes to grasping detection, resulting in large variance in grasping posture and insufficient parameter regression.

[0007] In summary, most of the existing reference segmentation methods only consider the effectiveness of grasping, ignore the fine-grained interaction of visual features and language features, and the relevance of visual attributes and grasping postures, resulting in high target misidentification rate, large grasping posture variance, and insufficient parameter regression. SUMMARY

[0008] To solve the above problems, the present application provides a robot reference segmentation and grasping detection method and system based on attribute perception, which not only considers the interaction and alignment of multi-modal features, but also extracts the visual attributes of the reference target to facilitate the refinement of the grasping posture, effectively improving the accuracy of reference segmentation and grasping detection.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides a robot reference segmentation and grasping detection method based on attribute perception, comprising:

[0011] Obtaining scene images and text descriptions of the reference target, extracting multi-scale visual features from the scene images, and extracting word-level features from the text descriptions of the reference target;

[0012] Performing average pooling operation on the multi-scale visual features, and obtaining interaction features by using cross-modal dot product attention operation on the multi-scale visual features after average pooling and the word-level features, and connecting the multi-scale visual features after average pooling with the multi-scale visual features in a top-down manner to obtain multi-scale visual attribute features of the reference target;

[0013] Calculating the segmentation mask according to the last layer of visual attribute features;

[0014] Constraining the segmentation mask, extracting the grasping features of the reference target according to the last layer of visual features and multi-scale visual attribute features, and predicting the grasping position, width and angle based on the grasping features, thereby obtaining the optimal grasping posture.

[0015] As an alternative implementation, the process of obtaining the interaction feature comprises: projecting the average-pooled multi-scale visual feature and the word-level feature into an intermediate representation, applying a scaled dot-product attention operation, and then outputting the accumulated connection and instance normalization operation with all previous inputs, and finally outputting the interaction feature.

[0016] As an alternative implementation, when extracting the interaction feature, a three-layer cross-modal dot-product attention operation is included, and the cross-modal dot-product attention operation of the first layer and the second layer includes an instance normalization operation, an average pooling operation and a cross-modal dot-product attention operation in turn; the average pooling operation is removed in the cross-modal dot-product attention operation of the third layer, and the instance normalization operation is connected after the output.

[0017] As an alternative implementation, the process of connecting with the multi-scale visual feature in a top-down manner comprises: the multi-scale visual feature includes four layers of visual features; wherein the fourth layer and the third layer are connected with the word-level feature for interaction, and then the output visual attribute feature is connected with the second layer and then interacts with the word-level feature, and finally the output visual attribute feature is connected with the first layer and then interacts with the word-level feature, thereby obtaining the last layer of visual attribute feature.

[0018] As an alternative implementation, after the last layer of visual feature and the multi-scale visual attribute feature are subjected to convolution operation, they are subjected to two structures composed of 3x3 convolution, instance normalization operation and activation function, and cross-modal dot-product attention operation, to extract the grasping feature of the referential target. The grasping feature is subjected to four 1x1 convolution to predict the grasping position, width and grasping angle respectively.

[0019] As an alternative implementation, the process of calculating the segmentation mask according to the last layer of visual attribute feature comprises: the last layer of visual attribute feature is subjected to 3x3 convolution, instance normalization operation and activation function in turn, and then subjected to 1x1 convolution and up-sampling operation to predict the segmentation mask.

[0020] As an alternative implementation, the calculated segmentation mask is filtered according to the set threshold to obtain the final predicted segmentation mask.

[0021] In a second aspect, the present application provides a robot referential segmentation and grasping detection system based on attribute perception, comprising:

[0022] A feature extraction module configured to obtain a scene image and a text description of a referential target, extract a multi-scale visual feature from the scene image, and extract a word-level feature from the text description of the referential target;

[0023] The attribute-aware multi-modal interaction module is configured to perform average pooling operation on the multi-scale visual features, perform cross-modal dot product attention operation on the average pooled multi-scale visual features and the word-level features to obtain interaction features, and connect the interaction features with the multi-scale visual features in a top-down manner to obtain multi-scale visual attribute features of the referential target.

[0024] The segmentation mask generation module is configured to generate a segmentation mask according to the last layer visual attribute features.

[0025] The grasping pose prediction module is configured to extract grasping features of the referential target according to the last layer visual features and the multi-scale visual attribute features under the constraint of the segmentation mask, and predict a grasping position, a width and a grasping angle based on the grasping features, so as to obtain an optimal grasping pose.

[0026] In a third aspect, the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method of the first aspect is completed.

[0027] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] The present application proposes a robot referential segmentation and grasping detection method and system based on attribute perception, which not only considers the interaction and alignment of multi-modal features, but also considers the extraction of visual attributes of the referential target to promote the refinement of the grasping pose, effectively improving the accuracy and performance of referential segmentation and grasping detection.

[0030] The present application proposes a robot referential segmentation and grasping detection method and system based on attribute perception, which designs the extraction and alignment of word-level and visual features, multi-modal interaction, visual attribute extraction and segmentation mask generation, visual attribute feature integration and grasping pose refinement, three-part reasoning process, extracts features and attributes of the referential target and generates fine-grained segmentation mask, provides prior constraints for grasping detection, aggregates the perceived visual attribute features and the semantic rich image visual features, constrains and refines the spatial position and width of the referential target, generates high-quality grasping pose, improves the ability of multi-modal fusion and alignment, and realizes effective target segmentation and optimal grasping pose.

[0031] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated by reference in their entirety. The embodiments of the application, together with its advantages, can best be understood by referring to the following description taken in connection with the accompanying drawings in which:

[0033] Figure 1 A flow chart of attribute perception based robot pointing segmentation and grasp detection method provided for embodiment 1 of the present application;

[0034] Figure 2 A data example diagram provided for embodiment 1 of the present application;

[0035] Figure 3 A framework diagram of attribute perception based robot pointing segmentation and grasp detection provided for embodiment 1 of the present application;

[0036] Figure 4 An attribute perception multi-modal interaction module schematic diagram provided for embodiment 1 of the present application;

[0037] Figure 5 A result schematic diagram after training and inference provided for embodiment 1 of the present application. DETAILED DESCRIPTION

[0038] The application will be further described below in conjunction with the drawings and embodiments.

[0039] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0040] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the application will be limited only by the appended claims. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. Unless otherwise required by context, singular terms shall include pluralities and vice versa. Plural elements can be separated by a hyphenated form of the element name, for example, "multiple components", unless otherwise indicated. Unless otherwise indicated, the use of "or" in the examples herein shall not be understood as an exclusive "or" unless explicitly indicated otherwise. Unless otherwise indicated, the use of "and" in the examples herein is intended to represent a functional "and" rather than a conjunctive "and" unless explicitly indicated otherwise. Although the terms "first", "second", "third", etc. can be used in this disclosure to describe various steps or units, these terms are not intended to denote a chronological order, a sequential order, or a specific order, but are intended to denote different components or steps. The use of "about" in the examples herein shall not be understood as excluding the stated value or limit unless explicitly indicated otherwise. The use of "comprise", "comprises" or "comprising" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "include", "includes" or "including" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "or" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "and" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "include", "includes" or "including" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "or" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise. The use of "and" in the examples herein shall not be understood as excluding the presence of elements other than those stated unless explicitly indicated otherwise.

[0041] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] Term explanation:

[0043] Reference segmentation and grasping detection: Given a scene image and a natural language description of a target to be grasped (i.e., a referent target), the pixels of a specific target object or region in the image are segmented according to the language description, and the optimal grasping pose of the target is given.

[0044] CLIP: Contrastive Language-Image Pre-Training (CLIP) is a pre-trained neural network model for matching images and text, commonly used for downstream tasks in multi-modal tasks.

[0045] Transformer: A neural network architecture for natural language processing (NLP) tasks, consisting of an encoder and a decoder. The encoder converts input text into a series of vector representations, while the decoder decodes these vectors into corresponding output text. Transformer is a neural network architecture based on self-attention mechanism, which can better handle sequence data and is widely used in natural language processing tasks.

[0046] Embodiment 1

[0047] As shown in Figure 1 , the embodiment provides a robot reference segmentation and grasping detection method based on attribute perception, comprising:

[0048] Obtain the scene image and the text description of the referent target, extract multi-scale visual features from the scene image, and extract word-level features from the text description of the referent target;

[0049] Perform average pooling operation on the multi-scale visual features, and use cross-modal dot product attention operation on the average pooled multi-scale visual features and word-level features to obtain interaction features, and connect the multi-scale visual features through top-down manner to obtain multi-scale visual attribute features of the referent target;

[0050] Calculate the segmentation mask according to the last layer visual attribute feature;

[0051] Constrain the segmentation mask, extract the grasping features of the referent target according to the last layer visual feature and multi-scale visual attribute feature, and predict the grasping position, width and grasping angle to obtain the optimal grasping pose.

[0052] In this embodiment, the robot obtains the scene image and the text description of the target

[0053] Randomly crop the edges of the scene image, randomly generate the crop width w and the crop height h, the cropping process is represented as C(w, h); randomly generate the rotation angle θ, and rotate around the center, represented as R(θ); randomly generate the adjustment parameters a, b, and c of brightness, contrast, and saturation, and then adjust them step by step, represented as L(a, b, c).

[0054] After the above processing of the scene image, an enhanced image I e is obtained, which increases the diversity of data.

[0055] The enhanced image I e is normalized to the range [0, 1] by dividing all pixels by 255.0, and then the pixel values are adjusted to the range [-1, 1] according to the mean α and standard deviation σ of the pixels, thus obtaining the normalized image I n = (I e / 255.0-α) / σ.

[0056] The tokenizer tokenizer of CLIP is used to vectorize the text description; first, the text sequence is marked with [SOS] and [EOS], and each word e is encoded by lowercase byte pair encoding (BPE), with a vocabulary size of 49152, so the text description ζ is encoded into a text vector E S .

[0057] In this embodiment, the RefOCIDGrasp dataset suitable for robot reference segmentation and grasp detection is constructed based on the OCID-grasp and OCID-Ref datasets. The reference expressions and object masks in OCID-Ref are used to filter out grasp candidate objects of specific objects from OCID-grasp, and then some incorrect and mismatched targets are removed for data cleaning. Finally, a large-scale dataset containing associated language expressions, grasp annotations and segmentation masks is generated for training and evaluating related models.

[0058] These datasets are collected in an indoor cluttered scene, including three spatial cases: free, touch and stack, containing a total of 210057 instance-text pairs, with an average text length of 8.4; the dataset is divided into training set, validation set and training set, containing 178647, 12606 and 18804 expression numbers respectively; an example of the data is shown in Figure 2 , where the referred target is a bottle in the scene.

[0059] In this embodiment, the model inference part is divided into three parts: CLIP-based word-level and visual feature extraction and alignment, multi-modal interaction, attribute extraction and segmentation mask generation, visual attribute feature integration and grasp pose refinement. The three parts are respectively called listening module, perception module and grasping module, and the specific process is as shown in Figure 3 The prior knowledge of the pre-trained model CLIP is used to build the listening module to realize the alignment of visual and language concepts. The perception module is built in a top-down manner to realize the recursive integration of specific target features and the segmentation of the referred target. The perception module is used to mine the features and attributes (such as boundaries and spatial positions) of the referred target and generate fine-grained segmentation masks to provide prior constraints for grasp detection. The grasping module aggregates the visual attribute features in the perception module and the semantic-rich features in the listening module to constrain and refine the spatial position and width of the grasp rectangle, which facilitates the generation of high-quality grasp poses.

[0060] The above will be described in detail in combination with Figure 3 .

[0061] Listening module

[0062] For the normalized image The multi-scale visual features F i are extracted using the CLIP pre-trained ResNet50 multi-level image encoder.

[0063] Among them,

[0064] It is worth noting that the low-level features maintain higher resolution and clearer spatial information, which is conducive to the positioning of the target; the high-level features contain more semantic information, which is helpful to understand complex attributes and relationships.

[0065] For the text vector E S , a modified Transformer model with 12 layers of 512 width and 8 attention heads is used to extract word-level features of natural language description

[0066] Perception module

[0067] Generally, the target pixels show mapping correlation with the pixel distribution in both vertical and horizontal dimensions. If the shape distribution in both directions is known, the approximate distribution of the target pixels can be inferred, and vice versa.

[0068] In order to build the above relationship, the attribute-aware multi-modal interaction (TAMMI) is designed to align the image and language context; as Figure 4As shown, it includes three key structures: average pooling, cross-modal dot product attention (CMDPA) and dense residual connection.

[0069] Specifically:

[0070] The segmentation mask not only indicates the spatial position of the object, but also provides visual attribute information such as shape, boundary, scale, etc. in the image. Therefore, combining pixel-wise attribute features can accelerate object perception and promote network convergence; Specifically, the average pooling method is used to calculate the area distribution in the height or width direction of the multi-scale visual features, which effectively avoids the problem of too large gradient caused by operation and the problem of scale change caused by upsampling;

[0071]

[0072] where k∈{H′,W′},Y (k) represents the feature mapping in the k direction, H′, W′, C′ represent the height, width and channel of the feature mapping respectively.

[0073] In order to analyze the expected target of language reference, cross-modal dot product attention (CMDPA) is used to extract pixel-level interaction features, which can focus on specific features corresponding to word-level features and reduce irrelevant redundant information. First, the multi-scale visual features Y (generally referring to Figure 4 Y H 1, Y W 2 and Y3) and word-level features l after average pooling are projected into intermediate representations, which include query keys keys Then, the scaled dot product attention operation is applied to obtain key information.

[0074]

[0075] where W Q , W K and W V are linear mapping parameters for calculating Q Y , K l and V l , and C K is the dimension of K l .

[0076] In order to enrich the visual attributes, dense residual connectivity is used to cascade the input and output in different directions. Specifically, the input of each attribute perception module is the element-wise accumulation of all previous scale-level attention mapping features in a feedforward manner and instance normalization (IN), which realizes better gradient propagation, promotes feature reuse, and reduces overfitting;

[0077]

[0078] wherein, is the attribute perception module, which consists of IN, AveragePooling and CMDPA; and l is the level of the attribute perception module. is the attribute perception feature output of the lth level.

[0079] When i = 1 or 2, the attribute perception module performs area distribution interaction in the width and height directions; when i = 3, the attribute perception module removes AveragePooling and performs pixel-by-pixel interaction; and when i = 4, only IN is used.

[0080] Object segmentation is a fine-grained perception task. The segmentation mask can reflect the boundaries, shapes, scales, spatial positions and other basic visual attributes of objects, so that the robot can more accurately grasp or operate. The main task of the perception module is to transfer the prior knowledge of CLIP from the language to image paradigm to the word to pixel paradigm. Generally speaking, large-scale feature maps contain more accurate spatial information, which can promote the positioning of the target, and small-scale feature maps contain more semantic information, which is beneficial to the inference of the relationship between pixels.

[0081] In order to capture complex boundaries and details, multi-scale visual features F i , i = 1, 2, 3, 4; that is, F4 and F3 are connected and input into the TAMMI module with the word-level feature, and the output V3 of the TAMMI module is connected with F2 and then input into the TAMMI module with the word-level feature, and the output V2 of the TAMMI module is connected with F1 and then input into the TAMMI module with the word-level feature, and finally output V1; as shown in Figure 3

[0082] The multi-scale visual attribute feature of the target of reference obtained in this way is:

[0083] V i = p ([Up2 (V i+1 ), F i ]), i = 3, 2, 1 (4)

[0084] wherein, p (·) represents a Conv-IN-LeakyReLU structure consisting of a 3x3 convolution, IN and LeakyReLU; and Up2 is an up-sampling with a factor of 2.

[0085] The segmentation mask S is generated by using a Conv-IN-LeakyReLU structure, a 1x1 convolution, up-sampling and a sigmoid function.

[0086] ​​​​S = sigmoid(Up4(Conv(φ(V1))))

[0087] where φ(·) denotes a Conv-IN-LeakyReLU structure, and Up4 is a 4 times up-sampling.

[0088] grasping module

[0089] The center position of the grasping rectangle is usually contained in the segmentation mask of the target, and the grasping angle and width are closely related to the shape, boundary and scale of the target, which is a powerful prerequisite for generating the grasping pose. Therefore, the grasping module is designed to refine the grasping pose, rather than sharing a decoder with the segmentation mask prediction.

[0090] Specifically, the attribute perception object feature map V i , i ∈ {2, 3, 4} further refines the grasping pose; in addition, the semantic-rich multi-scale visual feature F4 in CLIP provides global context to help fill in the missing information of the asymmetric object;

[0091] P2 = Conv([F4, V1, V2, V3]) (5)

[0092] where the convolution is a 1x1 convolution;

[0093] Then, two Conv-IN-LeakyReLU structures and a TAMMI module are used to further extract the grasping features of the referential target;

[0094] P1 = φ(Up2(TAMMI(φ(P2)))) (6)

[0095] where φ(·) is a Conv-IN-LeakyReLU structure; the convolution is a 3x3 convolution;

[0096] Finally, four 1x1 convolutions are used to predict the grasping position Q, width W and grasping angle Φ of P1, and the grasping angle

[0097] Therefore, the feedforward process is described as:

[0098]

[0099] In this embodiment, the training phase of the above model, the grasping pose G of the referential target and the segmentation mask S are jointly supervised;

[0100] First, given the segmentation mask The segmentation error is calculated using the binary cross-entropy loss:

[0101]

[0102] Where N is the number of pixels; S i To predict the score of the i-th pixel in the segmentation mask.

[0103] Then, given the actual grasping posture The error in the capture mapping is calculated using SmoothL1 loss:

[0104]

[0105] Wherein, the captured graph M∈{Q,W,cos(2Φ),sin(2Φ)}; To capture the predicted value of the i-th pixel in image m; To capture the true value of the i-th pixel in image m.

[0106] Finally, the overall training loss is obtained to supervise model training:

[0107] L joint =αL s +βL g (10)

[0108] Where α and β are the balancing weights for segmentation and grasping training.

[0109] During the inference phase, the segmentation mask S is calculated, and a threshold λ is set. s The final predicted segmentation mask S is obtained. * Then, using this as a constraint, the optimal grasping posture G is obtained. * =(x * ,y * ,θ * ,w * ,q * );

[0110]

[0111] Among them, (x * ,y * θ is the pixel location with the highest score in the quality map Q, representing the center point of the optimal grasping posture; * The position (x) in the fraction graph * ,y * The angle value below represents the optimal grasping angle; w * and q * They represent the positions (x) * ,y * ) Optimal crawl width and crawl quality.

[0112] The results after training and inference are as follows: Figure 5As shown, the method of the embodiment not only considers the interaction and alignment of multi-modal features, but also considers the extraction of visual attributes of the referent target to facilitate the refinement of the grasping pose, effectively improving the accuracy of referent segmentation and grasp detection.

[0113] Embodiment 2

[0114] The embodiment provides a robot referent segmentation and grasp detection system based on attribute perception, comprising:

[0115] The feature extraction module is configured to acquire a scene image and a text description of a referent target, extract multi-scale visual features from the scene image, and extract word-level features from the text description of the referent target;

[0116] The attribute perception multi-modal interaction module is configured to perform average pooling operation on the multi-scale visual features, and obtain interaction features by using cross-modal dot product attention operation on the multi-scale visual features after the average pooling and the word-level features, and obtain multi-scale visual attribute features of the referent target by connecting the multi-scale visual features in a top-down manner.

[0117] The segmentation mask generation module is configured to generate a segmentation mask according to the last layer of visual attribute features.

[0118] The grasp pose prediction module is configured to constrain the segmentation mask, extract grasp features of the referent target according to the last layer of visual features and the multi-scale visual attribute features, and predict a grasp position, a width and a grasp angle according to the grasp features, so as to obtain an optimal grasp pose.

[0119] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0120] In more embodiments, there are also provided:

[0121] An electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the method described in Embodiment 1 is completed. For brevity, it will not be repeated here.

[0122] It should be understood that in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSPs, application-specific integrated circuits ASICs, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0123] The memory can include read-only memory and random access memory, and provide the processor with instructions and data. A portion of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0124] A computer readable storage medium for storing computer instructions, which, when executed by a processor, completes the method described in Embodiment 1.

[0125] The method in Embodiment 1 can be directly embodied as a hardware processor to complete, or be completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0126] Those of ordinary skill in the art can realize that the units of the examples described in combination with the embodiments, i.e. the algorithm steps, can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software mode depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0127] Although the specific embodiments of the present application are described above in combination with the drawings, it is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A robot referencing segmentation and grasping detection method based on attribute awareness, characterized in that, include: Acquire scene images and text descriptions of the referents; extract multi-scale visual features from the scene images; and extract word-level features from the text descriptions of the referents. Average pooling is performed on multi-scale visual features. Cross-modal dot product attention is applied to the average pooled multi-scale visual features and word-level features to obtain interactive features. These interactive features are then connected to the multi-scale visual features in a top-down manner to obtain multi-scale visual attribute features that refer to the target. The process of obtaining interactive features includes: projecting the multi-scale visual features and word-level features after average pooling onto the intermediate representation, applying the scaling dot product attention operation, and then accumulating and connecting the output with all previous inputs and performing instance normalization operations to finally output the interactive features; When extracting interaction features, three layers of cross-modal dot product attention operations are included. The first and second layers of cross-modal dot product attention operations include instance normalization, average pooling, and cross-modal dot product attention operations, respectively. The third layer of cross-modal dot product attention operations removes the average pooling operation, and its output is then concatenated with the instance normalization operation. The process of connecting to multi-scale visual features in a top-down manner includes: the multi-scale visual features consist of four layers of visual features; the fourth layer is connected to the third layer and then interacts with the word-level features; the output visual attribute features are then connected to the second layer and then interact with the word-level features; finally, the output visual attribute features are connected to the first layer and then interact with the word-level features, thus obtaining the last layer of visual attribute features. Calculate the segmentation mask based on the visual attribute features of the last layer; Using the segmentation mask as a constraint, the grasping features of the target are extracted based on the last layer of visual features and multi-scale visual attribute features. These features are then used to predict the grasping position, width, and grasping angle, thereby obtaining the optimal grasping posture.

2. The attribute-aware robot referencing segmentation and grasping detection method as described in claim 1, characterized in that, After convolution of the last layer of visual features and multi-scale visual attribute features, it is then processed by two more convolution operations. The structure, composed of convolution, instance normalization, and activation functions, along with cross-modal dot product attention operations, extracts grasping features representing the target. These grasping features are then processed through four... Convolutions predict the grasping position, width, and grasping angle, respectively.

3. The attribute-aware robot referencing segmentation and grasping detection method as described in claim 1, characterized in that, The process of calculating the segmentation mask based on the last layer of visual attribute features includes: sequentially processing the last layer of visual attribute features... After convolution, instance normalization, and activation functions, then... Convolution and upsampling operations predict the segmentation mask.

4. The attribute-aware robot referencing segmentation and grasping detection method as described in claim 3, characterized in that, The calculated segmentation mask is filtered according to the set threshold to obtain the final predicted segmentation mask.

5. An attribute-aware robot referencing segmentation and grasping detection system, characterized in that, include: The feature extraction module is configured to acquire scene images and text descriptions of referential targets, extract multi-scale visual features from the scene images, and extract word-level features from the text descriptions of referential targets. The attribute-aware multimodal interaction module is configured to perform average pooling on multi-scale visual features, and then use cross-modal dot product attention operation on the average pooled multi-scale visual features and word-level features to obtain interaction features. After connecting with the multi-scale visual features in a top-down manner, the multi-scale visual attribute features of the referent target are obtained. The process of obtaining interactive features includes: projecting the multi-scale visual features and word-level features after average pooling onto the intermediate representation, applying the scaling dot product attention operation, and then accumulating and connecting the output with all previous inputs and performing instance normalization operations to finally output the interactive features; When extracting interaction features, three layers of cross-modal dot product attention operations are included. The first and second layers of cross-modal dot product attention operations include instance normalization, average pooling, and cross-modal dot product attention operations, respectively. The third layer of cross-modal dot product attention operations removes the average pooling operation, and its output is then concatenated with the instance normalization operation. The process of connecting to multi-scale visual features in a top-down manner includes: the multi-scale visual features consist of four layers of visual features; the fourth layer is connected to the third layer and then interacts with the word-level features; the output visual attribute features are then connected to the second layer and then interact with the word-level features; finally, the output visual attribute features are connected to the first layer and then interact with the word-level features, thus obtaining the last layer of visual attribute features. The segmentation mask generation module is configured to generate a segmentation mask based on the visual attribute features of the last layer; The grasping posture prediction module is configured to extract grasping features of the target based on the last layer of visual features and multi-scale visual attribute features, constrained by the segmentation mask, and use these features to predict the grasping position, width and grasping angle, thereby obtaining the optimal grasping posture.

6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-4.

Images