A pixel-level grasping and detection method, device and storage medium

Abstract

This invention provides a pixel-level grasping detection method, apparatus, and storage medium, relating to the field of machine vision technology. The method includes: introducing skip connections: based on a generative residual convolutional neural network (GRN), concatenating the output features of the convolutional layers and the output features of the deconvolutional layers of the GRN; introducing an attention mechanism: introducing an ECA channel attention mechanism into the residual blocks of the GRN, assigning weights to each channel of the feature map through an ECA module; improving the loss function: assigning higher weights to the region where the object is located through a weighted mask to obtain an optimized loss function; and detecting the object to be grasped based on the improved GRN. The pixel-level grasping detection method provided by this invention can more accurately capture the local geometric information and global structural relationships of an object, thereby improving the accuracy and robustness of detection.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and more specifically, to a pixel-level grasping and detection method, apparatus, and storage medium. Background Technology

[0002] As a core capability of robotics, robot grasping and detection plays an indispensable role in fields such as home, industry, medicine, and military, and is key to measuring the quality of robot perception and interaction with unknown environments. The robot's ability to accurately perceive and locate the geometric posture of a target object directly affects grasping efficiency and accuracy. However, how to perform fast and stable grasping and detection using visual information when faced with objects of unknown shape and position remains a major challenge. In recent years, to improve the accuracy of grasping and detecting unknown objects in unstructured environments, many studies have chosen to use deep learning methods to improve detection efficiency: Solution 1: Using the sliding window method to predict the grasping box. This involves sliding windows of different sizes and positions on the image to detect potential grasping areas. However, with increasing image resolution and sliding step size, the computational load increases exponentially, leading to low computational efficiency and making it difficult to meet the requirements of real-time robot grasping tasks. Solution 2: Using an anchor-box-based grasping detection algorithm. This divides the image into an N×N grid network for regression prediction, taking the cell with the highest grasping probability as the prediction result. The drawback is that it cannot directly handle rotation angles; it must use a regression network to predict the grasping box angle, which adds extra complexity to the network and easily leads to prediction errors. Option 3: A robot grasping detection algorithm based on keypoint estimation, using CenterNet as the basic framework, combines improved residual blocks and the SE attention module to find the center point of the grasping rectangle. It achieved an accuracy of 97.75% on the Cornell grasping dataset. However, the SE attention module introduces a certain number of parameters, which significantly increases computational overhead when the number of channels is large. In summary, existing technical solutions generally have low accuracy or efficiency in detecting and grasping unknown objects in unstructured environments, and further improvements in both accuracy and efficiency are needed.

[0003] Therefore, how to improve the accuracy and performance of the grasping and detection task without significantly increasing the computational load has become an urgent technical problem to be solved. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art, and discloses a pixel-level grasping detection method, device and storage medium, which can pay more attention to the characteristics of objects and has good robustness and accuracy in grasping detection.

[0005] The first aspect of this invention discloses a pixel-level grasping detection method, comprising: introducing skip connections: based on a generative residual convolutional neural network (GR-ConvNet), concatenating the output features of the convolutional layers of the GR-ConvNet with the output features of the deconvolutional layers; introducing an attention mechanism: introducing an ECA channel attention mechanism in the residual blocks of the GR-ConvNet, assigning weights to each channel of the feature map through an ECA module; improving the loss function: generating a mask matrix, which is a matrix with the same size as the input image, where each element of the mask matrix represents whether the corresponding pixel belongs to the area where the item is located; generating a weight matrix, where the weight values ​​in the weight matrix correspond one-to-one with the elements of the mask matrix, and each weight value represents the degree of attention paid to the pixel when calculating the loss, assigning a first weight to mask matrix elements located in the area where the item is located, and assigning a second weight to mask matrix elements outside the item area, with the first weight being greater than the second weight; multiplying the weight matrix by the calculation result of the Smooth L1 loss function of the GR-ConvNet to obtain an optimized loss function; and detecting the item to be grasped based on the improved GR-ConvNet.

[0006] The pixel-level grasping detection method disclosed in this invention preferably further includes: training an improved generative residual convolutional neural network; and outputting grasping quality, grasping angle, and grasping width based on the acquired image of the object to be grasped, so that the grasping device can perform a grasping action.

[0007] According to the pixel-level grasping detection method disclosed in this invention, preferably, the step of generating a mask matrix includes: creating a blank mask matrix with the same size as the input image; determining the grasping region based on grasping rectangle information or grasping annotation information; setting the mask of pixels within the grasping region to 1, and setting the mask of pixels outside the grasping region to 0.

[0008] According to the pixel-level grasping detection method disclosed in this invention, preferably, for the Cornell dataset, the grasping point is represented by a rectangle, and the coordinates of the four vertices of the rectangle are recorded in the annotation file. For each grasping rectangle, the mask corresponding to the pixels within the rectangle is set to 1, and the other areas are set to 0. For the Jacquard dataset, each grasping point is represented by a center point and a direction angle. The center point and direction angle information are recorded in the annotation file. For each grasping point, a rectangular region is generated based on the center point and direction angle, and the mask corresponding to the pixels within the rectangular region is set to 1, and the other areas are set to 0.

[0009] According to the pixel-level grasping and detection method disclosed in this invention, preferably, the processing procedure of the ECA module includes: performing global average pooling on the input feature map; performing a one-dimensional convolution operation on the channel information after global average pooling; generating channel weights through the non-linear activation function Sigmoid; and multiplying the channel weights by the input feature map channel by channel.

[0010] According to the pixel-level grasping and detection method disclosed in this invention, preferably, the activation function used in the feature extraction process of generating the residual convolutional neural network is the Mish function.

[0011] According to the pixel-level grasping and detection method disclosed in this invention, preferably, generating a residual convolutional neural network specifically includes: a convolutional layer for downsampling feature extraction of the input image; a residual block for feature extraction of the feature map output by the convolutional layer; and a deconvolutional layer for upsampling the feature map output by the residual block.

[0012] According to the pixel-level capture detection method disclosed in this invention, preferably, the input image is sequentially passed through 3 convolutional layers, 5 residual blocks, and 3 deconvolutional layers to extract features.

[0013] A second aspect of the present invention discloses a pixel-level grasping detection device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory to implement the pixel-level grasping detection method as described in any of the above technical solutions.

[0014] A third aspect of the present invention discloses a computer-readable storage medium storing program code for implementing the pixel-level grasping and detection method as described in any of the above technical solutions.

[0015] The beneficial effects of this invention include at least the following: Generative Residual Convolutional Neural Network (GR-ConvNet) is a model capable of learning the quality of grasping points, grasping width, and grasping angle. It can take RGB images and corresponding depth images as multimodal inputs and effectively learn multidimensional feature representations through end-to-end joint training. Considering that in pixel-level grasping detection tasks, the detection accuracy of GR-ConvNet is easily affected by the ambiguity of the relative positions between objects and the heterogeneity of object sizes, this invention reduces the loss of fine-grained information by establishing skip connections between convolutional and deconvolutional layers; by fusing the ECA channel attention mechanism, it effectively helps the model to filter in the high-dimensional feature space, ignoring noise features irrelevant to the grasping task and focusing on features closely related to grasping detection; and by improving the loss function through weighted masking, the network pays more attention to the effective region where the target object is located, significantly increasing the accuracy of grasping detection. Attached Figure Description

[0016] Figure 1 A schematic diagram of the model structure corresponding to a pixel-level grasping and detection method according to an embodiment of the present invention is shown.

[0017] Figure 2 A schematic diagram of the internal structure of a convolutional layer according to an embodiment of the present invention is shown.

[0018] Figure 3 A schematic diagram of the internal structure of the Res_ECA module according to an embodiment of the present invention is shown.

[0019] Figure 4 A schematic diagram of the internal structure of a deconvolution layer according to an embodiment of the present invention is shown.

[0020] Figure 5 A schematic diagram of the ECA module structure according to an embodiment of the present invention is shown.

[0021] Figure 6 A schematic block diagram of a pixel-level grasping and detection device according to an embodiment of the present invention is shown. Detailed Implementation

[0022] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the invention is not limited to the specific embodiments disclosed below.

[0024] According to an embodiment of the present invention, the pixel-level grasping detection method disclosed herein includes: introducing skip connections: based on a generative residual convolutional neural network, the output features of the convolutional layer of the generative residual convolutional neural network are concatenated with the output features of the deconvolutional layer; introducing an attention mechanism: an ECA channel attention mechanism is introduced into the residual block of the generative residual convolutional neural network, and weights are assigned to each channel of the feature map through the ECA module; improving the loss function: generating a mask matrix, which is a matrix with the same size as the input image, and each element of the mask matrix represents whether the corresponding pixel belongs to the area where the item is located; generating a weight matrix, where the weight values ​​in the weight matrix correspond one-to-one with the elements of the mask matrix, and each weight value represents the degree of attention to the pixel when calculating the loss, assigning a first weight to mask matrix elements located in the area where the item is located, and assigning a second weight to mask matrix elements outside the item area, with the first weight being greater than the second weight; multiplying the weight matrix with the calculation result of the Smooth L1 loss function of the generative residual convolutional neural network to obtain the optimized loss function; and detecting the item to be grasped based on the improved generative residual convolutional neural network.

[0025] According to the above embodiments, preferably, it further includes: training an improved generative residual convolutional neural network; and outputting grasping quality, grasping angle, and grasping width based on the acquired image of the object to be grasped, so that the grasping device can perform grasping actions.

[0026] According to the above embodiments, preferably, the step of generating the mask matrix specifically includes: generating the mask matrix based on the coordinates of the grab rectangle or grab points. For the Cornell dataset, the mask matrix is ​​generated using a method based on the grab rectangle; for the Jacquard dataset, the mask matrix is ​​generated using a method based on grab annotations. The mask matrix is ​​generated before model training to give higher weights to the region where the object is located, so that the model pays more attention to the pixel features of the region where the object is located during training.

[0027] According to the above embodiments, preferably, the processing procedure of the ECA module includes: performing global average pooling on the input feature map; performing a one-dimensional convolution operation on the channel information after global average pooling; generating channel weights through the non-linear activation function Sigmoid; and multiplying the channel weights by the input feature map channel by channel.

[0028] According to the above embodiments, preferably, the activation function used in the feature extraction process of generating the residual convolutional neural network is the Mish function.

[0029] According to the above embodiments, preferably, generating a residual convolutional neural network specifically includes: a convolutional layer for downsampling feature extraction of the input image; a residual block for feature extraction of the feature map output by the convolutional layer; and a deconvolutional layer for upsampling the feature map output by the residual block. The input image sequentially passes through 3 convolutional layers, 5 residual blocks, and 3 deconvolutional layers to perform feature extraction.

[0030] like Figure 1 As shown, according to another embodiment of the present invention, a specific implementation process of the pixel-level grasping and detection method disclosed in the above embodiments is also disclosed:

[0031] Establish as described in the above embodiments Figure 1 The grabbing and detection model shown is as follows: The input image is downsampled through convolutional layers 1, 2, and 3. The output of the convolutional layers is fed into the improved residual blocks (Res_ECA1, Res_ECA2, Res_ECA3, Res_ECA4, Res_ECA5). At the same time, the output of the convolutional layers is concatenated with the output of the deconvolutional layers through skip connections. After upsampling through deconvolutional layers 1, 2, and 3, the feature map is restored to the same size as the input image. Then, the quality image, angle image, and width image are obtained through conv2D and Dropout operations.

[0032] like Figure 2 As shown, the internal structure of each convolutional layer is the same, including: Conv2D (convolution) operation, BatchNorm (batch normalization) operation, and activation function Mish.

[0033] like Figure 3 As shown, the ECA attention mechanism is introduced into the residual block of GR-ConvNet to form the Res_ECA module. The internal structure of each Res_ECA module is the same, including: Conv2D operation, BatchNorm operation, activation function Mish, Conv2D operation, BatchNorm operation and ECA operation.

[0034] like Figure 4 As shown, the internal structure of each deconvolution layer is the same, including: ConvT2D (deconvolution) operation, BatchNorm operation, and activation function Mish.

[0035] Specifically, the pixel-level grasping and detection model provided by this invention has the following improvements compared to the existing GR-ConvNet model:

[0036] (1) Adding an ECA module: The Generative Residual Convolutional Neural Network (GR-ConvNet) can take RGB images and depth images as multimodal inputs and effectively learn multidimensional feature representations through end-to-end joint training. In pixel-level grasping detection tasks, the detection accuracy is significantly affected by the ambiguity of the relative positions between objects and the heterogeneity of object sizes. Assuming an input RGB image and the corresponding depth image, GR-ConvNet extracts multi-channel features by stacking convolutional layers, but it still has shortcomings in capturing the global dependencies between the features of each channel. To overcome this limitation, this invention introduces the ECA channel attention mechanism on the basis of the existing GR-ConvNet, forming the Res_ECA module. ECA (Efficient Channel Attention) optimizes the network's efficiency in utilizing high-dimensional features by adaptively assigning weights to different channels.

[0037] (2) Adding Skip Connections: In generative residual convolutional neural networks, as the network depth increases, the spatial resolution of the feature maps gradually decreases, causing high-level features to lose fine-grained information from lower layers. To alleviate this problem, this invention introduces skip connections in GR-ConvNet to transfer fine-grained features F from shallow layers. (s) The expression is passed directly to the upsampling part:

[0038] F (out) =Concat(F (s) Upsample(F (k) ))

[0039] Among them, F (out) The output of the upsampling operation is represented by `Upsample`, and `Concat` represents the concatenation of features along the channel dimension. Shallow fine-grained features refer to the features output by the three convolutional layers of GR-ConvNet, each of which halves the spatial resolution of the feature map. The specific steps of feature concatenation include: during downsampling, each layer produces feature maps of different sizes; similarly, during subsequent upsampling, feature maps of different sizes are produced. When two feature maps are of the same size, they can be directly stacked together to form a new multi-channel feature map. This new feature map contains information from all the original feature maps, thus providing richer contextual information to subsequent layers.

[0040] (3) Improved loss function: GR-ConvNet uses the Smooth L1 loss function to learn the quality of the grasp point, the grasp width, and the grasp angle. The expression of the loss function is:

[0041]

[0042] Where s represents the Smooth L1 loss function, and G represents the grasping pose predicted by the network. This represents the actual grasping pose. It can be seen that GR-ConvNet uses the Smooth L1 loss function to calculate the error between the predicted and actual values ​​for each pixel. That is, the traditional GR-ConvNet assigns equal weight to the object region and the background region, causing the network to favor optimizing invalid grasping points outside the masked area. Since most pixels are invalid grasping points, this causes the network to ignore grasping points in the object region, thus reducing the overall grasping detection accuracy.

[0043] To address the aforementioned issues with the Smooth L1 loss function, this invention assigns higher weights to the regions containing objects using weighted masks, allowing the network to focus more on the effective regions where objects reside. For the Cornell and Jacquard datasets used in the experiments, mask matrices M are generated using methods based on grasping rectangles and grasping annotations, respectively. Specifically, for the Cornell dataset, grasping rectangles are first read; each grasping point is represented by a rectangle, and the coordinates of the four vertices of the rectangle are recorded in the annotation file. Then, an empty mask matrix M with the same size as the input image is created. For each grasping rectangle, the pixel values ​​within the rectangle are set to 1, while other areas are set to 0. For the Jacquard dataset, grasping annotations are first read; each grasping point is represented by a center point and an orientation angle, and this information is recorded in the annotation file. Then, an empty mask matrix M with the same size as the input image is created. For each grasping point, a rectangular region is generated based on the center point and orientation angle, and the pixel values ​​within this region are set to 1, while other areas are set to 0. The mask matrices are generated before model training to give higher weights to the regions containing objects, making the model focus more on the pixel features of these regions during training. The mask matrix is ​​a matrix of the same size as the input image. Each element indicates whether a pixel belongs to the region containing the item. If the pixel at coordinates (i, j) is within the valid region, then M... ij =1, if the pixel with coordinates (i, j) is located in the invalid region, then M ij =0.

[0044] A weight matrix W can be generated using the mask matrix M, where each element represents the degree of attention given to a pixel when calculating the loss. Pixels in the effective region are assigned a higher weight α, while pixels in the background region (ineffective region) are assigned a lower weight β. The weight matrix W corresponds to each element of the mask matrix. ij The definition is as follows:

[0045]

[0046] The weight matrix W ijIntroducing the Smooth L1 loss function multiplies the loss of each pixel by its corresponding weight. This amplifies the loss in the object's region and reduces the loss in the background region, thus guiding the network to focus more on the effective grasping area where the object is located. Its definition is as follows:

[0047]

[0048] Among them, L weighted-smoothL1 Here, is the optimized loss function, i is the x-coordinate of a pixel, j is the y-coordinate of a pixel, H is the image height, W is the image width, and s is the Smooth L1 loss function.

[0049] like Figure 5 As shown, another embodiment of the present invention discloses the specific structure of the ECA module of the above embodiment: ECA first performs global average pooling on the features of each channel to obtain the global statistics of each channel, and then models the local dependencies between channels through one-dimensional convolution operations. In this way, the network can more effectively utilize the features of key channels and improve the ability to extract fine-grained features. The specific process includes: first, performing global average pooling (GAP) on the input feature map, assuming that the size of the input feature map is H×W×C, the feature map is compressed to 1×1×C after global average pooling. Then, a one-dimensional convolution of size k is used to model the dependencies between channels, and a convolution operation is performed on the channel information after global average pooling. Compared with the SE module, the ECA module avoids fully connected layers, thus significantly reducing the number of parameters. Then, the final channel weights are generated through the nonlinear activation function Sigmoid, and the obtained weights are multiplied by the original feature map by channel, thereby realizing the recalibration of the importance of different channels. In pixel-level grasping and detection tasks, the introduction of the ECA channel attention mechanism enables the network to more accurately capture the local geometric information and global structural relationships of objects, thereby improving the detection accuracy and robustness.

[0050] like Figure 6 As shown, according to another embodiment of the present invention, a pixel-level grasping detection device 600 is also disclosed, including: a memory 601 for storing program instructions; and a processor 602 for calling the program instructions stored in the memory to implement the pixel-level grasping detection method as described in the above embodiment.

[0051] According to another embodiment of the present invention, a computer-readable storage medium is also disclosed, which stores program code for implementing the pixel-level grasping and detection method as described in the above embodiments.

[0052] Based on the above embodiments, this invention addresses the problem of low accuracy in robot grasping and detecting unknown objects in unstructured environments by proposing a pixel-level grasping and detection method that integrates skip connections and attention mechanisms: The method improves the generative residual convolutional neural network model by introducing skip connections and ECA channel attention modules to further enhance the network's feature learning ability; it also improves the Smooth L1 loss function by assigning different weights to the regions where objects are located through weighted masks; experimental verification shows that the improved model achieves accuracies of 98.8% and 96.4% on the Cornell and Jacquard datasets, respectively.

[0053] All or part of the steps in the various methods of the above embodiments can be implemented by a program controlling the relevant hardware. The program can be stored in a readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A pixel-level grasping and detection method, characterized in that, include: Introducing skip connections: Based on a generative residual convolutional neural network, the output features of the convolutional layers of the generative residual convolutional neural network are concatenated with the output features of the deconvolutional layers, thereby combining the fine-grained features F of the shallow layers. (s) The expression is passed directly to the upsampling part: ； Among them, F (out) This represents the output of the upsampling operation. Indicates an upsampling operation. This indicates feature concatenation along the channel dimension; shallow fine-grained features refer to the features output by the three convolutional layers of GR-ConvNet, each of which halves the spatial resolution of the feature map; the specific steps of feature concatenation include: during downsampling, each layer will have feature maps of different sizes, and during subsequent upsampling, there will also be feature maps of different sizes. When two feature maps are of the same size, they can be directly stacked together to form a new multi-channel feature map. An attention mechanism is introduced: an ECA channel attention mechanism is introduced into the residual block of the generated residual convolutional neural network. The ECA module assigns weights to each channel of the feature map. The processing steps of the ECA module include: performing global average pooling on the input feature map; performing a one-dimensional convolution operation on the channel information after global average pooling; generating channel weights using the non-linear activation function Sigmoid; and multiplying the channel weights by the input feature map channel by channel. Improved Loss Function: Generate a mask matrix, which is a matrix with the same size as the input image. Each element of the mask matrix indicates whether the corresponding pixel belongs to the object's region. Generate a weight matrix, where the weight values ​​in the weight matrix correspond one-to-one with the elements of the mask matrix. Each weight value represents the degree of attention given to a pixel when calculating the loss. A first weight is assigned to mask matrix elements located in the object's region, and a second weight is assigned to mask matrix elements outside the object's region. The first weight is greater than the second weight. Multiply the weight matrix by the calculation result of the Smooth L1 loss function used to generate the residual convolutional neural network to obtain the optimized loss function. GR-ConvNet uses the Smooth L1 loss function to learn the grasp point quality, grasp width, and grasp angle. The loss function expression is: ； Where s represents the Smooth L1 loss function, This represents the grab pose predicted by the network. This indicates the actual grasping pose; The improved generative residual convolutional neural network is used to detect the items to be grasped.

2. The pixel-level grasping and detection method according to claim 1, characterized in that, Also includes: Train the improved generative residual convolutional neural network; Based on the acquired image of the object to be grasped, the grasping quality, grasping angle, and grasping width are output so that the grasping device can perform the grasping action.

3. The pixel-level grasping and detection method according to claim 1, characterized in that, The step of generating the mask matrix specifically includes: Create a blank mask matrix with the same size as the input image; The capture area is determined based on the capture rectangle information or capture annotation information; Set the mask of pixels within the capture area to 1, and set the mask of pixels outside the capture area to 0.

4. The pixel-level grasping and detection method according to claim 3, characterized in that, For the Cornell dataset, each grab point is represented by a rectangle, and the coordinates of the four vertices of the rectangle are recorded in the annotation file. For each grab rectangle, the mask corresponding to the pixels within the rectangle is set to 1, and the mask for other areas is set to 0. For the Jacquard dataset, each grab point is represented by a center point and an orientation angle. The annotation file records the center point and orientation angle information. For each grab point, a rectangular region is generated based on the center point and orientation angle, and the mask corresponding to the pixels within the rectangular region is set to 1, and the mask for other areas is set to 0.

5. The pixel-level grasping and detection method according to claim 1, characterized in that, The activation function used in the feature extraction process for generating the residual convolutional neural network is the Mish function.

6. The pixel-level grasping and detection method according to claim 1, characterized in that, The generated residual convolutional neural network specifically includes: Convolutional layers are used to extract downsampled features from the input image; Residual blocks are used to extract features from the feature maps output by convolutional layers. The deconvolutional layer is used to upsample the feature map output by the residual block.

7. The pixel-level grasping and detection method according to claim 6, characterized in that, The input image is sequentially passed through 3 of the convolutional layers, 5 of the residual blocks, and 3 of the deconvolutional layers for feature extraction.

8. A pixel-level grasping and detection device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke the program instructions stored in the memory to implement the pixel-level grasping and detection method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for implementing the pixel-level grasping and detection method as described in any one of claims 1 to 7.

Images