Pixel-level grabbing detection method and device and storage medium

Abstract

The invention provides a pixel-level grabbing detection method and device and a storage medium, and relates to the technical field of machine vision, and the method comprises the steps: introducing jump connection: carrying out the feature splicing of the output features of a convolution layer of a generated residual convolutional neural network and the output features of a deconvolution layer based on the generated residual convolutional neural network; an attention mechanism is introduced, wherein an ECA channel attention mechanism is introduced into a residual block for generating the residual convolutional neural network, and weights are distributed to all channels of the feature map through an ECA module; improving a loss function: giving a higher weight to an area where the object is located through a weighted mask to obtain an optimized loss function; and detecting an article to be grabbed based on the improved generated residual convolutional neural network. According to the pixel-level capture detection method provided by the invention, the local geometric information and global structure relationship of the object can be captured more accurately, so that the detection precision and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a pixel-level capture detection method, device and storage medium. Background Art

[0002] As one of the core capabilities of robotics technology, robot grasping detection plays an indispensable role in the fields of home, industry, medical care, and military, and is the key to measuring the quality of robot perception and interaction with unknown environments. Whether the robot can accurately perceive and locate the geometric posture of the target directly affects the grasping efficiency and accuracy. However, how to use visual information for fast and stable grasping detection in the face of objects of unknown shape and position is still the main problem at present. In recent years, in order to improve the accuracy of grasping detection of unknown objects in unstructured environments, many studies have chosen to use deep learning methods to improve detection efficiency: Solution 1: Use the sliding window method to predict the grasping box, and slide windows of different sizes and positions on the image to detect potential grasping areas. With the increase of image resolution and sliding step size, the exponential growth of calculation leads to low calculation efficiency, which is difficult to meet the requirements of real-time grasping tasks of robots. Solution 2: Use the grasping detection algorithm based on anchor boxes, divide the image into N×N grid unit networks for regression prediction, and take the highest grasping probability among all cells as the prediction result; the disadvantage is that the rotation angle cannot be directly processed, and the grasping box angle must be predicted through the regression network, which increases the complexity of the network and easily leads to prediction errors. Solution 3: Robot grasping detection algorithm based on key point estimation, using CenterNet as the basic framework, combined with the improved residual block and SE attention module to find the center point of the grasping rectangle, achieved an accuracy of 97.75% on the Cornell grasping dataset, but the SE attention module will introduce a certain amount of parameters, which will significantly increase the computational overhead when the number of channels is large. In summary, the existing technical solutions for robots grasping unknown objects in unstructured environments are generally low in accuracy or efficiency, and need to further improve the accuracy and efficiency of grasping detection.

[0003] Therefore, how to improve the accuracy and performance of grasping detection tasks without significantly increasing the amount of computation has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a pixel-level grasping detection method, device and storage medium, which can pay more attention to the characteristics of the object and have good robustness and accuracy in grasping detection.

[0005] The first aspect of the present invention discloses a pixel-level grasping detection method, comprising: introducing a skip connection: based on a generative residual convolutional neural network (GR-ConvNet), the output features of the convolution layer of the generative residual convolutional neural network are feature spliced ​​with the output features of the deconvolution layer; introducing an attention mechanism: introducing an ECA channel attention mechanism in the residual block of the generative residual convolutional neural network, and assigning weights to each channel of the feature map through an ECA module; improving a loss function: generating a mask matrix, the mask matrix is ​​a matrix with the same size as the input image, and each element of the mask matrix indicates whether the corresponding pixel belongs to the area where the object is located; generating a weight matrix, the weight values ​​in the weight matrix correspond to the elements of the mask matrix one by one, each weight value indicates the degree of attention to the pixel when calculating the loss, and a first weight is assigned to the mask matrix elements in the area where the object is located, and a second weight is assigned to the mask matrix elements outside the area of ​​the object, and the first weight is greater than the second weight; multiplying the weight matrix by the calculation result of the Smooth L1 loss function of the generative residual convolutional neural network to obtain an optimized loss function; and detecting the object to be grasped based on the improved generative residual convolutional neural network.

[0006] According to the pixel-level grasping detection method disclosed in the present invention, preferably, it also includes: training an improved generative residual convolutional neural network; outputting grasping quality, grasping angle and grasping width according to the collected image of the object to be grasped, so that the grasping device can perform the grasping action.

[0007] According to the pixel-level capture detection method disclosed in the present invention, preferably, the step of generating a mask matrix specifically includes: creating a blank mask matrix with the same size as the input image; determining the capture area according to the capture rectangle information or the capture annotation information; setting the mask of the pixels within the capture area to 1, and setting the mask of the pixels outside the capture area to 0.

[0008] According to the pixel-level grasping detection method disclosed in the present invention, preferably, for the Cornell data set, the grasping point is represented by a rectangle, the coordinates of the four vertices of the rectangle are recorded in the annotation file, and for each grasping rectangle, the mask corresponding to the pixels in the rectangle is set to 1, and the other areas are set to 0; for the Jacquard data set, each grasping point is represented by a center point and a direction angle, and the center point and direction angle information are recorded in the annotation file. For each grasping point, a rectangular area is generated according to the center point and the direction angle, and the mask corresponding to the pixels in the rectangular area is set to 1, and the other areas are set to 0.

[0009] According to the pixel-level capture detection method disclosed in the present invention, preferably, the processing process of the ECA module includes: performing global average pooling on the input feature map; performing a convolution operation on the channel information after global average pooling using a one-dimensional convolution; generating channel weights through a nonlinear activation function Sigmoid; and multiplying the channel weights by the input feature map by channel.

[0010] According to the pixel-level capture detection method disclosed in the present invention, preferably, the activation function used in the feature extraction process of generating the residual convolutional neural network is the Mish function.

[0011] According to the pixel-level capture detection method disclosed in the present invention, preferably, generating a residual convolutional neural network specifically includes: a convolution layer, used to downsample the input image for feature extraction; a residual block, used to extract features from the feature map output by the convolution layer; and a deconvolution layer, used to upsample the feature map output by the residual block.

[0012] According to the pixel-level capture detection method disclosed in the present invention, preferably, the input image passes through 3 convolutional layers, 5 residual blocks and 3 deconvolution layers in sequence to perform feature extraction.

[0013] The second aspect of the present invention discloses a pixel-level capture detection device, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement a pixel-level capture detection method as any of the above technical solutions.

[0014] A third aspect of the present invention discloses a computer-readable storage medium, which stores a program code, and the program code is used to implement a pixel-level capture detection method as described in any of the above technical solutions.

[0015] The beneficial effects of the present invention include at least: the generative residual convolutional neural network (GR-ConvNet) is a model that can learn the quality of grasping points, grasping width and grasping angle, and can use RGB images and corresponding depth images as multimodal inputs. Through end-to-end joint training, it can effectively learn multidimensional feature representation. Considering that in pixel-level grasping detection tasks, the detection accuracy of GR-ConvNet is easily affected by the ambiguity of relative positions between objects and the heterogeneity of object sizes, the present invention reduces the loss of fine-grained information by establishing a jump connection between the convolution layer and the deconvolution layer; by fusing the ECA channel attention mechanism, it effectively helps the model to screen in the high-dimensional feature space, ignores noise features that are not related to the grasping task, and focuses on features that are closely related to grasping detection; the loss function is improved by weighted masking, so that the network pays more attention to the effective area where the target object is located, and significantly increases the accuracy of grasping detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a model structure corresponding to a pixel-level capture detection method according to an embodiment of the present invention is shown.

[0017] Figure 2 A schematic diagram of the internal structure of a convolutional layer according to an embodiment of the present invention is shown.

[0018] Figure 3 A schematic diagram of the internal structure of a Res_ECA module according to an embodiment of the present invention is shown.

[0019] Figure 4 A schematic diagram of the internal structure of a deconvolution layer according to an embodiment of the present invention is shown.

[0020] Figure 5 A schematic diagram of the ECA module structure according to an embodiment of the present invention is shown.

[0021] Figure 6 A schematic block diagram of a pixel-level capture detection device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] According to one embodiment of the present invention, the pixel-level grasping detection method disclosed in the present invention includes: introducing a jump connection: based on a generative residual convolutional neural network, the output features of the convolution layer of the generative residual convolutional neural network are feature spliced ​​with the output features of the deconvolution layer; introducing an attention mechanism: introducing an ECA channel attention mechanism in the residual block of the generative residual convolutional neural network, and assigning weights to each channel of the feature map through the ECA module; improving the loss function: generating a mask matrix, the mask matrix is ​​a matrix with the same size as the input image, and each element of the mask matrix indicates whether the corresponding pixel belongs to the area where the object is located; generating a weight matrix, the weight values ​​in the weight matrix correspond to the elements of the mask matrix one by one, each weight value indicates the degree of attention to the pixel when calculating the loss, and the mask matrix elements in the area where the object is located are assigned a first weight, and the mask matrix elements outside the object area are assigned a second weight, and the first weight is greater than the second weight; multiplying the weight matrix by the calculation result of the Smooth L1 loss function of the generative residual convolutional neural network to obtain an optimized loss function; and detecting the object to be grasped based on the improved generative residual convolutional neural network.

[0025] According to the above embodiment, preferably, it also includes: training the improved generative residual convolutional neural network; outputting the grasping quality, grasping angle and grasping width according to the collected image of the object to be grasped, so that the grasping device can perform the grasping action.

[0026] According to the above embodiment, preferably, the step of generating a mask matrix specifically includes: generating a mask matrix according to the grab rectangle or the grab point coordinates. For the Cornell dataset, the mask matrix is ​​generated by a method based on the grab rectangle; for the Jacquard dataset, the mask matrix is ​​generated by a method based on the grab annotation. The mask matrix is ​​generated before the model is trained, and the purpose is to give a higher weight to the area where the object is located, so that the model training pays more attention to the pixel features of the area where the object is located.

[0027] According to the above embodiment, preferably, the processing process of the ECA module includes: performing global average pooling on the input feature map; performing a convolution operation on the channel information after global average pooling using a one-dimensional convolution; generating channel weights through a nonlinear activation function Sigmoid; and multiplying the channel weights by the input feature map by channel.

[0028] According to the above embodiment, preferably, the activation function used in the feature extraction process of generating the residual convolutional neural network is the Mish function.

[0029] According to the above embodiment, preferably, the residual convolutional neural network is generated specifically including: a convolution layer for downsampling feature extraction of the input image; a residual block for extracting features from the feature map output by the convolution layer; and a deconvolution layer for upsampling the feature map output by the residual block. The input image passes through 3 convolution layers, 5 residual blocks and 3 deconvolution layers in sequence to extract features.

[0030] like Figure 1 As shown, according to another embodiment of the present invention, a specific implementation process of the pixel-level capture detection method disclosed in the above embodiment is also disclosed:

[0031] Corresponding to the above embodiment, the following is established: Figure 1 The grasp detection model shown: the input image is downsampled through convolution layer 1, convolution layer 2 and convolution layer 3, and the output of the convolution layer enters the improved residual block (Res_ECA1, Res_ECA2, Res_ECA3, Res_ECA4, Res_ECA5). At the same time, the output of the convolution layer is feature concatenated with the output of the deconvolution layer through a jump connection. After upsampling through deconvolution layer 1, deconvolution layer 2 and deconvolution layer 3, the feature map is restored to the same size as the input image, and then the quality image, angle image and width image are obtained through conv2D operation and Dropout operation.

[0032] like Figure 2 As shown, the internal structure of each convolution layer is the same, including: Conv2D (convolution) operation, BatchNorm (batch normalization) operation and activation function Mish.

[0033] like Figure 3 As shown in the figure, the ECA attention mechanism is introduced into the residual block of GR-ConvNet to form a Res_ECA module. The internal structure of each Res_ECA module is the same, including: Conv2D operation, BatchNorm operation, activation function Mish, Conv2D operation, BatchNorm operation and ECA operation.

[0034] like Figure 4 As shown, the internal structure of each deconvolution layer is the same, including: ConvT2D (deconvolution) operation, BatchNorm operation and activation function Mish.

[0035] Specifically, the pixel-level grasping detection model provided by the present invention has the following improvements compared with the existing GR-ConvNet model:

[0036] (1) Adding ECA module: The generated residual convolutional neural network (GR-ConvNet) can take RGB images and depth images as multi-modal inputs, and effectively learn multi-dimensional feature representations through end-to-end joint training. In the pixel-level grasping detection task, the detection accuracy is significantly affected by the ambiguity of the relative positions between objects and the heterogeneity of object sizes. Assuming the input RGB image and the corresponding depth image, GR-ConvNet extracts multi-channel features by stacking convolutional layers, but it is still insufficient in capturing the global dependencies between the features of each channel. In order to overcome this limitation, the present invention introduces the ECA channel attention mechanism based on the existing GR-ConvNet to form the Res_ECA module. ECA (Efficient Channel Attention) optimizes the network's utilization efficiency of high-dimensional features by adaptively assigning weights to different channels.

[0037] (2) Adding skip connections: In the generation of residual convolutional neural networks, as the network depth increases, the spatial resolution of the feature map gradually decreases, resulting in the loss of low-level fine-grained information in high-level features. To alleviate this problem, the present invention introduces skip connections in GR-ConvNet to transfer the shallow fine-grained features F (s) Directly passed to the upsampling part, the expression is:

[0038] F (out) =Concat(F (s) ,Upsample(F (k) ))

[0039] Among them, F (out) Represents the output of the upsampling operation, Upsample represents the upsampling operation, and Concat represents the feature concatenation in the channel dimension. Shallow fine-grained features refer to the features output by the three convolutional layers of GR-ConvNet. Each convolutional layer can halve the spatial resolution of the feature map. The specific operation steps of feature concatenation include: in the downsampling process, each layer will have feature maps of different sizes, and in the subsequent upsampling process, there will also be feature maps of different sizes. When the sizes of two feature maps are the same, they can be directly stacked together to form a new multi-channel feature map. The new feature map contains all the information of the original feature map, so it can provide richer contextual information to subsequent layers.

[0040] (3) Improved loss function: GR-ConvNet uses the Smooth L1 loss function to learn the grasping point quality, grasping width, and grasping angle. The loss function expression is:

[0041]

[0042] Among them, s represents the Smooth L1 loss function, G represents the grasping posture predicted by the network, Represents the actual grasping posture. It can be seen that GR-ConvNet uses the Smooth L1 loss function to calculate the error between the predicted value and the actual value of each pixel. That is, the traditional GR-ConvNet assigns equal weights to the area where the object is located and the background area, which causes the network to be more inclined to optimize invalid grasping points outside the mask area. Since most pixels are invalid grasping points, this will cause the network to ignore the grasping points in the area where the object is located, thereby reducing the overall grasping detection accuracy.

[0043] In view of the above problems existing in the Smooth L1 loss function, the present invention gives a higher weight to the area where the object is located through a weighted mask, so that the network pays more attention to the effective area where the object is located: the Cornell and Jacquard data sets used in the experiment are respectively generated by the method based on the grab rectangle and the grab annotation. Specifically, for the Cornell data set, the grab rectangle is first read, each grab point is represented by a rectangle, and the coordinates of the four vertices of the rectangle are recorded in the annotation file. Then, an empty mask matrix M with the same size as the input image is created. For each grab rectangle, the pixel value in the rectangle is set to 1, and the other areas are set to 0. For the Jacquard data set, the grab annotation is first read. Each grab point is represented by a center point and a direction angle. This information is recorded in the annotation file. Then, an empty mask matrix M with the same size as the input image is created. For each grab point, a rectangular area is generated according to the center point and the direction angle, and the pixel value in the area is set to 1, and the other areas are set to 0. The mask matrix is ​​generated before the model is trained, and the purpose is to give a higher weight to the area where the object is located, so that the model training pays more attention to the pixel features of the area where the object is located. The mask matrix is ​​a matrix of the same size as the input image. Each element of the matrix indicates whether the pixel belongs to the area where the object is located. If the pixel with coordinates (i, j) is in the valid area, then M ij = 1, if the pixel with coordinates (i, j) is located in the invalid area, then M ij =0.

[0044] Through the mask matrix M, a weight matrix W can be generated, each element of which indicates the degree of attention paid to the pixel when calculating the loss. Pixels in the valid area are given a higher weight α, while pixels in the background area (invalid area) are given a lower weight β. The weight matrix W corresponding to each element of the mask matrix is ij The definition is as follows:

[0045]

[0046] The weight matrix W ijIntroducing the Smooth L1 loss function, the loss of each pixel is multiplied by the corresponding weight, the loss of the object area will be amplified, and the loss of the background area will be reduced, thereby guiding the network to pay more attention to the effective grasping area where the object is located. Its definition is shown in the following formula:

[0047]

[0048] Among them, L weighted-smoothL1 is the optimized loss function, i is the horizontal coordinate of the pixel, j is the vertical coordinate of the pixel, H is the image height, W is the image width, and s is the Smooth L1 loss function.

[0049] like Figure 5 As shown, according to another embodiment of the present invention, the specific structure of the ECA module of the above embodiment is also disclosed: ECA first performs global average pooling on the features of each channel to obtain the global statistics of each channel, and then models the local dependencies between channels through a one-dimensional convolution operation. In this way, the network can more effectively utilize the features of key channels and improve the ability to extract fine-grained features. The specific process includes: first, global average pooling (GAP) is performed on the input feature map. Assuming that the size of the input feature map is H×W×C, the feature map is compressed to 1×1×C after global average pooling. Then, a one-dimensional convolution of size k is used to model the dependencies between channels, and a convolution operation is performed on the channel information after global average pooling. Compared with the SE module, the ECA module avoids the fully connected layer, thereby significantly reducing the number of parameters. Then, the final channel weight is generated by the nonlinear activation function Sigmoid, and the obtained weight is multiplied by the original feature map by channel, thereby realizing the recalibration of the importance of different channels. In the pixel-level grasping detection task, the introduction of the ECA channel attention mechanism enables the network to more accurately capture the local geometric information and global structural relationship of the object, thereby improving the accuracy and robustness of detection.

[0050] like Figure 6 As shown, according to another embodiment of the present invention, a pixel-level capture detection device 600 is also disclosed, including: a memory 601, used to store program instructions; a processor 602, used to call the program instructions stored in the memory to implement the pixel-level capture detection method as in the above embodiment.

[0051] According to yet another embodiment of the present invention, a computer-readable storage medium is disclosed. The computer-readable storage medium stores program codes. The program codes are used to implement the pixel-level capture detection method as described in the above embodiment.

[0052] According to the above embodiments, the present invention aims at the problem of low accuracy of robot grasping detection of unknown objects in unstructured environments, and proposes a pixel-level grasping detection method that integrates skip connection and attention mechanism: improves the generative residual convolutional neural network model, introduces skip connection and ECA channel attention module to further enhance the feature learning ability of the network; improves the Smooth L1 loss function, and gives different weights to the area where the object is located through weighted masks; experimentally verifies that the improved model has an accuracy of 98.8% and 96.4% on the Cornell and Jacquard datasets, respectively.

[0053] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pixel-level capture detection method, characterized in that: include: Introducing skip connection: Based on generating a residual convolutional neural network, the output features of the convolutional layer of the generated residual convolutional neural network are feature concatenated with the output features of the deconvolutional layer; Introducing an attention mechanism: Introducing an ECA channel attention mechanism in the residual block of the generated residual convolutional neural network, and assigning weights to each channel of the feature map through the ECA module; Improve the loss function: generate a mask matrix, where the mask matrix is ​​a matrix with the same size as the input image, and each element of the mask matrix indicates whether the corresponding pixel belongs to the area where the object is located; Generate a weight matrix, wherein the weight values ​​in the weight matrix correspond one-to-one to the elements of the mask matrix, each of the weight values ​​represents the degree of attention paid to the pixel when calculating the loss, assign a first weight to the mask matrix elements in the area where the object is located, and assign a second weight to the mask matrix elements outside the area where the object is located, wherein the first weight is greater than the second weight; multiply the weight matrix by the calculation result of the Smooth L1 loss function of the generated residual convolutional neural network to obtain an optimized loss function; The objects to be grasped are detected based on the improved generative residual convolutional neural network.

2. The pixel-level capture detection method according to claim 1, characterized in that: Also includes: Train the improved generative residual convolutional neural network; According to the collected image of the object to be grasped, the grasping quality, grasping angle and grasping width are output so that the grasping device can perform the grasping action.

3. The pixel-level capture detection method according to claim 1, characterized in that: The step of generating a mask matrix specifically includes: Create a blank mask matrix of the same size as the input image; Determine the capture area according to the capture rectangle information or the capture annotation information; The mask of the pixels within the captured area is set to 1, and the mask of the pixels outside the captured area is set to 0.

4. The pixel-level capture detection method according to claim 3, characterized in that: For the Cornell dataset, the grasping point is represented by a rectangle, and the coordinates of the four vertices of the rectangle are recorded in the annotation file. For each grasping rectangle, the mask corresponding to the pixels in the rectangle is set to 1, and the other areas are set to 0. For the Jacquard dataset, each grasping point is represented by a center point and a direction angle. The center point and direction angle information are recorded in the annotation file. For each grasping point, a rectangular area is generated according to the center point and the direction angle, and the mask corresponding to the pixels in the rectangular area is set to 1, and the other areas are set to 0.

5. The pixel-level capture detection method according to claim 1, characterized in that: The processing of the ECA module includes: Perform global average pooling on the input feature map; Use one-dimensional convolution to perform convolution operation on the channel information after global average pooling; Generate channel weights through the nonlinear activation function Sigmoid; The channel weight is multiplied by the input feature map channel by channel.

6. The pixel-level capture detection method according to claim 1, characterized in that: The activation function used in the feature extraction process of generating the residual convolutional neural network is the Mish function.

7. The pixel-level capture detection method according to claim 1, characterized in that: The generating residual convolutional neural network specifically includes: Convolutional layer, used to downsample the input image and extract features; The residual block is used to extract features from the feature map output by the convolutional layer; The deconvolution layer is used to upsample the feature map output by the residual block.

8. The pixel-level capture detection method according to claim 7, characterized in that: The input image passes through the three convolutional layers, the five residual blocks and the three deconvolutional layers in sequence to perform feature extraction.

9. A pixel-level capture detection device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the pixel-level capture detection method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code, and the program code is used to implement the pixel-level capture detection method according to any one of claims 1 to 8.

Images