Detection method for transparent object grabbing

Abstract

The invention relates to the technical field of image processing, in particular to a transparent object grabbing-oriented detection method, which comprises the following steps of: constructing a four-stage LFRM module, and taking the output of the LFRM module in the previous stage as one end of the input of the LFRM module in the next stage; in each stage, depth images with different resolutions are adopted as the other input end to perform down-sampling operation; performing channel segmentation on the input feature map by using an LGMA module, performing expansion convolution by using different expansion coefficients, and then performing channel dimension splicing; performing up-sampling operation by using a four-stage CFIUM module to fuse shallow features, and extracting feature information of deep and shallow feature maps in width and height directions; fusing the feature information of the low layer and the deep layer by using an RC module of the capture detection output module; and obtaining a captured width graph, a quality score graph and an angle graph. According to the method, the problem that an existing grabbing detection method is difficult to obtain balance between precision and real-time performance is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a detection method for grasping transparent objects. Background Art

[0002] With the rapid development of intelligent robots, robot grasping technology is widely used in industrial, agricultural and household scenarios. For grasping problems in unstructured environments, it is necessary to generate the grasping configuration of the grasped object from environmental perception, including the required grasping width and the position and posture of the grasped object. Among them, two-dimensional grasping technology is a key issue in robot planar grasping technology. Its core is to obtain the grasping posture of the grasped object from the input sensor data.

[0003] Existing methods for deep restoration of transparent objects based on deep learning, such as the author Douglas Morrison's "Learning robust, real-time, reactive robotic grasping" which uses residual networks to extract features of the original input image, do not take into account the fusion between shallow and deep features, and the obtained effect is suboptimal. In addition, in Y. Wu's "Real-time robotic multigrasp detection using anchor-free fully convolutional grasp detector", although the gradient explosion and gradient vanishing are taken into account through the deep residual module, the deep features are effectively extracted and the accuracy of grasp detection is improved, but the parameters of the model are too redundant, which affects the real-time performance in actual scenes. Summary of the invention

[0004] In view of the shortcomings of existing methods, the present invention designs a lightweight model and method for transparent object grasping detection, which addresses the problem that existing grasping detection methods are difficult to strike a balance between accuracy and real-time performance. The designed multi-stage local feature extraction module extracts RGB-D features hierarchically according to different stages. In addition, a multi-resolution depth image is introduced in each stage, and the geometric characteristics in the depth image are considered to further enhance the feature extraction capability of the model. The designed lightweight global multi-scale attention module fully considers the multi-scale nature of deep features, is plug-and-play, and does not bring the burden of computational parameters. The designed multi-stage collaborative feature fusion upsampling module collaboratively upsamples shallow features with deep features, while considering the width and height dimensions of the features, further enhancing the fitting capability of the model.

[0005] The technical solution adopted by the present invention is: a detection method for grasping a transparent object comprises the following steps:

[0006] Step 1: Collect RGB-D images of transparent objects; construct a four-stage LFRM module, and use the output of the previous stage LFRM module as the input of the next stage LFRM module; each stage uses depth images of different resolutions as the input of the other end of the LFRM module for downsampling;

[0007] As a preferred embodiment of the present invention, step one specifically includes:

[0008] Step 11: Input the RGB image and the D depth image into the first LFRM module to obtain the feature map F local1 ;

[0009] Step 12: The feature map F local1 And D / 2 depth image is input into the second LFRM module to obtain the feature map F local2 ;

[0010] Step 13: The feature map F local2 The D / 4 depth image is input into the third LFRM module to obtain the feature map F local3 ;

[0011] Step 14: The feature map F local3 And D / 8 depth image is input into the fourth LFRM module to obtain the feature map F local4 .

[0012] As a preferred embodiment of the present invention, the LFRM module includes: after one end of the input passes through 3x3 DW convolution, BN layer and Relu activation function, a Concat operation is performed with the other end of the input; after passing through 3x3 convolution, BN layer, Relu activation function, 3x3 convolution and BN layer, a pixel-by-pixel addition operation is performed with the output of the Concat operation; after passing through the Relu activation function, a 1x1 DW convolution, 3x3 PW convolution, BN layer and Relu activation function are input.

[0013] Step 2: Input the output feature map of the fourth-stage LFRM module into the LGMA module to obtain the global deep RGB-D feature map. Use the LGMA module to perform channel segmentation on the input feature map, and then use dilated convolution with different dilation coefficients to perform channel dimension splicing, and then use DW convolution to calculate the feature weight information;

[0014] As a preferred embodiment of the present invention, the formula of the LGMA module is:

[0015]

[0016] Among them, x1, x2, x3, x4 are channel dimension segmentation features; channel_split is the channel segmentation function; dilation_conv() is the dilation convolution function; y1, y2, y3, y4 are the dilation convolution output values; Q, K, V are the query matrix, key matrix, and value matrix respectively; F local4 Output feature map for the fourth LFRM module; F global It is the output feature map of the LGMA module.

[0017] Step 3: Input the output feature map of the LGMA module and the output feature map of the four-stage LFRM module into the four-stage CFIUM module for upsampling operation to fuse shallow features, and adopt the global average pooling and global maximum pooling strategies in width and height dimensions to extract the feature information of deep and shallow feature maps in both width and height directions;

[0018] As a preferred embodiment of the present invention, step three specifically includes:

[0019] Step 31: Feature map F global and feature map F local4 Input the first CFIUM module and output the feature map F fusioinl ;

[0020] Step 32: Feature map F fusioinl and feature map F local3 Input the second CFIUM module and output the feature map F fusioin2 ;

[0021] Step 33: Feature map F fusioin2 and feature map F local2 Input the third CFIUM module and output the feature map F fusioin3 ; Step 34, feature map F fusioin3 and feature map F local1 Input the fourth CFIUM module and output the feature map F fusioin4 As a preferred embodiment of the present invention, the CFIUM module includes:

[0022] Input one end feature map F high After the upsampling factor of 2, the transposed convolution Conv2T, the BN layer and the Relu activation function, the feature map Input1 is obtained. low ;

[0023] Input the feature map F at the other end low After a 3×3 convolution, we get the feature map Input2 low ;

[0024] Feature map Input1 lowInput parallel global submodules to obtain feature maps Input1 global_w 、Input1 global_h 、Input1 max_w and Input1 max_h ;

[0025] Input2 low Input parallel global submodules to obtain feature maps Input2 global_w 、Input2 global_h 、Input2 max_w and Input2 max_h ;

[0026] Input1 global_w With Input1 global_h Perform matrix multiplication to obtain feature map Input1 global , the feature map Input2 global_w With Input2 global_h Perform matrix multiplication to obtain feature map Input2 global ; Input1 max_w With Input1 max_h Perform matrix multiplication to obtain feature map Input1 max , the feature map Input2 max_w With Input2 max_h Perform matrix multiplication to obtain feature map Input2 max ;

[0027] Input1 global 、Input2 global 、Input1 max and Input2 max Concat splicing is performed according to the channel dimension; then a fully connected layer, a Dropout layer and a Sigmoid activation function are passed to obtain the global attention coefficient Atttention;

[0028] Input1 low After a 3×3 convolution, we get the feature map Input1 low_2 ; Combine the global attention coefficient Atttention with Input1 low_2 Multiply pixel by pixel to get the feature map Input1 output ;

[0029] Combine the global attention coefficient Atttention with the feature map Input2 low Multiply to get feature map Input2 output ;

[0030] Input1 output 、Input2 output 、Input1 low_2 and Input2 low Add pixel by pixel to get the feature map F fusion .

[0031] As a preferred embodiment of the present invention, the parallel global sub-module includes: 1 global average pooling performed only on the width dimension, 1 global average pooling performed only on the height dimension, 1 global maximum pooling performed only on the width dimension, and 1 global maximum pooling performed only on the height dimension.

[0032] Step 4: Input the output feature map of the first LFRM module and the output feature map of the fourth CFIUM module into the grasping detection output module, and use the RC module of the grasping detection output module to fuse the low-level and deep-level feature information; obtain the grasping width map, grasping quality score map and grasping angle map respectively;

[0033] As a preferred implementation of the present invention, the RC module includes: one end of the feature map input is fed into a Conv2D convolution, a BN layer and a Relu activation function, and then a channel dimension splicing operation is performed with the other end of the feature map input.

[0034] As a preferred embodiment of the present invention, the grasping detection output module includes: the output feature map of the RC module is sent to a Conv2T convolution, a BN layer and a Relu activation function, followed by four parallel Dropout functions and a 1×1 convolution to obtain a grasping quality score map I quality , Cosine graph of grab angle I angle_cos And the sine graph I angle_sin and Crawl Width Graph I width .

[0035] As a preferred embodiment of the present invention, the calculation formula of the grab angle diagram is:

[0036]

[0037] Among them, I angle_sin The sinusoidal graph of the grasping angle predicted by the model, I angle_cos Cosine graph showing the grab angle.

[0038] Beneficial effects of the present invention:

[0039] 1. Compared with only accepting the output of the previous layer of network structure as input, the multi-stage local feature extraction module of the present invention also combines multi-resolution depth images at different stages, fully considering the geometric properties of the depth image, and can effectively improve the accuracy of grasping detection;

[0040] 2. After the lightweight global multi-scale attention module of the present invention performs channel segmentation on the input feature map, it performs channel dimension splicing after dilated convolution with different dilation coefficients. Such an operation can fully consider the multi-scale feature information of the feature map, and then use DW convolution to replace the fully connected layer of many existing methods to calculate the weight information of the feature, which can ensure that the global feature information is obtained while reducing the parameters of the module;

[0041] 3. The multi-stage collaborative feature fusion upsampling module of the present invention takes shallow features into account while upsampling deep features, and through the designed global average pooling and global maximum pooling strategies in width and height dimensions, it can well extract feature information of deep and shallow feature maps in both width and height directions, which helps to improve the robustness of the model;

[0042] 4. Grasp detection output module Taking into account that as the convolutional neural network structure deepens, the extracted features are often accompanied by deep semantic feature information from low-level edge, color, texture and other features. First, the low-level feature information is fully considered to perform an RC module on the deep features, which helps to improve the accuracy of grasp detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a logic diagram of a lightweight model and method for transparent object grasping and detection of the present invention;

[0044] Figure 2 It is a structural diagram of a multi-stage local feature extraction module of the present invention;

[0045] Figure 3 It is a structural diagram of the lightweight global multi-scale attention module of the present invention;

[0046] Figure 4 It is a structural diagram of a multi-stage collaborative feature fusion upsampling module of the present invention;

[0047] Figure 5 It is the structure diagram of the grab detection output module;

[0048] Figure 6 This is a visualization of the experimental results of grasping and detection in real scenes. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0050] like Figure 1 As shown, a detection method for grasping a transparent object includes the following steps:

[0051] Step 1: Use four LFRMs (Local feature refinement modules) to take the original RGB and multi-resolution depth image D as input to obtain the local RGB-D features corresponding to the four stages [F local1 ,F local2 ,F local3 ,F local4 ];

[0052] First, the original RGB∈R collected by Intel’s D435i depth camera 224×224×3 And the depth image D∈R 224 ×224 Input the output feature F in the LFRM1 module of the first stage local1 ∈R 112×112×32 ;

[0053] like Figure 2 As shown in the figure, the processing process of the LFRM1 module includes: after RGB passes through 3x3 DW convolution, BN layer and Relu activation function, it performs channel Concat operation with D; then passes through 3x3 convolution, BN layer, Relu activation function, 3x3 convolution and BN layer one by one, and then performs pixel-by-pixel addition operation with the output of Concat operation; then passes through Relu activation function, 1x1 DW convolution, 3x3 PW convolution, BN layer and Relu activation function one by one.

[0054] The second to fourth stage processes are similar to the first stage processing, and the second to fourth stages LFRM2 module to LFRM4 module are described below.

[0055] The LFRM2 module includes: bilinear interpolation of D to downsample the resolution by 1 / 2 and output feature F with the LFRM1 module local1 Input LFRM2 module, output feature F local2 ∈R 56×56×64 .

[0056] The LFRM3 module includes: bilinear interpolation of D to downsample the resolution by 1 / 4 and combine it with the feature F local2 Input LFRM3 module, output feature F local3 ∈R 28×28×128 .

[0057] The LFRM4 module includes: bilinear interpolation of D to downsample the resolution by 1 / 8 and combine it with the feature F local3 Input into the fourth stage LFRM4, output feature F local4 ∈R 14×14×256 .

[0058] That is, in the first stage, input 1 is the original RGB image, input 2 is the original depth image D, and the output feature F local1 ; In the second stage, input 1 is feature F local1 , input 2 is the depth image obtained by downsampling D by 1 / 2, and the output is feature F local2 ; In the third stage, input 1 is feature F local2 , input 2 is the depth image obtained after D is downsampled to 1 / 4, and the output is feature F local3 ; In the fourth stage, input 1 is feature F local3 , input 2 is the depth image obtained after D is downsampled to 1 / 8, and the output is feature F local4 , the overall process from LFRM1 module to LFRM4 module is as follows:

[0059]

[0060] Among them, D downsample,p Indicates that the depth image D of the p-th stage undergoes resolution downsampling operations, which are 1 / 2, 4 / 1, and 8 / 1 respectively.

[0061] Step 2: Construct the LGMA module (Lightweight global multi-scale attention module) to transform the feature F local4 Input into the LGMA module to obtain the global deep RGB-D feature F global ;

[0062] like Figure 3 As shown, F local4 Input the LGMA module to obtain the global deep RGB-D feature F global ; Specifically: First, F local4 According to the channel dimension, it is divided into four features [x1,x2,x3,x4]∈R 14×14×64 , using expansion coefficients of 4, 6, 8, and 12 for [x1,x2,x3,x4]∈R 14×14×64 Perform dilated convolution to obtain [y1,y2,y3,y4]∈R 14×14×64 ; Then, concatenate the features [y1, y2, y3, y4] to obtain a feature map with a dimension of 14×14×256; and then use 3x3 DW convolution to obtain Q∈R 7×7×256 , adjust the dimension of Q to 49×num_heds×(256 / num_heds), where num_heds represents the number of heads in the multi-head attention module. In this embodiment, num_heds is set to 8, that is, the dimension of Q is: 49×8×32, and then adjust the dimension order to: 8×49×32; then Flocal4 After a 3x3 DW convolution, K∈R 7×7×256 and V∈R 7×7×256 ; Adjust K and V to 49×num_heds×(256 / num_heds), K is adjusted to 8×32×49, V is adjusted to 8×49×32; Finally, Q∈R 8×49×32 With K∈R 8×32×49 After the matrix multiplication operation, multiply by the scaling factor, the scaling factor is the number of channels of the feature map / num_heds and then take the square root, in this embodiment, that is: 256 / 8 and then take the square root; then, after the softmax activation function and the DropOut function, the dimension is: 8×49×49 feature map, which is then multiplied by V∈R 8×49×32 After matrix multiplication, we get F global ∈R 8×49×32 , and then adjust the dimension to get the final F global ∈R 7×7×256 .

[0063] The flow expression of the LGMA module is as follows:

[0064]

[0065] Step 3: Design the CFIUM module (Collaborative Feature Integration Upsampling Module) to transform the feature [F local1 ,F local2 ,F local3 ,F local4 ] and feature F global As input to four CFIUM modules, we obtain local-global efficient fusion features [F fusion1 ,F fusion2 ,F fusion3 ,F fusion4 ]; four CFIUM modules, namely CFIUM1, CFIUM2, CFIUM3, and CFIUM4.

[0066] like Figure 4 As shown in Figure 1, the CFIUM module has two inputs. Input 1 first passes through the transposed convolution Conv2T with an upsampling factor of 2, the BN layer and the Relu activation function to obtain the shallow feature map Input1 of input 1. low , after a 3×3 convolution, input 2 obtains the shallow feature map Input2 of input 2 low , Input1 lowAt the same time, the parallel global submodule is input. The parallel global submodule includes: 1 global average pooling for only the width dimension, 1 global average pooling for only the height dimension, 1 global maximum pooling for only the width dimension, and 1 global maximum pooling for only the height dimension, and the feature map Input1 is obtained respectively. global_w 、Input1 global_h 、Input1 max_w and Input1 max_h , Input2 low After the parallel global submodule, we get the feature map Input2 global_w 、Input2 global_h 、Input2 max_w and Input2 max_h , and then the feature map Input1 global_w With the feature map Input1 global_h Perform matrix multiplication to obtain feature map Input1 global , feature map Input2 global_w With feature map Input2 global_h Perform matrix multiplication to obtain feature map Input2 global , feature map Input1 max_w With the feature map Input1 max_h Perform matrix multiplication to obtain feature map Input1 max , feature map Input2 max_w With feature map Input2 max_h Perform matrix multiplication to obtain feature map Input2 max ; Then Input1 global 、Input2 global 、Input1 max and Input2 max After concat concatenation according to the channel dimension, it passes through a fully connected layer, a Dropout layer, and a Sigmoid activation function to obtain the global attention coefficient Atttention; then, the feature map Input1 low After a 3×3 convolution, we get Input1 low_2 , Attention and Input1 low_2 Multiply pixel by pixel to get Input1 output , Attention and Input2 low After multiplication, we get the feature map Input2 output ;Finally, the feature map Input1 output 、Input2 output 、Input1 low_2and Input2 low And perform pixel-by-pixel addition to output the feature map.

[0067] CFIUM2 to CFIUM4 modules use the same structure; the input 1 of CFIUM1 is the feature map F global , input 2 is feature map F local4 , the output is the feature map F fusioinl ; Input 1 of CFIUM2 module is the output feature map F of CFIUM1 module fusioinl , input 2 is feature map F local3 , the output is the feature map F fusioin2 ; Input 1 of CFIUM3 module is feature map F fusioin2 , input 2 is feature map F local2 , the output is the feature map F fusioin3 ; Input 1 of CFIUM4 module is the feature map F of CFIUM3 module fusioin3 , input 2 is feature map F local1 , the output is the feature map F fusioin4 .

[0068] Specific example: The feature map [F local1 ,F local2 ,F local3 ,F local4 ] and feature map F global As input, we get the local-global efficient fusion feature map [F fusion1 ,F fusion2 ,F fusion3 ,F fusion4 ]; the feature map F global ∈R 7×7×256 And the feature map F local4 ∈R 14×14×256 Input CFIUM1 module output feature map F fusion1 ∈R 14×14×256 ; Feature map F global ∈R 7×7×256 After the upsampling factor of 2, the transposed convolution Conv2T, the BN layer and the Relu activation function, the feature map Input1 is obtained low ∈R 14×14×256 , feature map F local4 ∈R 14×14×256 After a 3×3 convolution, we get the feature map Input2 low ∈R 14×14×256 , then Input1 lowAt the same time, after one global average pooling only on the width dimension, one global average pooling only on the height dimension, one global maximum pooling only on the width dimension, and one global maximum pooling only on the height dimension, Input1 is obtained respectively. global_w ∈R 1×14×256 、Input1 global_h ∈R 14×1×256 、Input1 max_w ∈R 1×14×256 and Input1 max_h ∈R 14×1×256 , Input2 low Also through the parallel global submodule, we get Input2 global_w ∈R 1×14×256 、Input2 global_h ∈R 14×1×256 、Input2 max_w ∈R 1×14×256 and Input2 max_h ∈R 14 ×1×256 , then Input1 global_w With Input1 global_h Perform matrix multiplication to get Input1 global ∈R 14×14×256 , Input2 global_w With Input2 global_h Perform matrix multiplication to get Input2 global ∈R 14×14×256 , Input1 max_w With Input1 max_h Perform matrix multiplication to get Input1 max ∈R 14×14×256 , Input2 max_w With Input2 max_h Perform matrix multiplication to get Input2 max ∈R 14×14×256 ; Next, Input1 global 、Input2 global 、Input1 max and Input2 max After concatenation according to the channel dimension, the dimension is: 14×14×2048. After 1 fully connected layer, Dropout layer and Sigmoid activation function, the global attention coefficient Atttention∈R is obtained. 14×14×256 ; Then, Input1 low After a 3×3 convolution, we get Input1 low_2 ∈R 14×14×256, Attention and Input1 low_2 Multiply pixel by pixel to get Input1 output ∈R 14×14×256 , Attention and Input2 low After multiplication, it is recorded as Input2 output ∈R 14×14×256 ; Finally, Input1 output 、Input2 output 、Input1 low_2 and Input2 low Add pixel by pixel to get the final output F of stage one fusion1 ∈R 14×14×256 Same as CFIUM1 module, proceed to CFIUM2 module to CFIUM4 module, and change F fusion1 ∈R 14×14×256 With F local3 ∈R 28×28×128 Input CFIUM2 module and get output F fusion2 ∈R 28×28×128 Then, F fusion2 ∈R 28×28×128 With F local2 ∈R 56×56×64 Input CFIUM3 module and get output F fusion3 ∈R 56×56×64 Finally, F fusion3 ∈R 56×56×64 With F local1 ∈R 112×112×32 Input CFIUM4 module and get output F fusion4 ∈R 112×112×32 .

[0069] Step 4: The feature map F local1 and feature map F fusion4 As input, the detection output module OutputHead is used to obtain the grasp quality score graph I quality 、Grab Angle Diagram I angle , and grab width graph I width , and generate the final grasping detection map I based on these three maps pred ;

[0070] like Figure 5 As shown in the figure, the grasp detection output module first converts the feature map F local1 and feature map F fusion4 As the input of the RC module (raw compensation, original compensation submodule), the RC module converts F local1 Input 1 Conv2D convolution, BN layer and Relu activation function and F fusioin4Perform splicing operations in the channel dimension;

[0071] The output feature map of the RC module is input into a Conv2T convolution, a BN layer and a Relu activation function; followed by 4 parallel Dropout functions and 1×1 convolution, and the captured quality score map I is directly output. quality , Cosine graph of grab angle I angle_cos And the sine graph I angle_sin and Crawl Width Graph I width ; The random deactivation rate of the Dropout function is set to 0.1; The final grasping angle Figure I angle Obtained by the following formula:

[0072]

[0073] Among them, I angle_sin The sinusoidal graph of the grasping angle predicted by the model, I angle_cos Cosine graph showing the grab angle.

[0074] Then, for I quality ,I angle and I width After Gaussian blur processing, select I quality The point with the largest pixel value is taken as the optimal grasping point. The image coordinate system position of the optimal grasping point is recorded as: (m,n). Select I angle The pixel value at position (m,n) is the optimal rotation angle angle, and I width The pixel value at the position (m, n) is the optimal grasping width. Finally, a grasping detection map is generated in the original RGB image based on the center point position (m, n), the rotation angle angle of the grasping rectangle, and the width width of the grasping rectangle.

[0075] Experimental configuration environment:

[0076] The operating system of the experiment is Windows 11, the CPU is 13th Gen Intel (R) Core (TM) i9-13900x32, the graphics card is 1 NVIDIA GeForce RTX4090, and the memory is 24G.

[0077] Experimental content:

[0078] The grasping detection dataset of the present invention is the Cornell and Jacquard dataset. At the same time, in order to fully verify the effectiveness of the present invention, it is compared with existing deep learning-based methods, including GG-CNN in "Learning robust, real-time, reactive robotic grasping" by Douglas Morrison; GR-ConvNet in "Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network" by Sulabh Kumra; grasp-transformer in "When Transformer Meets Robotic Grasping: Exploits Context for Efficient Grasp Detection" by Shaochen Wang; ResNet-50 in "Real-time robotic multigrasp detection using anchor-freefully convolutional grasp detector" by Y. Wu. All comparison methods use the setting parameters of the original authors.

[0079] The present invention adopts some common evaluation criteria in grasping detection tasks, including:

[0080] 1. The rotation angle difference between the predicted grab box and the true grab box should be less than 30 degrees, where the formula is as follows:

[0081] |θ-θ GT |<30° (4)

[0082] Among them, θ is the angle between the predicted grab box and the positive direction of the x-axis in the pixel coordinate system, θ GT It is the angle between the real grab box and the positive direction of the horizontal x-axis in the pixel coordinate system.

[0083] 2. The Jaccard coefficient between the prediction and the true value is greater than 25%, as shown in the formula:

[0084]

[0085] The Jaccard coefficient is used to measure the similarity between the real grab box and the grab box predicted by the model. The larger the Jaccard coefficient, the more accurate the prediction of the grab detection model.

[0086] Table 1 Experimental results on the Cornell dataset

[0087]

[0088] Table 2 Experimental results on the Jacquard dataset

[0089] method Modality Accuracy(%) GG-CNN D 84 GR-ConvNet RGB-D 95.1 ResNet-50 RGB-D 95.7 The present invention RGB-D 96.3

[0090] It can be seen from Table 1 and Table 2 that the present invention is optimal in all measurement indicators. Even if the parameter amount of the present invention is not optimal, it is considered that in actual application scenarios, such parameter amount is acceptable while ensuring accuracy. In addition, Figure 6 The figure shows a grasping quality score graph, a grasping angle graph, and a grasping width graph obtained by grasping and detecting a common transparent bottle in life in a real scene, and a final grasping detection graph is generated based on these three graphs; the visualization results show that the present invention is effective.

[0091] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A detection method for grasping transparent objects, characterized in that: The following steps are involved: Step 1: Collect RGB-D images of transparent objects; construct a four-stage LFRM module, and use the output of the previous stage LFRM module as the input of the next stage LFRM module; each stage uses depth images of different resolutions as the input of the LFRM module for downsampling; Step 2: Input the output feature map of the fourth-stage LFRM module into the LGMA module to obtain the global deep RGB-D feature map; use the LGMA module to perform channel segmentation on the input feature map, and use dilated convolution with different dilation coefficients to perform channel dimension splicing, and then use DW convolution to calculate the feature weight information; Step 3: Input the output feature map of the LGMA module and the output feature map of the four-stage LFRM module into the four-stage CFIUM module for upsampling operation to fuse shallow features, and adopt the global average pooling and global maximum pooling strategies in width and height dimensions to extract the feature information of deep and shallow feature maps in both width and height directions; Step 4: Input the output feature map of the first LFRM module and the output feature map of the fourth CFIUM module into the grasping detection output module, and use the RC module of the grasping detection output module to fuse the low-level and deep-level feature information; obtain the grasping width map, grasping quality score map and grasping angle map respectively.

2. The detection method for grasping transparent objects according to claim 1, characterized in that: Step 1 specifically includes: Step 11: Input the RGB image and the D depth image into the first LFRM module to obtain the feature map F local1 ; Step 12: The feature map F local1 And D / 2 depth image is input into the second LFRM module to obtain the feature map F local2 ; Step 13: The feature map F local2 The D / 4 depth image is input into the third LFRM module to obtain the feature map F local3 ; Step 14: The feature map F local3 And D / 8 depth image is input into the fourth LFRM module to obtain the feature map F local4 .

3. The detection method for grasping transparent objects according to claim 2, characterized in that: The LFRM module includes: input feature map F rgb After 3x3DW convolution, BN layer and Relu activation function, it is combined with the feature map F at the other end of the input d After the Concat operation, after 3x3 convolution, BN layer, Relu activation function, 3x3 convolution and BN layer, pixel-by-pixel addition operation is performed with the output of the Concat operation; after Relu activation function, 1x1DW convolution, 3x3PW convolution, BN layer and Relu activation function are input.

4. The detection method for grasping transparent objects according to claim 3, characterized in that: The formula for the LGMA module is: Among them, x1, x2, x3, x4 are channel dimension segmentation features; channel_split is the channel segmentation function; dilation_conv() is the dilation convolution function; y1, y2, y3, y4 are the dilation convolution output values; Q, K, V are the query matrix, key matrix, and value matrix respectively; F local4 Output feature map for the fourth LFRM module; F global It is the output feature map of the LGMA module.

5. The detection method for grasping transparent objects according to claim 1, characterized in that: Step three specifically includes: Step 31: Feature map F global and feature map F local4 Input the first CFIUM module and output the feature map F fusioinl ; Step 32: Feature map F fusioinl and feature map F local3 Input the second CFIUM module and output the feature map F fusioin2 ; Step 33: Feature map F fusioin2 and feature map F local2 Input the third CFIUM module and output the feature map F fusioin3 ; Step 34: Feature map F fusioin3 and feature map F local1 Input the fourth CFIUM module and output the feature map F fusioin4 .

6. The detection method for grasping transparent objects according to claim 5, characterized in that: CFIUM modules include: Input one end feature map F high After the upsampling factor of 2, the transposed convolution Conv2T, the BN layer and the Relu activation function, the feature map Input1 is obtained low ; Input the feature map F at the other end low After a 3×3 convolution, we get the feature map Input2 low ; Feature map Input1 low Input parallel global submodules to obtain feature maps Input1 global_w , Input1 global_h 、Input1 max_w and Input1 max_h ; Input2 low Input parallel global submodules to obtain feature maps Input2 global_w 、Input2 global_h , Input2 max_w and Input2 max_h ; Input1 global_w With Input1 global_h Perform matrix multiplication to obtain feature map Input1 global , the feature map Input2 global_w With Input2 global_h Perform matrix multiplication to obtain feature map Input2 global ; Input1 max_w With Input1 max_h Perform matrix multiplication to obtain feature map Input1 max , the feature map Input2 max_w With Input2 max_h Perform matrix multiplication to obtain feature map Input2 max ; Input1 global 、Input2 global 、Input1 max and Input2 max Concat splicing is performed according to the channel dimension; then a fully connected layer, a Dropout layer and a Sigmoid activation function are passed to obtain the global attention coefficient Atttention; Input1 low After a 3×3 convolution, we get the feature map Input1 low_2 ; Combine the global attention coefficient Atttention with Input1 low_2 Multiply pixel by pixel to get the feature map Input1 output ; Combine the global attention coefficient Atttention with the feature map Input2 low Multiply to get feature map Input2 output ; Input1 output 、Input2 output 、Input1 low_2 and Input2 low Add pixel by pixel to get the feature map F fusion .

7. The detection method for grasping transparent objects according to claim 6, characterized in that: The parallel global submodules include: 1 global average pooling for width dimension only, 1 global average pooling for height dimension only, 1 global maximum pooling for width dimension only, and 1 global maximum pooling for height dimension only.

8. The detection method for grasping transparent objects according to claim 1, characterized in that: The grasp detection output module includes: the output feature map of the RC module is fed into a Conv2T convolution, a BN layer and a Relu activation function, and then input into four parallel Dropout functions and 1×1 convolution to obtain the grasp quality score map I quality , Cosine graph of grab angle I angle_cos And the sine graph I angle_sin and Crawl Width Graph I width .

9. The detection method for grasping transparent objects according to claim 8, characterized in that: The RC module includes: one end of the feature map input is sent to a Conv2D convolution, BN layer and Relu activation function, and then the channel dimension is spliced ​​with the other end of the feature map input.

10. The detection method for grasping transparent objects according to claim 8, characterized in that: The calculation formula for grabbing the angle graph is: Among them, I angle_sin The sinusoidal graph of the grasping angle predicted by the model, I angle_cos Cosine graph showing the grab angle.

Images