Robot-based grabbing detection method

Abstract

The invention relates to the technical field of image processing, in particular to a robot-based grabbing detection method, which comprises the following steps: acquiring an RGBD image; constructing an encoder of a grabbing detection model, and sequentially performing feature extraction by using the first PRCBlock, the second PRCBlock, the third PRCBlock and the fourth PRCBlock to obtain features F1, F2, F3 and F4; respectively obtaining characteristics F123, XHigh, XLow, XLH and F5; constructing a decoder of the capture detection model, and decoding the output characteristics of the encoder by using the decoder; performing feature extraction on the output features of the decoder to respectively obtain a grabbing quality graph, a grabbing angle graph and a grabbing width graph; and optimizing the capture detection model by using the loss function. The method solves the problems that an existing grabbing detection network cannot fully extract feature information, and important grabbing area features are prone to being neglected and more likely to be lost, so that the grabbing detection accuracy is reduced; and the existing method cannot fully fuse low-level feature information and high-level feature information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a robot-based grasping detection method. Background Art

[0002] In recent years, the rapid development of robotics has had a huge impact on various fields such as industry, medicine, agriculture and education. In the industrial field, the widespread use of robots has improved production efficiency and reduced production costs, while also reducing the labor intensity of workers and improving the working environment. Grasping, as a key operation in robotics, has aroused widespread research interest. In industrial scenarios, unordered sorting tasks account for a considerable proportion. Unlike objects in daily environments, industrial parts often have highly reflective and textureless features, which poses a challenge to the accuracy of grasping tasks. At present, there are two main types of grasping detection methods based on deep learning, one is the generative grasping strategy, and the other is the discriminative grasping estimation strategy. The current grasping detection methods still have many challenges in feature fusion and attention mechanisms, and fail to fully extract feature information and fuse feature information at multiple scales and levels. This has low success in complex environments with variable sizes and unknown objects.

[0003] Wang et al. combined Transformer with encoder and decoder to propose TF-Grasp. Although it achieved good results, it was weak in processing the edges of objects due to the lack of global and local information.

[0004] Zhang et al. proposed the Double Strand Robotic Grasp Detection Network Based on Cross Attention, which combines the transformer branch and the U-Net branch in the encoder-decoder structure, but fails to effectively utilize the multi-scale receptive field, reducing the feature extraction capability of multiple grasp targets.

[0005] The Efficient Grasp Detection Network With Gaussian-BasedGrasp Representation for Robotic Manipulation proposed by Cao et al. simply utilizes the fusion of channel and pixel attention, and does not make good use of contextual information to obtain local feature information; and does not utilize the self-attention mechanism, which can effectively connect contextual information. Summary of the invention

[0006] In view of the shortcomings of the existing methods, the present invention solves the problems that the existing grasping detection network cannot fully extract feature information, easily ignores or is more likely to lose important grasping area features, resulting in reduced grasping detection accuracy; and the existing methods cannot fully integrate low-level features and high-level feature information, resulting in the robot or mechanical arm being unable to quickly grasp the target object.

[0007] The technical solution adopted by the present invention is: a robot-based grasping detection method comprises the following steps:

[0008] Step 1: Obtain the RGBD image of the object to be grasped by the robot and build an RGBD image dataset;

[0009] Step 2: Construct an encoder for the grasp detection model, and use the first to fourth PRCBlocks to extract features in sequence to obtain features F1, F2, F3, and F4; downsample F1, upsample F3, and then combine with F2 to obtain feature F 123 ; Upsample F4 and then F 123 Downsample, concatenate with F3 and input into the first MAFM to get feature X High ; F 123 Input the second MAFM to get the feature X Low , X High Upsampling, X Low After downsampling, input IFM to obtain feature X LH ; X LH Downsample, input F4 into WREM, and then add it to F4 to get feature F5;

[0010] As a preferred embodiment of the present invention, PRCBlock includes: Feature X pr in Input 1x1 convolution to get feature X pr 1. X pr 1 Input PCconv to get feature X pr 11 , X pr 1 Input 3x3DW convolution, LayerNorm to get feature X pr 12 , X pr 11 With X pr 12 Add to get feature X pr 2. X pr 2 After 1x1 convolution, GELU, 1x1 convolution and X pr 1 Perform residual connection; then output feature X through BatchNorm pr out .

[0011] As a preferred embodiment of the present invention, WREM comprises: Feature X wr in After CCEM, we get feature X ccw , X ccw Get feature X through SDEM sdw ;X wr in After WSEM, we get feature X ws , X sdw Through 1x1 convolution, and X ws After concatenation, the feature X is output through 1x1 convolution wr out .

[0012] As a preferred embodiment of the present invention, WSEM includes three branches. The first branch converts the feature X wr in Input Linear to get feature Q; the second branch converts X wr in Input Linear and WTM to get feature F, input F to Linear to get feature V, input F to IWTM and Linear to get feature K, multiply Q and K, pass through the softmax function, multiply with V, pass through Linear and multiply with X wr in Perform addition operation and output X ws .

[0013] As a preferred embodiment of the present invention, WTM includes 1x1 convolution, DWT and 3x3 convolution, each convolution is connected to BN layer and RELU; and is residually connected with average pooling and then undergoes PCconv convolution; IWTM includes: DWConv, IDWT are connected and then residually connected with average pooling, and then undergoes PCconv convolution.

[0014] As a preferred embodiment of the present invention, CCEM comprises: Features X wr in After inputting into the GN layer, it is weighted, and then passes through the sigmoid function and gated by the threshold T to obtain the information weight W1 and the non-information weight W2 respectively. W1 and W2 are respectively related to X wr in Multiply them together to get feature X ccw 1 and feature X ccw 2. For X ccw 1 and X ccw 2 After cross reconstruction and cross phase difference operation, splicing is performed to obtain feature X ccw1 and feature X ccw2 , X ccw1 and X ccw2Add together to get feature X ccw .

[0015] As a preferred embodiment of the present invention, SDEM includes: using the segmentation weight α to segment the feature X ccw The number of channels C is divided, and then three 1x1 convolutions are input respectively to obtain the feature X sd up , Feature X sd low and feature F sd 3. X sd up The feature F is obtained by adding the GWC convolution and 1x1 convolution respectively. sd 1. X sd low Through 1x1 convolution and combined with X sd low Perform residual splicing to output feature F sd 2; F sd 3 respectively with F sd 1. F sd 2 and then concatenate to get feature F sd ; for F sd Perform global average pooling and Softmax operations and combine with F sd Multiply them and split them to get feature weight vectors β1 and β2. sd 1 multiplied, β2 and F sd 2, and then add to get feature X sdw .

[0016] As a preferred embodiment of the present invention, the first branch convolution kernel of MAFM is 1x1, the second branch convolution is 1x1, 3x1, 1x3 convolution kernel series, the third branch convolution is 3x3, 5x1, 1x5 convolution kernel series, the fourth branch convolution is 5x5, 7x1, 1x7 convolution kernel series, the features extracted by the four branches are spliced, and the residual connection is performed with the input feature to output the feature X MA out .

[0017] As a preferred embodiment of the present invention, IFM converts the feature X Low and feature X High Concatenate to get feature X HL , X HL After passing through the average pooling layer and the maximum pooling layer, the sum is added to X HL Point multiplication, and then through 1x1 convolution and BN layer to get feature X global ; X HL After two 1x1 convolutions, the feature X is obtained through softmax. local, two 1x1 convolutions are connected to the BN layer; X HL Respectively with X local , X global Multiply and add X HL Add, and finally get feature X through 1x1 convolution IF out .

[0018] Step 3: Build a decoder for the capture detection model and use the decoder to decode the output features of the encoder;

[0019] As a preferred embodiment of the present invention, the decoder includes: the feature F5 is input into the first ConVT to obtain the feature F6, the feature F6 is combined with the feature X LH Add together to get feature F 1 6. Feature F 1 6 Input the second ConVT to get feature F7, feature F7 and feature X LH After upsampling, add the input to the third ConVT to obtain feature F8 and feature X LH After upsampling, add it to feature F8 to get feature F 1 8. Feature F 1 8 Input the fourth ConVT to get feature X out .

[0020] Step 4: Extract the features of the decoder output to obtain a grasping quality map, a grasping angle map, and a grasping width map;

[0021] Step 5: Use the loss function to optimize the grasping detection model.

[0022] Beneficial effects of the present invention:

[0023] 1. The present invention designs a new downsampling module, which is simple and has few parameters, and can effectively extract rich semantic feature information;

[0024] 2. The present invention designs an enhancement module to enhance target feature information while suppressing redundant features in space and channels; the designed wavelet self-attention enhancement module improves the model's ability to capture important information through the combination of wavelet transform and self-attention mechanism;

[0025] 3. The fusion module designed in the present invention captures feature information and context information at different scales, effectively mining shallow and deep features; the shallow information and deep information interactive fusion module further integrates these features to obtain more target feature information;

[0026] 4. The present invention achieves accuracies of 99.1% and 95.7% on the publicly available crawled datasets Cornell and Jacquard, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall structure diagram of the crawling detection network model of the present invention;

[0028] Figure 2 It is the structural diagram of PRCBlock of the present invention;

[0029] Figure 3 This is the overall structure diagram of the WREM of the present invention;

[0030] Figure 4 This is a CCEM structure diagram of the present invention;

[0031] Figure 5 This is the SDEM structure diagram of the present invention;

[0032] Figure 6 This is a structural diagram of the MAFM of the present invention;

[0033] Figure 7 This is the IFM structure diagram of the present invention;

[0034] Figure 8 The detection graph is for grasp detection. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0036] A robot-based grasping detection method comprises the following steps:

[0037] Step 1: First, obtain the RGBD image of the object to be grasped by the robot and build an RGBD image dataset;

[0038] The dataset can be Cornell dataset, Jacquard dataset or self-built dataset; the images are uniformly preprocessed into 224x224 size;

[0039] like Figure 1 As shown, step 2 is to build a grasping detection model; the grasping detection model includes an encoder and a decoder.

[0040] Encoders include: PRCBlock, WREM, MAFM and IFM;

[0041] The RGBD image of the object to be grasped is subjected to four feature extractions through the first to fourth PRCBlocks to obtain features F1, F2, F3, and F4; feature F1 is downsampled, and feature F3 is upsampled and concatenated with feature F2 to obtain feature F4. 123 ; Upsample feature F4 and then upsample feature F 123Downsample, concatenate with feature F3 and input into the first MAFM to obtain feature X High ; Feature F 123 Input the second MAFM to get the feature X Low ; Set feature X High Upsampling, feature X Low After downsampling, the IFM output feature X is input together LH ; Set feature X LH Downsampling, feature F4 is sent to WREM and then combined with features F4 and X LH The downsampled features are added to obtain feature F5.

[0042] Step 3: Construct a decoder for the capture detection model. The decoder includes four ConVTs. Feature F5 is input into the first ConVT to obtain feature F6. Feature F6 is combined with feature X. LH Add together to get feature F 1 6. Feature F 1 6 Input the second ConVT to get feature F7, feature F7 and feature X LH After upsampling, add the input to the third ConVT to obtain feature F8 and feature X LH After upsampling, add it to feature F8 to get feature F 1 8. Feature F 1 8 Input the fourth ConVT to get feature X out ; Among them, the first to third ConVT are 3 4x4 deconvolutions; the fourth ConVT is 9x9 deconvolution.

[0043] like Figure 2 The figure shows the schematic diagram of the PRCBlock structure (Partial residual convolutionBlock, partial residual convolution module), which combines 1x1 convolution, mixes spatial dimension and channel dimension information, and uses PRCBlock for inverted bottleneck downsampling, which has lightweight and efficient feature extraction capabilities.

[0044] PRCBlock will feature X pr in Input 1x1 convolution to get feature X pr 1; Set feature X pr 1 Input PCconv to get feature X pr 11 , the feature X pr 1 Input 3x3DW convolution, LayerNorm to get feature X pr 12 , the feature X pr 11 With feature X pr 12Further spatial and channel feature extraction, and addition operation to obtain feature X pr 2. Set feature X pr 2 After 1x1 convolution, GELU, 1x1 convolution and feature X pr 1 Perform residual connection; finally output feature X through BatchNorm pr out .

[0045] like Figure 3 The figure shows the schematic diagram of the WREM (Wavelet Information Reconstruction Enhancement Module) structure, which uses the wavelet self-attention mechanism to enhance the target feature information and suppress redundant features in space and channels; WREM contains multiple enhancement modules to enhance the target feature information and suppress redundant features in space and channels; in order to fuse shallow and deep feature information and context information at multiple scales.

[0046] WREM includes: Input feature X wr in The CCEM (Channel Complementarity Enhancement Module) is used to split, add and differentiate spatial features to reorganize spatial features to suppress spatial redundancy and enhance feature information to obtain feature X. ccw , feature X ccw The SDEM (Spatial Differential Enhancement Module) is used to further suppress the redundancy of the channel and extract the channel information to obtain the feature X sdw ; Input feature X wr in After WSEM (Wavelet Self-Attention Enhancement Module, wavelet enhancement module) we get feature X ws , and finally the feature X sdw Through 1x1 convolution with feature X ws After concatenation, the feature X is output through 1x1 convolution. wr out ;

[0047] Among them, WSEM includes three branches. The first branch converts the feature X wr in Input Linear to get feature Q; the second branch converts X wr inInput Linear and WTM to get feature F, input feature F to Linear to get feature V, input feature F to IWTM and Linear to get feature K, multiply feature Q with feature K, pass through softmax function, multiply with feature V, pass through Linear and multiply with feature X of the third branch. wr in Perform addition operation and output X ws .

[0048] Among them, WTM (Wavelet Transform Module) includes 1x1 convolution, DWT and 3x3 convolution. Each convolution is connected to the BN layer and RELU; and it is residually connected with the average pooling and then convolved by PCconv; among them, DWT is Discrete Wavelet Transform, and wavelet transform is downsampled to the existing module.

[0049] IWTM (Inverse Wavelet Transform Module) includes: DWConv, IDWT connected with average pooling for residual connection, and then convolved with PCconv; among them, IDWT is the inverse wavelet transform downsampling, which is an existing module.

[0050] like Figure 4 is the CCEM structure diagram, feature X wr in After inputting into the GN layer, it is re-weighted to obtain a new weight, and then mapped to the range of (0,1) by the sigmoid function. After that, the threshold T is used for gating to obtain the information weight W1 and the non-information weight W2 respectively. The weights W1 and W2 are respectively related to the feature X wr in Multiply them together to get rich feature information X ccw 1 and rare feature information X ccw 2; Then perform the reconstruction operation and change X ccw 1 and X ccw 2. Perform cross reconstruction and cross phase difference operations; the phase difference operation highlights the part without feature information, fully combines different information features, strengthens the information interaction between features, and then splices them to obtain feature X ccw1 and feature X ccw2 , X ccw1 and X ccw2 Add together to get the spatial feature information X ccw ; CCEM can be expressed by the formula:

[0051] W=f Gate (f sigmoid (f GN (Xwr in )))

[0052]

[0053] like Figure 5 is the SDEM (Spatial Differential Enhancement Module) structure diagram, for feature X ccw The number of channels C is segmented according to the segmentation weight α, and then 1x1 convolution compression is performed to obtain the deep feature information X sd up and shallow feature information X sd low ; The deep feature information X sd up Through GWC convolution and 1x1 convolution respectively, and then add them to get the output feature F sd 1; The shallow feature information X sd low Through 1x1 convolution and combined with X sd low Concatenate and output feature F sd 2. For X ccw Perform 1x1 convolution compression to obtain feature F sd 3, respectively with F sd 1. F sd 2 is added and then concatenated to obtain feature F sd ; For feature F sd Perform global average pooling and Softmax operations and F sd Multiply them, split them, and get the feature weight vectors β1 and β2 respectively. Finally, add β1 and F sd 1 multiplied, β2 and F sd 2 multiply and add to get the enhanced channel feature X sdw .

[0054] like Figure 6As shown in FIG. 1 , MAFM (Multiscale Asymmetric Fusion Module) has multiple asymmetric convolution branches to improve the feature extraction capability of context and obtain richer context depth features; multiple asymmetric convolution branches are mainly divided into 4 branches, the first branch convolution kernel is 1x1, the second branch convolution is 1x1, 3x1, 1x3 convolution kernel series, the third branch convolution is 3x3, 5x1, 1x5 convolution kernel series, the fourth branch convolution is 5x5, 7x1, 1x7 convolution kernel series, the features extracted by the four branches are spliced, and residual connection is performed with the original input features to supplement the feature information, and the final multi-scale asymmetric feature output X is obtained. MA out .

[0055] like Figure 7 The IFM (Interaction Fusion Module) is shown here, which combines the shallow features X Low and deep features X High Concatenate to get feature X HL , X HL Through the global and local branches, we can obtain the global information X global and local information X local ;X HL After passing through the average pooling layer and the maximum pooling layer respectively, they are added together with X HL Point multiplication, and then through 1x1 convolution and BN layer to get the global feature X global ; The local branch is mainly to convert X HL First, 1x1 convolution is performed to shuffle the channels, then 1x1 convolution is performed to integrate feature information, and finally softmax is used to obtain the local feature X. local , two 1x1 convolutions are connected to the BN layer; X HL Respectively with X local , X global Multiply and add X HL Add, and finally get the final fusion feature X through 1x1 convolution IF out .

[0056] Step 4: Output feature X from the decoder out The final grasp quality map, grasp angle map, and grasp width map are obtained through four 1x1 convolutional layers respectively, and the output optimal grasp is expressed as follows:

[0057]

[0058] Among them, G Q To determine the center point position, the angle is given by (G sin(2θ) / G cos (2θ)) / 2 to determine, G w to determine the width.

[0059] We select smooth L1 loss as the loss function to optimize the grasp detection model. For the input image I = {I 1 ,I 2 ,...,I n}, and the true label L = {L1, L2, ..., L n}; Use G i Representing the crawl prediction of the network output, the formula of the loss function L1 is as follows:

[0060]

[0061] Among them, N represents the number of samples, G im and L m They represent the mth generated grasp and its corresponding true label respectively.

[0062] Through the hand-eye transformation matrix, the image coordinate system is transformed into the camera coordinate system, and then transformed into the robot coordinate system, and finally the robot grasps the target. Assume that the predicted grasp is represented by Gr:

[0063] G r =(P,Q,Θ r ,W r )

[0064] Where P = (x, y, z) is the center point of the grasp position, Q is the grasp quality score of the predicted grasp, θr is the rotation angle of the end point around the z axis, and W r is the width of the end opening.

[0065] In the RGBD N channel, the image is height and width Predicted grasping posture:

[0066] G r =(u,v,Q,Θ i ,W i )

[0067] Where (u, v) represents the coordinate position of the grasp center point in the RGBD image, Q represents the grasp quality score, Θ i is the rotation angle in the camera coordinate system, W i is the grasp width in RGBD image coordinates; the grasp quality score Q evaluates the grasp quality of each point, and its range is between (0, 1); Θ i Indicates the rotation angle required to grab the object, its range is Between; W iThe range is [0, W max ], W max is the maximum width of the claw clamp.

[0068] In order to allow the robot to actually grasp, after obtaining the network output G i Then, G is obtained by the following formula r :

[0069] G r =T cr (T ic (G i ))

[0070] Among them, T ic It is expressed as converting the image coordinate system to the camera coordinate system, T cr It is expressed as converting the camera coordinate system to the robot coordinate system.

[0071] The effect of the model of the present invention is described below in conjunction with the Cornell dataset and the Jacquard dataset, wherein the specific details of the two datasets are shown in Table 1.

[0072] Table 1 Dataset data

[0073] Dataset Modality Objects Images Grasps Cornell RGB-D 240 885 8019 Jacquard RGB-D 11k 54k 1.1M

[0074] The Cornell dataset contains 885 RGBD images of 240 different real objects, covering a total of 8019 manually annotated grasp rectangles, 5110 positive grasps and 2909 negative grasps. Data augmentation strategies such as random rotation and scaling were applied to the Cornell dataset to enhance the robustness and generalization ability of the model. In contrast, the Jacquard dataset is larger, including 11k different real objects and a total of 54k RGBD images. In the training dataset, no data augmentation strategy was used because the data scale was large enough to fully train the model. When dividing the dataset, 90% of the data was used as the training set and the remaining 10% was used as the validation set. In order to comprehensively evaluate the performance of the model, five-fold cross-validation was performed to ensure the robustness and reliability of the results.

[0075] Training experiment settings: CUDA 11.6 and PyTorch 1.13 deep learning framework training were used on the RTX 3090 GPU; during the model training phase, Adam optimizer was used, the batch size was set to 8, the initial learning rate was set to 0.001, and the evaluation indicators used in most related works were used. If the capture meets the following two conditions, it is considered correct:

[0076] 1. The angle difference between the estimated grasping angle and the actual grasping angle should be kept within 30 degrees;

[0077] 2. Estimate the generated grab rectangle G r and the ground truth grasp rectangle G t The IoU score must be greater than 0.25, and the equation is as follows:

[0078]

[0079] For the Cornell data, two evaluation methods are used, one is to divide IW by image, and the other is to divide OW by object. The comparison of the detection results of the Cornell dataset between the model of the present invention and other models is shown in Table 2.

[0080] Table 2 Comparison of the test results of the present invention and other methods

[0081]

[0082] As can be seen from Table 2, the model of the present invention has a significant improvement in IW and OW performance on the Cornell dataset, reaching 99.1% and 98.3% in IW and OW respectively.

[0083] The comparison of the detection results of the Jacquard dataset in the model of the present invention and other models is shown in Table 3:

[0084] Table 3 Comparison of detection results of Jacquard dataset

[0085]

[0086] It can be seen from Table 3 that the model of the present invention reaches 95.7% on the Jacquard dataset.

[0087] like Figure 8 As shown, the grasping rectangle box generated by the present invention on the Cornell dataset indicates that the model of the present invention effectively improves the grasping detection performance.

[0088] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A robot-based grasping detection method, characterized in that: The following steps are involved: Step 1: Obtain the RGBD image of the object to be grasped by the robot; Step 2: Construct an encoder for the grasp detection model, and use the first to fourth PRCBlocks to extract features in sequence to obtain features F1, F2, F3, and F4; downsample F1, upsample F3, and then combine with F2 to obtain feature F 123 ; Upsample F4 and then F 123 Downsample, concatenate with F3 and input into the first MAFM to get feature X High ; F 123 Input the second MAFM to get the feature X Low , X High Upsampling, X Low After downsampling, input IFM to obtain feature X LH ; X LH Downsample, input F4 into WREM, and then add it to F4 to get feature F5; Step 3: Build a decoder for the capture detection model and use the decoder to decode the output features of the encoder; Step 4: Extract the features of the decoder output to obtain a grasping quality map, a grasping angle map, and a grasping width map; Step 5: Use the loss function to optimize the grasping detection model.

2. The robot-based grasping detection method according to claim 1, characterized in that: The decoder includes: feature F5 is input into the first ConVT to obtain feature F6, feature F6 and feature X LH Add together to get feature F 1 6. Feature F 1 6 Input the second ConVT to get feature F7, feature F7 and feature X LH After upsampling, add the input to the third ConVT to obtain feature F8 and feature X LH After upsampling, add it to feature F8 to get feature F 1 8. Feature F 1 8 Input the fourth ConVT to get feature X out .

3. The robot-based grasping detection method according to claim 1, characterized in that: The decoder includes: PRCBlock includes: Features X pr in Input 1x1 convolution to get feature X pr 1. X pr 1 Input PCconv to get feature X pr 11 , X pr 1 Input 3x3DW convolution, LayerNorm to get feature X pr 12 , X pr 11 With X pr 12 Add to get feature X pr 2. X pr 2 After 1x1 convolution, GELU, 1x1 convolution and X pr 1 Perform residual connection; then output feature X through BatchNorm pr out .

4. The robot-based grasping detection method according to claim 1, characterized in that: WREM includes: Feature X wr in After CCEM, we get feature X ccw , X ccw Get feature X through SDEM sdw ;X wr in After WSEM, we get feature X ws , X sdw Through 1x1 convolution, and X ws After concatenation, the feature X is output through 1x1 convolution wr out .

5. The robot-based grasping detection method according to claim 4, characterized in that: WSEM consists of three branches. The first branch converts feature X wr in Input Linear to get feature Q; The second branch will be X wr in Input Linear and WTM to get feature F, input F to Linear to get feature V, input F to IWTM and Linear to get feature K, multiply Q and K, pass through the softmax function, multiply with V, pass through Linear and multiply with X wr in Perform addition operation and output X ws .

6. The robot-based grasping detection method according to claim 5, characterized in that: WTM includes 1x1 convolution, DWT and 3x3 convolution, each of which is connected to BN layer and RELU; and is convolved with PCconv after residual connection with average pooling; IWTM includes: DWConv, IDWT connected with average pooling for residual connection, and then convolution with PCconv.

7. The robot-based grasping detection method according to claim 4, characterized in that: CCEM includes: Features X wr in After inputting into the GN layer, it is weighted, and then passes through the sigmoid function and gated by the threshold T to obtain the information weight W1 and the non-information weight W2 respectively. W1 and W2 are respectively related to X wr in Multiply them together to get feature X ccw 1 and feature X ccw 2. For X ccw 1 and X ccw 2 After cross reconstruction and cross phase difference operation, splicing is performed to obtain feature X ccw1 and feature X ccw2 , X ccw1 and X ccw2 Add together to get feature X ccw .

8. The robot-based grasping detection method according to claim 4, characterized in that: SDEM includes: using the segmentation weight α to ccw The number of channels C is divided, and then three 1x1 convolutions are input respectively to obtain the feature X sd up , Feature X sd low and feature F sd 3. X sd up The feature F is obtained by adding the GWC convolution and 1x1 convolution respectively. sd 1. X sd low Through 1x1 convolution and combined with X sd low Perform residual splicing to output feature F sd 2; F sd 3 respectively with F sd 1. F sd 2 and then concatenate to get feature F sd ; for F sd Perform global average pooling and Softmax operations and combine with F sd Multiply them and split them to get feature weight vectors β1 and β2. sd 1 multiplied, β2 and F sd 2, and then add to get feature X sdw .

9. The robot-based grasping detection method according to claim 1, characterized in that: The first branch convolution kernel of MAFM is 1x1, the second branch convolution kernel is 1x1, 3x1, 1x3 convolution kernel series, the third branch convolution kernel is 3x3, 5x1, 1x5 convolution kernel series, the fourth branch convolution kernel is 5x5, 7x1, 1x7 convolution kernel series, the features extracted by the four branches are concatenated and residually connected with the input features to output feature X MA out .

10. The robot-based grasping detection method according to claim 1, characterized in that: IFM will feature X Low and feature X High Concatenate to get feature X HL , X HL After passing through the average pooling layer and the maximum pooling layer, the sum is added to X HL Point multiplication, and then through 1x1 convolution and BN layer to get feature X global ; X HL After two 1x1 convolutions, the feature X is obtained through softmax. local , two 1x1 convolutions are connected to the BN layer; X HL Respectively with X local , X global Multiply and add X HL Add, and finally get feature X through 1x1 convolution IF out .

Images