A robot-based method of grasp detection

Abstract

The application relates to the technical field of image processing, in particular to a robot-based grabbing detection method, which comprises the following steps: acquiring an RGBD image; constructing an encoder of a grabbing detection model, using the first to fourth PRCBlock to sequentially perform feature extraction, and obtaining features F1, F2, F3 and F4; respectively obtaining features F 123 , X High , X Low , X LH , F5; constructing a decoder of the grabbing detection model, decoding the output features of the encoder by using the decoder; performing feature extraction on the output features of the decoder to respectively obtain a grabbing quality graph, a grabbing angle graph and a grabbing width graph; and optimizing the grabbing detection model by using a loss function. The application solves the problems that the existing grabbing detection network cannot sufficiently extract feature information, is prone to neglecting more possible missing important grabbing area features, and reduces the accuracy of grabbing detection; and the existing method cannot sufficiently fuse low-level feature and high-level feature information.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a robot-based grasping and detection method. Background Technology

[0002] In recent years, the rapid development of robotics technology has had a profound impact on various fields such as industry, medicine, agriculture, and education. In the industrial sector, the widespread application of robots has improved production efficiency, reduced production costs, and also reduced the labor intensity of workers and improved the working environment. Grasping, as a key operation in robotics, has sparked extensive research interest. In industrial scenarios, unordered sorting tasks account for a significant proportion. Unlike objects in everyday environments, industrial parts often have highly reflective and textureless features, which poses challenges to the accuracy of grasping tasks. Currently, deep learning-based grasping detection methods are mainly divided into two types: generative grasping strategies and discriminative grasping estimation strategies. Current grasping detection methods still face many challenges in feature fusion and attention mechanisms, failing to fully extract feature information and fuse feature information across multiple scales and levels. This results in lower success rates when facing complex environments with varying sizes and unknown objects.

[0003] Wang et al. proposed TF-Grasp by combining Transformer with encoder and decoder. Although it achieved good results, it was weak in edge processing of objects due to the lack of global and local information.

[0004] Zhang et al. proposed Double Strand Robotic Grasp Detection Network Based on Cross Attention, which combines transformer and U-Net branches in the encoder-decoder structure. However, it does not make effective use of multi-scale receptive fields, which reduces the feature extraction capability for multiple grasping targets.

[0005] The Efficient Grasp Detection Network With Gaussian-Based Grasp Representation for Robotic Manipulation proposed by Cao et al. simply utilizes channel and pixel attention fusion without making good use of contextual information to obtain local feature information; and it does not utilize the self-attention mechanism, which can effectively connect contextual information. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention solves the problem that existing grasping detection networks cannot fully extract feature information, easily overlook or even lose important grasping region features, thus reducing the accuracy of grasping detection; and that existing methods cannot fully integrate low-level and high-level feature information, resulting in robots or robotic arms being unable to quickly grasp target objects.

[0007] The technical solution adopted in this invention is: a robot-based grasping and detection method comprising the following steps:

[0008] Step 1: Obtain RGBD images of the object to be grasped by the robot and construct an RGBD image dataset;

[0009] Step 2: Construct the encoder for the crawling detection model. Use the first to fourth PRCBlocks to extract features sequentially, obtaining features F1, F2, F3, and F4. Downsample F1, upsample F3, and then combine it with F2 to obtain feature F. 123 Upsample F4, then upsample F 123 Downsampling, concatenating with F3, and then inputting into the first MAFM yields feature X. High ;F 123 Input the second MAFM to obtain feature X Low X High Upsampling, X Low After downsampling, the feature X is obtained by inputting it into IFM. LH ; X LH Downsampling, inputting F4 into WREM, and then adding it to F4 yields feature F5;

[0010] As a preferred embodiment of the present invention, PRCBlock includes: feature X pr in Input a 1x1 convolution to obtain feature X pr 1; X pr 1. Input PCconv to obtain feature X pr 11 X pr 1. Input 3x3DW convolution and LayerNorm to obtain feature X pr 12 X pr 11 With X pr 12 Adding them together yields feature X. pr 2; X pr 2 after 1x1 convolution, GELU, and 1x1 convolution, and then with X pr 1. Perform residual connections; then output feature X using BatchNorm. pr out .

[0011] As a preferred embodiment of the present invention, WREM includes: feature X wr in Feature X is obtained through CCEM. ccw X ccw Feature X is obtained through SDEM. sdw ;X wr in Feature X was obtained after WSEM. ws X sdw Through 1x1 convolution, and with X ws After concatenation, the output feature X is then processed by a 1x1 convolution. wr out .

[0012] As a preferred embodiment of the present invention, WSEM includes three branches, the first branch of which includes feature X wr in Input Linear to obtain feature Q; the second branch will convert X wr in Inputting Linear and WTM yields feature F. Inputting F into Linear yields feature V. Inputting F into IWTM and Linear yields feature K. Multiplying Q and K, then passing the result through a softmax function, multiplying it again by V, passing it through a Linear function, and finally multiplying it by X. wr in Perform an addition operation and output X ws .

[0013] As a preferred embodiment of the present invention, WTM includes 1x1 convolution, DWT and 3x3 convolution, each convolution is followed by a BN layer and a ReLU layer; and after residual connection with average pooling, it is followed by PCconv convolution; IWTM includes: DWConv, IDWT connected and then residual connection with average pooling, and then followed by PCconv convolution.

[0014] As a preferred embodiment of the present invention, CCEM includes: feature X wr in After being input into the GN layer, the weights are applied, then the sigmoid function is used, and a threshold T is applied to obtain the information weights W1 and non-information weights W2. W1 and W2 are then compared with X. wr in Multiply to obtain feature X ccw 1 and feature X ccw 2. For X ccw 1 and X ccw 2. After performing cross-reconstruction and cross-difference operations, the data is concatenated to obtain feature X. ccw1 and feature X ccw2 X ccw1 and X ccw2Add them together to obtain feature X. ccw .

[0015] In a preferred embodiment of the present invention, SDEM includes: using segmentation weight α to segment feature X ccw The channel number C is segmented, and then each segment is input into three 1x1 convolutions to obtain the feature X. sd up Feature X sd low and feature F sd 3; X sd up The feature F is obtained by summing the GWC convolution and 1x1 convolution respectively. sd 1; X sd low Through 1x1 convolution and X sd low Perform residual splicing to output feature F sd 2; F sd 3 and F respectively sd 1. F sd The feature F is obtained by adding the two together and then splicing them. sd ; For F sd Perform global average pooling and softmax operations and combine with F sd Multiply the vectors, then split them to obtain feature weight vectors β1 and β2. Then, multiply β1 with F... sd 1 multiplied, β2 and F sd Multiply by 2 and then add them together to get the feature X. sdw .

[0016] In a preferred embodiment of the present invention, the first branch of MAFM has a 1x1 convolutional kernel, the second branch has cascaded 1x1, 3x1, and 1x3 convolutional kernels, the third branch has cascaded 3x3, 5x1, and 1x5 convolutional kernels, and the fourth branch has cascaded 5x5, 7x1, and 1x7 convolutional kernels. The features extracted from the four branches are concatenated and residually connected with the input features to output feature X. MA out .

[0017] As a preferred embodiment of the present invention, IFM will feature X Low and feature X High The feature X is obtained by splicing. HL X HL After passing through the average pooling layer and the max pooling layer, the result is added together and then summed with X. HL Dot product, then through 1x1 convolution and BN layer to obtain feature X global ; X HL After two 1x1 convolutions, the feature X is obtained through softmax. localBoth 1x1 convolutions are connected to BN layers; X HL respectively with X local X global Multiply, then multiply by X HL The features are added together and then obtained by a 1x1 convolution. IF out .

[0018] Step 3: Construct a decoder for the capture detection model and use the decoder to decode the output features of the encoder;

[0019] In a preferred embodiment of the present invention, the decoder includes: feature F5 inputting into the first ConVT to obtain feature F6, and feature F6 and feature X... LH Adding them together yields the characteristic F. 1 6. Feature F 1 6. Input the second ConVT to obtain feature F7. Feature F7 and feature X LH After upsampling, the samples are added together and input to the third ConVT to obtain feature F8, feature X. LH After upsampling, it is added to feature F8 to obtain feature F. 1 8. Feature F 1 8. Input the fourth ConVT to obtain feature X out .

[0020] Step 4: Extract features from the decoder output to obtain the capture quality map, capture angle map, and capture width map.

[0021] Step 5: Optimize the capture and detection model using the loss function.

[0022] The beneficial effects of this invention are:

[0023] 1. This invention designs a novel downsampling module that is simple, has few parameters, and effectively extracts rich semantic feature information;

[0024] 2. The present invention designs an enhancement module to enhance target feature information while suppressing redundant features in space and channels; the designed wavelet self-attention enhancement module improves the model's ability to capture important information by combining wavelet transform and self-attention mechanism.

[0025] 3. The present invention designs a fusion module to capture feature information and context information at different scales, effectively mining shallow and deep features; the shallow and deep information interaction fusion module further integrates these features, thereby obtaining more target feature information;

[0026] 4. The present invention achieved an accuracy of 99.1% and 95.7% on the publicly available crawled datasets Cornell and Jacquard, respectively. Attached Figure Description

[0027] Figure 1 This is a diagram showing the overall structure of the capture and detection network model of this invention;

[0028] Figure 2 This is a structural diagram of the PRCBlock of the present invention;

[0029] Figure 3 This is a schematic diagram of the overall structure of the WREM of the present invention;

[0030] Figure 4 This is a CCEM structural diagram of the present invention;

[0031] Figure 5 This is a structural diagram of the SDEM of the present invention;

[0032] Figure 6 This is a structural diagram of the MAFM of the present invention;

[0033] Figure 7 This is a structural diagram of the IFM of the present invention;

[0034] Figure 8 To capture and detect the image. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0036] A robot-based grasping and detection method includes the following steps:

[0037] Step 1: First, acquire the RGBD image of the object to be grasped by the robot and construct the RGBD image dataset;

[0038] The dataset can be the Cornell dataset, the Jacquard dataset, or a self-built dataset; the images should be preprocessed to a uniform size of 224x224;

[0039] like Figure 1 As shown, step two involves constructing a crawling detection model, which includes an encoder and a decoder.

[0040] Encoders include: PRCBlock, WREM, MAFM, and IFM;

[0041] The RGBD image of the object to be captured underwent four feature extractions using the first to fourth PRCBlocks, yielding features F1, F2, F3, and F4. Feature F1 was downsampled, and then feature F3 was upsampled and concatenated with feature F2 to obtain feature F. 123 Upsample feature F4, then upsample feature F 123Downsampling, concatenating with feature F3, and then inputting into the first MAFM yields feature X. High Feature F 123 Input the second MAFM to obtain feature X Low ; feature X High Upsampling, using feature X Low After downsampling, the common input to the IFM output feature X LH ; feature X LH Downsampling, feature F4 is fed into WREM and then compared with features F4 and X. LH The downsampled features are summed to obtain feature F5.

[0042] Step 3: Construct the decoder for the crawling detection model. The decoder includes four ConVTs. Feature F5 is input into the first ConVT to obtain feature F6. Feature F6 and feature X are then compared. LH Adding them together yields the characteristic F. 1 6. Feature F 1 6. Input the second ConVT to obtain feature F7, and feature F7 and feature X LH After upsampling, the samples are added together and input to the third ConVT to obtain feature F8, feature X. LH After upsampling, it is added to feature F8 to obtain feature F. 1 8. Feature F 1 8. Input the fourth ConVT to obtain feature X out Among them, the first to third ConVTs are three 4x4 deconvolutions; the fourth ConVT is a 9x9 deconvolution.

[0043] like Figure 2 The diagram shows the structure of PRCBlock (Partial residual convolutionBlock), which combines 1x1 convolution to mix spatial and channel dimension information. PRCBlock is used for inverted bottleneck downsampling, which has lightweight and efficient feature extraction capabilities.

[0044] PRCBlock will feature X pr in Input a 1x1 convolution to obtain feature X pr 1; The feature X pr 1. Input PCconv to obtain feature X pr 11 , feature X pr 1. Input 3x3DW convolution and LayerNorm to obtain feature X pr 12 , feature X pr 11 With feature X pr 12Further spatial and channel feature extraction is performed, and the features are summed to obtain feature X. pr 2; The feature X pr 2. After 1x1 convolution, GELU, and 1x1 convolution, the result is compared with feature X. pr 1. Perform residual connections; finally, output feature X using BatchNorm. pr out .

[0045] like Figure 3 The diagram shows the structure of WREM (Wavelet Information Reconstruction Enhancement Module), which uses wavelet self-attention mechanism to enhance target feature information and suppress redundant features in space and channels. WREM contains multiple enhancement modules to enhance target feature information while suppressing redundant features in space and channels. It is designed for multi-scale fusion of shallow and deep feature information and contextual information.

[0046] WREM includes: input features X wr in The spatial features are then decomposed, added, and differencing using the CCEM (Channel Complementarity Enhancement Module) to reconstruct the spatial features, thereby suppressing spatial redundancy and enhancing feature information, resulting in feature X. ccw Feature X ccw The Spatial Differential Enhancement Module (SDEM) is used to further suppress channel redundancy and extract channel information to obtain feature X. sdw Input feature X wr in Feature X is obtained after passing through WSEM (Wavelet Self-Attention Enhancement Module). ws Finally, feature X sdw Through 1x1 convolution with feature X ws After concatenation, the feature X is output through a 1x1 convolution. wr out ;

[0047] WSEM consists of three branches. The first branch will use feature X... wr in Input Linear to obtain feature Q; the second branch will convert X wr inInputting Linear and WTM yields feature F. Inputting feature F into Linear yields feature V. Inputting feature F into IWTM and Linear yields feature K. Multiplying feature Q by feature K, then passing it through a softmax function, multiplying it by feature V, then passing it through Linear again, and finally combining it with the third branch feature X. wr in Perform an addition operation and output X ws .

[0048] The WTM (Wavelet Transform Module) includes 1x1 convolution, DWT, and 3x3 convolution. Each convolution is followed by a BN layer and a ReLU layer. It is then connected to average pooling for residual convolution and then passed through PCconv convolution. DWT, or Discrete Wavelet Transform, is an existing module for wavelet transform downsampling.

[0049] The IWTM (Inverse Wavelet Transform Module) consists of: DWConv and IDWT connected together, followed by residual connection with average pooling, and then PCconv convolution; where IDWT is an inverse wavelet transform downsampling module, which is an existing module.

[0050] like Figure 4 This is a CCEM structure diagram, featuring X. wr in After being input into the GN layer, the weights are re-weighted to obtain new weights, which are then mapped to the (0,1) range using the sigmoid function. Finally, a threshold T is applied to gate the data to obtain information weights W1 and non-information weights W2. Weights W1 and W2 are then correlated with the feature X. wr in Multiplication yields rich feature information X. ccw 1 and scarce feature information X ccw 2; Next, a reconstruction operation will be performed, changing X... ccw 1 and X ccw 2. Perform cross-reconstruction and cross-difference operations; the difference operation highlights the parts without feature information, fully combines different information features, strengthens the information interaction between features, and then concatenates them to obtain feature X. ccw1 and feature X ccw2 X ccw1 and X ccw2 Adding them together yields the spatial feature information X. ccw CCEM can be expressed by a formula:

[0051] W = f Gate (f sigmoid (f GN (Xwr in )))

[0052]

[0053] like Figure 5 This is a structural diagram of the SDEM (Spatial Differential Enhancement Module), focusing on feature X. ccw The number of channels C is divided according to the segmentation weight α, and then each channel is compressed by 1x1 convolution to obtain the deep feature information X. sd up and shallow feature information X sd low ; to transfer deep feature information X sd up The output feature F is obtained by applying GWC convolution and 1x1 convolution respectively and then summing them. sd 1; Transfer shallow feature information X sd low Through 1x1 convolution and X sd low Perform splicing to output feature F sd 2; For X ccw Perform 1x1 convolution compression to obtain feature F sd 3, respectively with F sd 1. F sd The two are added together and then spliced ​​to obtain feature F. sd ; for feature F sd Perform global average pooling and softmax operations and F sd Perform multiplication, then split to obtain feature weight vectors β1 and β2 respectively, and finally multiply β1 with F. sd 1 multiplied, β2 and F sd Multiplying by 2 and then adding them together yields the enhanced channel feature X. sdw .

[0054] like Figure 6As shown, MAFM (Multiscale Asymmetric Fusion Module) has multiple branches of asymmetric convolutions, improving the ability to extract contextual features and obtaining richer contextual depth features. The multiple branches of asymmetric convolutions are mainly divided into four types: the first branch has a 1x1 kernel; the second branch has concatenated 1x1, 3x1, and 1x3 kernels; the third branch has concatenated 3x3, 5x1, and 1x5 kernels; and the fourth branch has concatenated 5x5, 7x1, and 1x7 kernels. The features extracted from these four branches are concatenated and residually connected with the original input features to supplement feature information, obtaining the final multiscale asymmetric feature output X. MA out .

[0055] like Figure 7 The image shows IFM (Interaction Fusion Module), which integrates shallow features X. Low and deep features X High The feature X is obtained by splicing. HL X HL Global information X is obtained through both global and local branches. global and local information X local ;X HL The average pooling layer and the max pooling layer are used separately, then their sums are added and then combined with X. HL Dot product, then through 1x1 convolution and BN layer to obtain global feature X global The local branch mainly involves X. HL First, a 1x1 convolution is used to shuffle the channels, then another 1x1 convolution is used to integrate the feature information, and finally, softmax is used to obtain the local feature X. local Both 1x1 convolutions are connected to a Batch Normalization (BN) layer; X HL respectively with X local X global Multiply, then multiply by X HL The features are added together, and finally, a 1x1 convolution is performed to obtain the final fused feature X. IF out .

[0056] Step 4: Output feature X from the decoder out The final grasping quality image, grasping angle image, and grasping width image are obtained by passing four 1x1 convolutional layers respectively. The output of the optimal grasping is represented as follows:

[0057]

[0058] Among them, G Q To determine the location of the center point, the angle is determined by (G). sin(2θ) / G cos (2θ)) / 2 is used to determine G w To determine the width.

[0059] We choose smooth L1 loss as the loss function to optimize the grasping and detection model. For the input image I = {I... 1 ,I 2 ,...,I n}, and the real labels L = {L1, L2, ..., L} n}; Using G i The L1 loss function, representing the crawling prediction output by the network, is expressed as follows:

[0060]

[0061] Where N represents the number of samples, G im and L m Let m represent the m-th generated crawl and its corresponding real label, respectively.

[0062] By using a hand-eye transformation matrix, the image coordinate system is transformed to the camera coordinate system, and then to the robot coordinate system, ultimately completing the robot's grasping of the target. Let the predicted grasping be represented as Gr:

[0063] G r =(P,Q,Θ) r W r )

[0064] Where P = (x, y, z) is the center point of the grasp position, Q is the predicted grasp quality fraction, Θr is the rotation angle of the end about the z-axis, and W... r The width of the end opening.

[0065] An image with height and width in the N channel of RGBD. Predicting the grasping posture:

[0066] G r =(u,v,Q,Θ) i W i )

[0067] Where (u,v) represents the coordinates of the capture center point in the RGBD image, Q represents the capture quality score, and Θ i W is the rotation angle in the camera coordinate system. i Θ is the capture width in RGBD image coordinates; the capture quality score Q evaluates the capture quality of each point, and its range is between (0, 1); Θ i This indicates the rotation angle required to grab the object; its range is within... Between; W iThe range is [0, W] max Within [the range], W max It is the maximum width of the gripper.

[0068] In order for the robot to perform actual grasping, after obtaining the G output from the network... i Then, G is obtained through the following formula. r :

[0069] G r =T cr (T ic (G i ))

[0070] Among them, T ic This is represented as transforming the image coordinate system to the camera coordinate system, T cr This is represented as transforming the camera coordinate system to the robot coordinate system.

[0071] The effectiveness of the model of this invention is explained below using the Cornell dataset and the Jacquard dataset, with specific details of the two datasets shown in Table 1.

[0072] Table 1 Dataset

[0073] Dataset Modality Objects Images Grasps Cornell RGB-D 240 885 8019 Jacquard RGB-D 11k 54k 1.1M

[0074] The Cornell dataset contains 885 RGBD images of 240 different real-world objects, encompassing 8019 hand-annotated grasp rectangles, with 5110 positive grasps and 2909 negative grasps. Data augmentation strategies such as random rotation and scaling were applied to the Cornell dataset to enhance the model's robustness and generalization ability. In contrast, the Jacquard dataset is larger, including 11k different real-world objects and 54k RGBD images. No data augmentation strategies were used in the training dataset because the data size was already large enough to adequately train the model. When splitting the dataset, 90% of the data was used as the training set, and the remaining 10% as the validation set. Five-fold cross-validation was performed to comprehensively evaluate the model's performance, ensuring the robustness and reliability of the results.

[0075] Training setup: The model was trained on an RTX 3090 GPU using the CUDA 11.6 and PyTorch 1.13 deep learning frameworks. During training, the Adam optimizer was used with a batch size of 8 and an initial learning rate of 0.001. Evaluation metrics similar to those used in most related works were employed; a model was considered correct if it met the following two conditions:

[0076] 1. The angle difference between the estimated grasping angle and the actual grasping angle should be kept within 30 degrees;

[0077] 2. Estimate the generated grab rectangle G r and ground truth grasp rectangle G t The IoU score must be greater than 0.25, as shown in the following equation:

[0078]

[0079] For the Cornell dataset, two evaluation methods were used: one for image segmentation (IW) and the other for object segmentation (OW). Table 2 shows a comparison of the detection results of the model in this invention with other models on the Cornell dataset.

[0080] Table 2 Comparison of detection results of the present invention with other methods

[0081]

[0082] As shown in Table 2, the model of this invention has significant performance improvements on the IW and OW datasets on the Cornell dataset, reaching 99.1% and 98.3% respectively.

[0083] Table 3 shows a comparison of the detection results of the Jacquard dataset between the model of this invention and other models:

[0084] Table 3 Comparison of detection results on the Jacquard dataset

[0085]

[0086] As shown in Table 3, the model of this invention achieved 95.7% accuracy on the Jacquard dataset.

[0087] like Figure 8 As shown, the grabbing rectangles generated by this invention on the Cornell dataset demonstrate that the model of this invention effectively improves grabbing detection performance.

[0088] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A robot-based grasping and detection method, characterized in that, Includes the following steps: Step 1: Obtain the RGBD image of the object to be grasped by the robot; Step 2: Construct the encoder of the crawling detection model, and extract features sequentially using the first to fourth PRCBlocks to obtain features. F 1 , F 2 , F 3 , F 4 ;Will F 1 Downsampling, then F 3 After upsampling and F 2 Obtain features F 123 ;Will F 4 Upsampling, then F 123 Downsampling, common and F 3 Concatenate the data and then input the first MAFM to obtain the features. X High ; F 123 Input the second MAFM to obtain features X Low ,Will X High Upsampling X Low After downsampling, the features are obtained by inputting into IFM. X LH ;Will X LH downsampling F 4 Enter WREM, then... F 4 Adding them together yields the features. F 5 ; WREM includes: features i>X wr in Features obtained through CCEM X ccw , X ccw Features obtained through SDEM X sdw ; X wr in Features were obtained through WSEM. X ws ,Will X sdw Through 1x1 convolution, and X ws After concatenation, the features are output through a 1x1 convolution. X wr out ; Step 3: Construct a decoder for the capture detection model and use the decoder to decode the output features of the encoder; Step 4: Extract features from the decoder output to obtain the capture quality map, capture angle map, and capture width map. Step 5: Optimize the capture and detection model using the loss function.

2. The robot-based grasping and detection method according to claim 1, characterized in that, The decoder includes: features F 5 Input the first ConVT to obtain features F 6 ,feature F 6 With features X LH Adding together yields the features F 1 6 ,feature F 1 6 Input the second ConVT to obtain features F 7 ,feature F 7 With features X LH The features are obtained by upsampling and summing the samples, then inputting them into the third ConVT. F 8 ,feature X LH Upsampling and features F 8 Adding together yields the features F 1 8 ,feature F 1 8 Input the fourth ConVT to obtain features X out .

3. The robot-based grasping and detection method according to claim 1, characterized in that, PRCBlock includes: features X pr in Input 1x1 convolution to obtain features X pr 1 ;Will X pr 1 Inputting PCconv yields features X pr 11 ,Will X pr 1 Input 3x3DW convolution and LayerNorm to obtain features X pr 12 ,Will X pr 11 and X pr 12 Adding them together yields the features. X pr 2 ;Will X pr 2 After 1x1 convolution, GELU, and 1x1 convolution, and... X pr 1 Perform residual connections; then output features using BatchNorm. X pr out .

4. The robot-based grasping and detection method according to claim 1, characterized in that, WSEM consists of three branches, the first branch of which will feature X wr in Inputting Linear yields the feature Q; The second branch will X wr in Inputting Linear and WTM yields feature F. Inputting F into Linear yields feature V. Inputting F into IWTM and Linear yields feature K. Multiplying Q and K, then passing the result through a softmax function, multiplying it again by V, and finally passing it through a Linear function. X wr in Perform addition operation and output X ws .

5. The robot-based grasping and detection method according to claim 4, characterized in that, WTM includes 1x1 convolution, DWT and 3x3 convolution, each convolution is followed by a BN layer and a ReLU layer; and after residual connection with average pooling, it is followed by PCconv convolution. IWTM consists of: DWConv and IDWT connected together, followed by residual connections with average pooling, and then PCconv convolution.

6. The robot-based grasping and detection method according to claim 1, characterized in that, CCEM includes: features X wr in After being input into the GN layer, the data is weighted, then passed through a sigmoid function, and finally gated with a threshold T to obtain the information weights. W 1. Non-information weights W 2, W 1 and W 2 respectively with X wr in Multiply to obtain the characteristics With features ,right and After performing cross-reconstruction and cross-difference operations, the data is concatenated to obtain the features. and characteristics ,Will and Add them together to obtain the features. .

7. The robot-based grasping and detection method according to claim 1, characterized in that, SDEM includes: utilizing segmentation weights Features The channels C are segmented, and then each segment is input into three 1x1 convolutions to obtain the features. ,feature and characteristics F sd 3; will Features are obtained by combining GWC convolution and 1x1 convolution respectively. F sd 1; will Through 1x1 convolution and Perform residual splicing to output features F sd 2; F sd 3 respectively with F sd 1. F sd The feature is obtained by adding the two together and then splicing them. F sd ;right F sd Perform global average pooling and softmax operations and combine with F sd Multiply the components and then split them to obtain the feature weight vector. and ,Will and F sd 1 multiplied, and F sd Multiply by 2 and then add them together to get the characteristic. X sdw .

8. The robot-based grasping and detection method according to claim 1, characterized in that, MAFM's first branch uses a 1x1 convolutional kernel, the second branch uses a concatenated set of 1x1, 3x1, and 1x3 kernels, the third branch uses a concatenated set of 3x3, 5x1, and 1x5 kernels, and the fourth branch uses a concatenated set of 5x5, 7x1, and 1x7 kernels. The features extracted from these four branches are concatenated and then residually connected to the input features to output the final features. X MA out .

9. The robot-based grasping and detection method according to claim 1, characterized in that, IFM will feature X Low and characteristics X High Features are obtained by splicing. X HL , X HL After summing the average pooling layer and the max pooling layer, then... X HL Dot product, then through 1x1 convolution and BN layer to obtain features. X global ;Will X HL Features are obtained after two 1x1 convolutions and then softmax. X local Both 1x1 convolutions are connected to BN layers; X HL respectively with X local , X global Multiply, then multiply X HL The features are then obtained by summing the results and finally performing a 1x1 convolution. X IF out .

Images