Robot object grabbing method and device based on object 6D pose estimation

Abstract

The invention discloses a robot object grabbing method and device based on object 6D pose estimation. The method comprises the following steps: firstly, carrying out perception processing on an RGB image to obtain a target area image of a to-be-grabbed object, then extracting features of the target area image based on a backbone network fusing a semi-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism, and then processing the features based on a neural network composed of a plurality of convolutional layers and a full connection layer to obtain a target area image of the to-be-grabbed object. And therefore, the 6D pose including the to-be-grabbed object is returned. And finally, based on the CNN model, evaluating the success probability that the robot grabs the to-be-grabbed object, and according to the success probability, carrying out grabbing or adjusting the tail end position of the robot to carry out new grabbing. According to the method, a brand new thought is provided for solving the shielding robustness problem, more importantly, a new normal form of space-frequency domain feature fusion and intelligent grabbing is constructed, the robustness of 6D pose estimation in a disordered and shielding scene is greatly improved, and the method has an excellent application prospect in the aspect of industrial vision.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and robotics, and in particular relates to a robot object grasping method and device based on object 6D pose estimation. Background Art

[0002] 6D pose estimation has become a cornerstone technology for robotic manipulation, augmented reality (AR) systems, and industrial automation. Emerging innovations in the multi-layer perceptron (MLP) paradigm have significantly improved the accuracy of RGB-based pose estimation frameworks, especially through innovative network architectures and sophisticated feature representation methods. The LINEMOD dataset and its occluded variants have become standard benchmarks, driving continued progress in this field.

[0003] Current methods mainly use convolutional neural networks (CNNs) for indirect or direct pose regression. Indirect methods first establish the correspondence between 2D positions and 3D object key points, and then use the PnP algorithm to estimate the 6D pose. Direct methods adopt a holistic end-to-end pose regression architecture, where a deep convolutional network can directly calculate the 6D pose of the object from the raw visual input. Pioneering works such as PoseCNN have established baseline performance by learning global features, while PVNet introduced vector field representation to improve occlusion handling. As shown in CBAM, attention mechanisms have shown potential in improving feature extraction. The method provided by PANet obtains rich appearance features from RGB images through a pyramid pixel-level attention module, which effectively improves the localization error tolerance of the 6D pose estimation task.

[0004] However, current 6D pose estimation schemes still have some key limitations. The PnP algorithm in the indirect method usually takes a long time in object pose estimation and cannot be applied to many tasks that require differentiable pose estimation, while the direct method is sensitive to noise and requires a large amount of labeled data. In addition, the existing attention mechanism mainly acts on the normal domain of the object, ignoring the more diverse complementary wavelet domain features. In addition, multi-scale feature integration usually adopts a simpler structure and cannot establish the best cross-scale relationship. The performance of some existing works has declined under severe occlusion, especially when the object is partially or completely occluded, and cannot fully meet the needs of robots for intelligent grasping with high success rate in complex scenes. Summary of the invention

[0005] In view of the fact that the 6D pose estimation scheme currently used by robots in grasping has significant defects in occlusion robustness, feature diversity acquisition and multi-scale feature extraction, as well as the lack of intelligence in robot grasping, the present invention provides a robot object grasping method and device based on 6D pose estimation of objects with high accuracy, anti-occlusion and robustness.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the present invention include:

[0007] On the one hand, the present invention provides a robot object grasping method based on 6D pose estimation of the object, comprising the following steps:

[0008] S1, obtaining an RGB image including the object to be grasped, and performing perception processing on the RGB image by a perception model to obtain a target area image of a standard size with the object to be grasped as the target area;

[0009] S2, extracting the features of the target area image based on a backbone network that integrates a half-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism, wherein the half-wavelet pyramid attention mechanism simultaneously collects and integrates spatial domain and wavelet domain features through a hierarchical pyramid structure; the multi-scale feature extraction attention mechanism uses differentiated receptive fields and multi-branch attention to extract multi-scale features; and then combining the spatial domain and wavelet domain features to obtain the network prediction intermediate features, including a coordinate map, a mask map, a surface area attention map, and a dense correspondence map of the object to be grasped;

[0010] S3, based on a neural network composed of multiple convolutional layers and fully connected layers, processes the surface area attention map and the dense correspondence map to regress the 6D pose including the position and posture of the object to be grasped;

[0011] S4, inputs the 6D pose and mask image into the CNN model, evaluates the success probability of the robot grasping the object to be grasped, and executes the grasping or adjusts the robot end position to perform a new grasp based on the success probability.

[0012] Furthermore, in step S1, the step of perceiving and processing the image includes:

[0013] The existing two-dimensional target recognition model is used to perform real-time analysis on the image including the object to be grasped, so as to obtain the regional coordinates of the object to be grasped in the image, and extract the original image area in the image according to the regional coordinates;

[0014] Then, the original image area is normalized according to a preset size, so as to adjust the original image area to a preset uniform pixel size.

[0015] Furthermore, in step S2, the backbone network is a ResNet-34 structure, the half-wavelet pyramid attention mechanism is set between the residual blocks of ResNet-34, and the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block.

[0016] Furthermore, in step S2, the half-wavelet pyramid attention mechanism is:

[0017] The half-wavelet pyramid attention mechanism includes a multi-layer pyramid structure, and each layer of the pyramid structure receives the features obtained after the residual block convolution operation separately. , and each layer of the pyramid structure will along and The direction is evenly divided into feature vectors, where R represents the real space, C represents the number of channels, H represents the height, and W represents the width; each feature vector is processed by the half-wavelet attention mechanism and then de-segmented to merge to obtain the output of each layer, and finally the output of each layer is fused as the output of the half-wavelet pyramid attention mechanism ; where n is determined by the level i of the pyramid structure where the segmentation is performed, n=i 2 ;

[0018] The half-wavelet attention mechanism is:

[0019] The feature vector input to the half-wavelet attention mechanism is ,Will Divided into spatial domain feature branches and wavelet domain feature branch Two parts, including Used to extract features in the spatial domain, The features in wavelet domain are obtained by discrete wavelet transform DWT , and then in the wavelet domain Convolution is performed through two 3×3 convolution kernels to obtain the convolution result. ; Then use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The weighted results are then concatenated and the concatenated results are convolved with a convolution kernel of size 1. Add them; the result of the addition is then obtained by inverse wavelet transform ; and After concatenation, it passes through a 3×3 convolution kernel and a ReLU activation function, and then Add them together to get ;

[0020] The spatial attention mechanism is:

[0021] ;

[0022] in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, Represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation;

[0023] The channel attention mechanism is:

[0024] ;

[0025] in, represents the result of the channel attention mechanism, and Respectively represent the convolution operation using a 1×1 filter, represents the average pooling layer, and ReLU represents the ReLU activation function.

[0026] Further, in step S2, the multi-scale feature extraction attention mechanism includes: the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block;

[0027] In the first stage, the feature map obtained after the residual block convolution operation is normalized, and then a multi-scale convolution architecture is applied:

[0028] 1) Adjust the channel dimension through point convolution PWConv, and then perform 5×5 standard convolution to extract basic features;

[0029] 2) Three depth-wise separable dilated convolutions are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields. The outputs of the three depth-wise separable dilated convolutions are concatenated along the channel dimension to double the feature depth. Finally, a nonlinear MLP fuses multi-scale features, and a residual connection is used to add the original input of the first stage and the output of the MLP to achieve cross-scale collaboration.

[0030] In the second stage, the output of the first stage is first normalized; then, the weighting coefficients are obtained and spliced ​​through a three-branch attention architecture, namely the pixel attention mechanism, the channel attention mechanism, and the simple pixel attention mechanism; then, the spatial attention mechanism is used for weighting; finally, a nonlinear MLP is used to fuse multi-scale features, and the output of the MLP is added to the original input of the second stage to output the final result;

[0031] The channel attention mechanism in the second stage is:

[0032] ;

[0033] in, represents the result of the second-stage channel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a given feature; GELU represents the GELU activation function;

[0034] The pixel attention mechanism is:

[0035] ;

[0036] in, represents the result of the pixel attention mechanism, and They represent the convolution operation with the convolution kernel of 1 and the convolution kernel of 1 respectively;

[0037] A simple pixel attention mechanism is:

[0038] ;

[0039] in, represents the result of a simple pixel attention mechanism, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a convolution operation with a convolution kernel of 3.

[0040] Furthermore, in step S2, the network predicts the intermediate features as follows:

[0041] Multi-domain fusion object coordinate map of the object to be grasped , surface area attention map of multi-domain fusion , Dense correspondence graph of multi-domain fusion And the mask map of multi-domain fusion ;

[0042] in It is estimated based on the spatial and wavelet domains by rendering the CAD model in the target pose;

[0043] It is based on the spatial domain and wavelet domain, through the object coordinate map Generate by sampling the farthest point on;

[0044] is through Each channel of the object's 3D bounding box size is normalized to [0,1], so that the side length of the object's 3D bounding box in the object coordinate space is 1, and then Add 2D pixel coordinates to construct it;

[0045] It is obtained by performing semantic segmentation on the grasped objects based on the spatial domain and wavelet domain.

[0046] Furthermore, the step S3 includes:

[0047] Will and After splicing, the images are input into a neural network composed of several convolutional layers and fully connected layers as a pose regression module to regress the 6D pose including the position and posture of the object to be grasped.

[0048] The translation amount t of the object to be grasped is calculated by the following process:

[0049] First, we get , and :

[0050] ;

[0051] in, , , For the pose regression module , , The relative translation prediction results in the direction, , The center of the object to be grasped is , The coordinates in the direction, , The center of the known detection bounding box is , The coordinates in the direction, , is the width and height of the known detection bounding box; is the actual depth value to be solved, The scaling factor for the target area perception module to magnify the image;

[0052] Then calculate the translation of the object to be grasped by back projection :

[0053] ;

[0054] in is the intrinsic parameter matrix of a given camera;

[0055] Then calculate the rotation matrix of the object to be grasped according to the following formula:

[0056] ;

[0057] in, , , Three column vectors representing the rotation matrix; , To splice the and Input to the pose regression module to predict the first three values ​​and the last three values ​​of the rotation result. It is a normalization operation;

[0058] Reconstruct L for constraining the rotation matrix information R , L used to constrain 2D center offset information center , L used to constrain depth information depth and L used to constrain geometric information geometry There are four loss functions:

[0059] ;

[0060] in and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, Indicates the first 3D models, L ce represents the cross entropy loss, ⊙ represents the Hadamard product operation, ‖·‖ represents the norm operation; and the overall loss function is established for:

[0061] ;

[0062] Then according to the pose regression module , , Relative translation prediction results in direction , , , translation , the calculation method of the rotation matrix and the overall loss function are used to regress the 6D pose information of the object to be grasped.

[0063] Further, the step S4 includes:

[0064] Input crawled data into the CNN model as well as , , , , and , to evaluate the success probability of grasping in real time, and then decide whether to execute the current grasping or to perform a new grasping after randomly adjusting the end position of the robot: if the success probability is not lower than the preset threshold, the current grasping is performed; if the success probability is lower than the preset threshold, the robot randomly adjusts the end position and re-evaluates the success probability of grasping, and executes the cycle until the success probability of grasping is not lower than the preset threshold, and then the grasping is performed.

[0065] Furthermore, the perception model, backbone network and neural network in steps S1-S3 are obtained by joint training based on an RGB image dataset containing objects with 6D poses annotated, and a Ranger optimizer is used in the training, and when a preset training progress is reached, a cosine-based learning rate decay is started to complete the training;

[0066] The CNN model in step S4 is obtained through pre-training, and the training process includes:

[0067] Collect visual data from the crawl, including , , , , , and ; Then the result of each grasping is marked as a supervision signal, where a marking result of 1 indicates a successful grasping, and a marking result of 0 indicates a failed grasping; thus forming training sample data for training the CNN model;

[0068] Then, a CNN network structure consisting of multiple convolutional layers, pooling layers, and fully connected layers is constructed as a model, and the model is trained to learn the mapping from 6D pose and mask map to the probability of successful grasping;

[0069] The visual data F captured for the i-th time i It is expressed as:

[0070] ;

[0071] in , , , , , and Respectively represent the visual data in the i-th grasp , , , , , and ;

[0072] Then the input of the CNN model is ,in is the total number of training samples;

[0073] The output of the CNN model is the success probability of each grasping strategy:

[0074] ;

[0075] in, Indicates successful crawling. Indicates that the crawl failed;

[0076] The model is trained using the Adam optimizer to obtain the trained CNN model, in which the cross entropy loss function is used as the loss function :

[0077] ;

[0078] in, It is The labeling result of the training sample is 0 or 1. The model is The prediction of the success probability of each sample.

[0079] On the other hand, the present invention also provides a method for estimating 6D pose of an object based on an attention mechanism and a robot intelligent grasping device, which are used in the aforementioned method, including:

[0080] The target area perception module performs perception processing on the RGB image according to the position of the object to be grasped in the RGB image, and obtains a target area image of a standard size with the object to be grasped as the target area;

[0081] The backbone network module receives the output of the target area perception module, extracts the features of the target area image based on the fusion of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism, and obtains the network prediction intermediate features including the coordinate map, mask map, surface area attention map and dense correspondence map of the object to be grasped;

[0082] The pose regression module receives the output of the backbone network module and regresses the 6D pose of the object to be grasped through the neural network structure;

[0083] The robot-evaluable intelligent grasping module receives the output of the pose regression module and evaluates the success probability of the robot's grasping strategy for the grasped object. It then performs the grasping or adjusts the robot's end position to perform a new grasp based on the success probability.

[0084] The technical effects of the present invention are as follows: (1) The technical solution of the present invention provides a half-wavelet pyramid attention mechanism, which innovatively constructs a dual-stream feature interaction channel in the spatial domain and the wavelet domain and integrates it into the ResNet-34 architecture. The mechanism uses the pyramid structure to simultaneously extract and fuse the spatial domain and wavelet domain features of the target, effectively improving the accuracy and anti-occlusion performance of 6D pose estimation.

[0085] (2) The technical solution of the present invention provides a multi-scale feature extraction attention mechanism, which is integrated into the residual block of ResNet-34, realizing multi-scale perception capabilities from microscopic local details to macroscopic overall structures, with excellent feature extraction capabilities, and greatly improving the robustness of 6D pose estimation in cluttered and occluded scenes.

[0086] (3) The end-to-end object 6D pose estimation method based on the attention mechanism provided by the technical solution of the present invention can obtain reliable 6D pose results when dealing with occlusion, thereby further effectively improving the reliability of the robot grasping task. The benchmark evaluation on the LINEMOD and LM-O datasets verifies that the method provided by the present invention achieves competitive performance.

[0087] (4) The technical solution of the present invention provides a robot-evaluable intelligent grasping module that can evaluate the current grasping operation and determine whether to execute the current grasping or whether the robot itself randomly adjusts the end position within a very small range to execute a new grasping strategy, so as to achieve intelligent grasping with a high success rate.

[0088] The present invention realizes intelligent control of robot grasping that can be applied to fields such as industrial production by processing visual images, and has excellent application prospects in fields such as robot operation, augmented reality (AR) systems and industrial automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a flow chart of a method for estimating 6D pose of an object and a robot intelligent grasping system based on an attention mechanism in one embodiment of the present invention;

[0090] Figure 2 A schematic diagram of the network structure of a method for estimating 6D pose of an object based on an attention mechanism in one embodiment of the present invention;

[0091] Figure 3 A schematic diagram of the network structure of a half-wavelet pyramid attention mechanism in one embodiment of the present invention;

[0092] Figure 4 A schematic diagram of a network structure of a multi-scale feature extraction attention mechanism in one embodiment of the present invention;

[0093] Figure 5This is an experimental visualization diagram of an embodiment of the present invention on the LINEMOD-Occluded dataset;

[0094] Figure 6 Schematic diagram of the CNN network structure of a robot-evaluable intelligent grasping module in one embodiment of the present invention. DETAILED DESCRIPTION

[0095] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0096] See also Figure 1 In one embodiment of the present invention, a method for estimating a 6D pose of an object based on an attention mechanism is provided, comprising the following steps:

[0097] S1, obtain an RGB image including the object to be grasped, and then perceive the object to be grasped in the RGB image based on the perception model, so as to enlarge the area where the object to be grasped is located as the target area. The processed target area image is used as the input of the backbone network. Specifically:

[0098] The existing two-dimensional target recognition model is used to analyze the input RGB image in real time and accurately locate the coordinates of the area of ​​the object to be captured. The two-dimensional target recognition model can use target detection models such as Detectron2 framework, fast cnn model, Adelaidet framework, yolo model, etc.

[0099] Subsequently, the target area image obtained by the detection is standardized according to a preset size, so as to be adjusted to a uniform pixel size. In this embodiment, the target area in the original RGB image is adjusted to a uniform 256×256 pixel size. This processing step helps to ensure input consistency in the subsequent feature extraction process, improve the accuracy of target recognition and system operation efficiency.

[0100] S2, based on the input target area image, uses a backbone network that integrates the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism to extract features of the object to be grasped. Among them, the half-wavelet pyramid attention mechanism simultaneously collects and fuses spatial domain and wavelet domain features through a hierarchical pyramid structure. The multi-scale feature extraction attention mechanism uses differentiated receptive fields and multi-branch attention design to effectively extract the multi-scale features of the object. Then, the spatial domain and wavelet domain features are combined to obtain the network prediction intermediate features, including the coordinate map, mask map, surface area attention map, and dense correspondence map of the object to be grasped.

[0101] In this step, the half-wavelet pyramid attention mechanism is integrated into the ResNet-34 architecture, specifically set between the residual blocks in ResNet-34, so as to use the pyramid structure to extract and fuse the wavelet domain and spatial domain features of the target in parallel, thereby improving the accuracy of the pose estimation results. Specifically:

[0102] See also Figure 3 , the half-wavelet pyramid attention mechanism takes the input features Through the pyramid structure Segmentation is performed, where R represents the real number space, C represents the number of channels, H represents the height, and W represents the width. That is, each layer of the pyramid structure will input features along and Direction average feature vectors, the number of segments Based on the number of pyramid levels It shows quadratic growth, that is ,in Indicates As the number of pyramid structures increases, more refined feature vectors can be obtained through segmentation, while the first layer of the pyramid structure does not segment. , so the first layer of the pyramid structure obtains the global feature map, and the subsequent pyramid layers divide the feature map into smaller dimensions, that is, the local feature map is obtained. Then, each layer of the pyramid structure applies the half-wavelet attention mechanism to the feature map obtained by itself. The segmented pyramid structure then performs an inverse segmentation operation based on the spatial position encoding information at the time of segmentation. The inverse segmentation is specifically as follows: first, the position of each local feature map, that is, the small block, is recorded during segmentation, and the order index of each small block is recorded in row priority. Then, after the half-wavelet attention mechanism is applied to each small block separately, the small blocks are rearranged according to the recorded order index, so as to be merged back to the original size. Then, in Generated , and generate the Pyramid Attention Map Finally, the pyramid attention maps of different layers are Fusion to obtain the final . Figure 3 A two-layer pyramid structure is adopted in the present invention, and during the specific implementation, a pyramid structure with more layers can be constructed as needed.

[0103] The half-wavelet attention mechanism used in the half-wavelet pyramid attention mechanism is specifically:

[0104] The single feature map in each pyramid is the minimum dimensional feature It is divided into two parts, namely the spatial domain feature branch and wavelet domain feature branch .in Used to extract features in the spatial domain. Then the features in the wavelet domain are obtained by discrete wavelet transform DWT , which is used to extract features in the wavelet domain. In the wavelet domain, Convolution is performed through two 3×3 convolution kernels to obtain the convolution result. Then, we use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The two weighted results are then added together, and the result after addition is convolved with a convolution kernel of size 1. Then the result of the addition is obtained by inverse wavelet transform , Reconciliation After concatenation, it is convolved with a 3×3 convolution kernel and then activated with a ReLU function. Add them together and finally get .

[0105] The discrete wavelet transform decomposes the feature sequence (feature map) into low-frequency components of different scales through multi-resolution analysis. and high frequency components , the expression is:

[0106] Low frequency component:

[0107] ;

[0108] High frequency components:

[0109] ;

[0110] in, is the input feature sequence, represents the number of eigenvectors, is the scaling function, defined as , used to extract the low-frequency information of the features, is a wavelet function, defined as , used to extract high-frequency information of features, is the number of decomposition layers, indicating the scale level of wavelet transform, is the translation parameter, which indicates the position of the wavelet in time or space.

[0111] The spatial attention mechanism mentioned above is specifically:

[0112] ;

[0113] in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, It represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation.

[0114] The channel attention mechanism mentioned above is specifically:

[0115] ;

[0116] in, represents the result of the channel attention mechanism, represents the sigmoid activation function, and represents two convolution operations using 1×1 filters, represents the average pooling layer, and ReLU represents the ReLU activation function.

[0117] The inverse wavelet transform mentioned above can reconstruct the original feature sequence (feature map), and the expression is:

[0118] ;

[0119] in, is the maximum number of decomposition levels, For the The approximation coefficient of the layer represents the low-frequency global characteristics of the feature, For the The detail coefficient of the layer represents the local variation of features at different resolutions.

[0120] In this step, the multi-scale feature extraction attention mechanism mentioned above is integrated into the first three residual blocks of ResNet-34, replacing the original third convolutional layer in the residual block. The multi-scale feature extraction attention mechanism of this embodiment adopts differentiated receptive field and multi-branch attention design to effectively extract the multi-scale features of the object, specifically:

[0121] See also Figure 4In the first stage, the feature maps obtained after the residual block convolution operation are normalized by BatchNorm preprocessing. Then a multi-scale convolutional architecture is applied: (1) the channel dimension is adjusted by point convolution (PWConv), followed by a 5×5 standard convolution to extract basic features; (2) three depthwise separable dilated convolutions (DWDConv7×7, DWDConv5×5, and DWDConv3×3) are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields, and the outputs of the three convolutions are concatenated along the channel dimension to double the feature depth. Finally, a nonlinear MLP (including two PWConv layers with GELU activation function) fuses the multi-scale features, while the residual connection coordinates the original and processed features across scales.

[0122] In the second stage, the output of the first stage is first processed by BatchNorm. Then, the weighting coefficients are obtained and spliced ​​through the three-branch attention architecture, namely the pixel attention mechanism, the channel attention mechanism and the simple pixel attention mechanism. After that, the spatial attention mechanism is used for weighting. The spatial attention mechanism here is the same as the semi-wavelet pyramid attention mechanism. Finally, a nonlinear MLP is used to fuse the multi-scale features and add the output of the MLP to the original input of the second stage to output the final result.

[0123] The channel attention mechanism of the second stage in the multi-scale feature extraction attention mechanism is as follows:

[0124] ;

[0125] in, represents the result of the channel attention mechanism in the second stage, represents the sigmoid activation function, represents the average pooling layer, and Represents two convolution operations, with a convolution kernel of 1; represents a given feature; GELU represents the GELU activation function.

[0126] The pixel attention mechanism is specifically:

[0127] ;

[0128] in, represents the result of the pixel attention mechanism, represents the sigmoid activation function, and Represents two convolution operations with a convolution kernel of 1.

[0129] The simple pixel attention mechanism is as follows:

[0130] ;

[0131] in, represents the result of a simple pixel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and Represents two convolution operations, the convolution kernel is 1, and Represents a convolution operation with a convolution kernel of 3.

[0132] Then, the spatial domain and wavelet domain features are combined to obtain the network prediction intermediate features, that is, the backbone network predicts the surface area attention map of the multi-domain fusion of the object to be grasped based on the RGB image. , Multi-domain fusion object coordinate map , Dense correspondence graph of multi-domain fusion And the mask map of multi-domain fusion .

[0133] The surface area attention map of multi-domain fusion It is based on the spatial domain and wavelet domain, through the object coordinate map The farthest point on is sampled.

[0134] Object coordinate diagram of multi-domain fusion It is estimated based on the spatial domain and wavelet domain by rendering the CAD model in the target pose.

[0135] Dense correspondence graph for multi-domain fusion is based on The 3D bounding box size of the object to be grasped , , , normalize each channel to [0,1], and then pass Add 2D pixel coordinates to construct .

[0136] Mask map for multi-domain fusion It is obtained by performing semantic segmentation on the grasped objects based on the spatial domain and wavelet domain.

[0137] S3, the obtained multi-domain fused surface area attention map and the multi-domain fused dense correspondence relationship map are input into the pose regression module, and the 6D pose information of the object to be grasped is directly regressed based on the neural network structure of the pose regression module. Specifically:

[0138] The obtained multi-domain fused surface area attention map and the multi-domain fused dense correspondence map are spliced ​​and input into the pose regression module. The pose regression module in this embodiment is a neural network structure composed of several convolutional layers and fully connected layers, which is used to regress the 6D pose information of the object to be grasped, including the position and posture.

[0139] First, we get , and :

[0140] ;

[0141] in, , , For the pose regression module , , The relative translation prediction results in the direction, , The center of the object to be grasped is , The coordinates in the direction, , The center of the known detection bounding box is , The coordinates in the direction, , is the width and height of the known detection bounding box. is the actual depth value to be solved, It is the scaling factor used by the target area perception module to magnify the image.

[0142] Then calculate the translation of the object to be grasped by back projection :

[0143] ;

[0144] in is the intrinsic parameter matrix of a given camera, thereby regressing the position of the object to be grasped.

[0145] According to the multi-domain fused surface area attention map of the object to be grasped and the multi-domain fused dense correspondence map, the rotation matrix of the object to be grasped can be calculated according to the following formula:

[0146] ;

[0147] in, , , The three column vectors representing the rotation matrix, , To splice the and Input the first three values ​​and the last three values ​​of the rotation result predicted by the pose regression module, It is a normalization operation.

[0148] At the same time, this embodiment constructs the corresponding loss function, which constrains the rotation information , 2D center offset information , depth information and geometric information , the loss functions of the four are as follows:

[0149] ;

[0150] in, and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, , Indicates the first 3D models, L ce represents the cross entropy loss, ⊙ represents the Hadamard product operation, and ‖·‖ represents the norm operation.

[0151] In particular, for symmetrical objects, Expands to:

[0152] ;

[0153] in, represents all valid rotations in the symmetric case.

[0154] Furthermore, based on the above loss function, the overall loss function constructed in this embodiment is:

[0155] ;

[0156] The calculation method based on the target translation prediction result is obtained , , and t, the calculation method of the rotation matrix and the overall loss function are used to regress the 6D pose information of the object to be grasped.

[0157] S4, see Figure 6 The obtained 6D pose information is sent to the robot's evaluable intelligent grasping module to enable the robot to grasp the object to be grasped. In addition, the module can predict the success probability of grasping based on the geometric features, position information and pose information of the object to be grasped, so as to help the robot make a more intelligent grasping strategy.

[0158] Specifically, we collect visual data for each grasping task: object mask map (reflects the geometric characteristics of the object), the predicted value of the object translation result , , (reflects the position information of the object) and the predicted value of the object rotation result , , (Reflects the relative relationship between the object and the environment), annotates the result of each grasping (success or failure) as a supervision signal, success (1): grasping is successful, the object is stably grasped and the task is completed, failure (0): grasping fails, the object is not successfully grasped, or the task is not completed.

[0159] Using the CNN network structure, which consists of multiple convolutional layers, pooling layers, and fully connected layers, the training model learns the mapping from the object mask map, the object translation result prediction value, and the object rotation result prediction value to the grasping success probability. Using the trained CNN model, the success probability of the grasping strategy is evaluated in real time, and intelligent decisions are made, that is, whether to execute the grasping or the robot itself randomly adjusts the end position in a very small range to form a new grasping strategy to continue to judge the success probability.

[0160] The visual data F captured each time i It can be expressed as:

[0161] ;

[0162] The subscript i of each symbol in the above formula represents the corresponding data at the i-th crawl.

[0163] Then the input of the CNN model is:

[0164] ;

[0165] The output of the CNN model is the success probability of each grasping strategy:

[0166] ;

[0167] in, Indicates that the crawl is successful, otherwise, Indicates that the crawl failed.

[0168] The present invention uses the Adam optimizer for training in the robot evaluable intelligent grasping module, and adopts the cross entropy loss function as the loss function, specifically:

[0169] ;

[0170] in, It is The true label of each sample is 0 or 1 (failure or success). The model is The prediction of the success probability of each sample, is the number of training samples.

[0171] In the crawling task, the system will , using the trained CNN model to predict the success probability of the grasping strategy. If the success probability is not lower than the set threshold, the robot executes the strategy; if the success probability is lower than the threshold, the robot randomly adjusts the end position in a very small range, and then predicts the success probability of grasping again, and executes the cycle until the success probability is not lower than the set threshold, then the corresponding grasping strategy is executed.

[0172] Furthermore, in steps S1-S3, the perception model, the backbone network and the neural network are jointly trained based on an RGB image dataset containing objects with 6D pose annotations. Specifically, in this embodiment, the widely used LINEMOD dataset and LINEMOD-Occluded dataset are selected for training, and can also be replaced by self-made private datasets in specific implementations. The LINEMOD dataset contains 15 objects with different colors, shapes and sizes, and 6D pose annotations are provided for 13 of them. The dataset is divided into 85% real test images and 15% real training images, and synthetic images are enhanced for each object. The LINEMOD-Occluded dataset is an expansion of the LINEMOD dataset, which contains 8 objects and has enhanced occlusion. During the training process, the real annotations are combined with synthetic samples of each object.

[0173] The model trained in steps S1-S3 of this embodiment is developed on PyTorch. The training configuration of this embodiment adopts the Ranger optimizer (batch size = 24, learning rate = 0.0001), and starts the cosine-based learning rate decay after the training progress reaches 72%.

[0174] For comparison, all the models mentioned below in this embodiment are trained, verified and tested on the GPU of RTX4090 using the PyTorch framework and CUDA=11.1. In order to verify the superiority of the method provided in this embodiment, the following experiments were conducted:

[0175] (1) Comparative experiment between the method provided in this embodiment and other models:

[0176] Table 1 lists the quantitative results of the method provided by the present invention and six SOTA algorithms in the LINEMOD benchmark test based on the ADD(-S) indicator under the same environment. Table 2 shows the performance of the method provided by the present invention and six SOTA algorithms on the LINEMOD-Occluded occlusion dataset under the same environment, evaluated using the ADD(-S) standard. The experimental results show that the method provided by the present invention surpasses all SOTA methods in ADD(-S) performance under standard conditions and occlusion conditions.

[0177] Table 1 Comparison results of the method provided by the present invention and other SOTA algorithms on LINEMOD.

[0178] method PoseCNN Pix2Pose DPOD PVNet CDPN GDR-Net Method of the present invention Ape Model - 58.10 53.28 43.62 - - 81.62 Bench - 91.00 95.34 99.90 - - 99.03 camera - 60.90 90.36 86.86 - - 95.78 kettle - 84.40 94.10 95.47 - - 98.13 Cat Model - 65.00 60.38 79.34 - - 94.81 Electric drill - 76.30 97.72 96.43 - - 96.93 Duck Model - 43.80 66.01 52.58 - - 83.29 Egg Carton - 96.80 99.72 99.15 - - 99.53 glue - 79.40 93.83 95.66 - - 99.42 Punch Machine - 74.80 65.83 81.92 - - 88.68 Iron - 83.40 99.80 98.88 - - 97.96 desk lamp - 82.00 88.11 99.33 - - 99.23 Telephone - 45.00 74.24 92.41 - - 93.58 average value 62.7 72.38 82.98 86.27 89.86 93.70 94.46

[0179] (“-” means the result is not available).

[0180] Table 2 Comparison results of the method provided by the present invention and other SOTA algorithms on LINEMOD-Occluded.

[0181] method PoseCNN Pix2Pose DPOD PVNet GeoPose GDR-Net Method of the present invention Ape Model 9.60 22.00 - 15.18 19.07 46.80 50.94 kettle 45.20 44.70 - 63.30 65.46 90.80 90.89 Cat Model 0.90 22.70 - 16.68 18.83 40.50 40.94 Electric drill 41.40 44.70 - 65.65 65.86 82.60 85.83 Duck Model 19.60 15.00 - 25.24 28.47 46.90 52.06 Egg Carton 22.00 25.20 - 50.17 54.73 54.20 61.36 glue 38.50 32.40 - 49.62 56.77 75.80 78.85 Punch Machine 22.10 49.50 - 39.67 49.21 60.10 75.37 average value 24.90 32.00 32.79 40.77 44.80 62.20 67.03

[0182] (“-” means the result is not available).

[0183] The verification results of the LINEMOD-Occluded dataset are visualized as follows Figure 5 As shown in Table 1 and Table 2, the method provided by the present invention improves ADD(-S) by 0.76% on the LINEMOD dataset and by 4.83% on the LINEMOD-Occluded dataset compared to the baseline model GDR-Net.

[0184] The above experimental data fully verify the breakthrough performance of the model framework provided by the present invention in solving complex 6D object pose estimation tasks.

[0185] (2) Ablation experiment of the method provided by the present invention:

[0186] Table 3 lists the ablation experimental results of the method provided by the present invention on the LINEMOD dataset under different levels of ADD(-S) under the same environment.

[0187] Table 3 Ablation experiment results of the method provided by the present invention.

[0188]

[0189] The experimental results in Table 3 reveal the functional complementarity between the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism. When implemented separately, each mechanism shows unique performance advantages: the half-wavelet pyramid attention mechanism improves the accuracy to 93.83% (0.1 ADD(-S)), which is due to its ability to simultaneously collect and integrate the spatial domain and wavelet domain features of the object, allowing the network to obtain more diverse feature information for 6D pose estimation. At the same time, the multi-scale feature extraction attention mechanism captures the multi-scale features and contextual relationships of the object, achieving an accuracy of 93.89% (0.1 ADD(-S)), which is particularly effective in solving the occlusion ambiguity problem commonly seen in complex scenes.

[0190] It is worth noting that the combination of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism produces an excellent synergistic effect. The combination of the two achieves a peak performance of 94.46%, which cannot be explained by a simple additive effect. This synergistic effect stems from two interdependent mechanisms: (1) The multi-receptive field features of the multi-scale feature extraction attention mechanism provide rich input for the attention weight calculation of the half-wavelet pyramid attention mechanism; (2) The multi-domain features extracted by the half-wavelet pyramid attention mechanism also provide diverse inputs for the multi-scale feature extraction attention mechanism.

[0191] On the other hand, see Figure 2 In one embodiment, the present invention also provides a robot object grasping device based on 6D pose estimation of an object, comprising:

[0192] The target area perception module enlarges the local area of ​​the image according to the position of the object to be grasped in the RGB image.

[0193] The backbone network module extracts multi-domain (spatial domain and wavelet domain) and multi-scale features of the grasped object based on the input RGB image and the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism, and predicts the intermediate features.

[0194] The pose regression module directly regresses the 6D pose of the object to be grasped through a neural network structure based on the intermediate features.

[0195] The robot's evaluable intelligent grasping module receives the 6D pose information of the object to be grasped and grasps the object. At the same time, it can evaluate the grasping operation, predict the success rate of the grasping, and then determine whether to execute the grasping or randomly adjust the end position of the robot in a very small range to execute a new grasping strategy.

[0196] The device runs on a structure composed of a robot, an end effector, and an industrial camera, and adopts an "eye in hand" visual model. Through robot vision calibration, the transformation matrix from the camera coordinate system to the robot end effector coordinate system can be obtained. Given an RGB image, the 6D pose information of the object to be grasped is first obtained, and then the information is converted through the robot hand-eye calibration matrix. Finally, the converted 6D pose is sent to the robot system to perform the grasping task. When performing the grasping task, the grasping operation can be evaluated, the success rate of the grasping can be predicted, and then it can be determined whether to perform this grasping or the robot itself randomly adjusts the end position in a very small range to execute a new grasping strategy.

[0197] The above is a detailed introduction to a 6D pose estimation method for objects based on an attention mechanism and a robot intelligent grasping system provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A robot object grasping method based on 6D pose estimation of an object, characterized in that: The following steps are involved: S1, obtaining an RGB image including the object to be grasped, and performing perception processing on the RGB image by a perception model to obtain a target area image of a standard size with the object to be grasped as the target area; S2, extracting the features of the target area image based on a backbone network that integrates a half-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism, wherein the half-wavelet pyramid attention mechanism simultaneously collects and integrates spatial domain and wavelet domain features through a hierarchical pyramid structure; the multi-scale feature extraction attention mechanism uses differentiated receptive fields and multi-branch attention to extract multi-scale features; and then combining the spatial domain and wavelet domain features to obtain the network prediction intermediate features, including a coordinate map, a mask map, a surface area attention map, and a dense correspondence map of the object to be grasped; S3, based on a neural network composed of multiple convolutional layers and fully connected layers, processes the surface area attention map and the dense correspondence map to regress the 6D pose including the position and posture of the object to be grasped; S4, inputs the 6D pose and mask image into the CNN model, evaluates the success probability of the robot grasping the object to be grasped, and executes the grasping or adjusts the robot end position to perform a new grasp based on the success probability.

2. The method according to claim 1, characterized in that In step S1, the step of perceiving and processing the image includes: The existing two-dimensional target recognition model is used to perform real-time analysis on the image including the object to be grasped, so as to obtain the regional coordinates of the object to be grasped in the image, and extract the original image area in the image according to the regional coordinates; Then, the original image area is normalized according to a preset size, so as to adjust the original image area to a preset uniform pixel size.

3. The method according to claim 1, characterized in that In the step S2, the backbone network is a ResNet-34 structure, the half-wavelet pyramid attention mechanism is set between the residual blocks of ResNet-34, and the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block.

4. The method according to claim 3, characterized in that In step S2, the half-wavelet pyramid attention mechanism is: The half-wavelet pyramid attention mechanism includes a multi-layer pyramid structure, and each layer of the pyramid structure receives the features obtained after the residual block convolution operation separately. , and each layer of the pyramid structure will along and The direction is evenly divided into feature vectors, where R represents the real space, C represents the number of channels, H represents the height, and W represents the width; each feature vector is processed by the half-wavelet attention mechanism and then de-segmented to merge to obtain the output of each layer, and finally the output of each layer is fused as the output of the half-wavelet pyramid attention mechanism ; where n is determined by the level i of the pyramid structure where the segmentation is performed, n=i 2 ; The half-wavelet attention mechanism is: The feature vector input to the half-wavelet attention mechanism is ,Will Divided into spatial domain feature branches and wavelet domain feature branch Two parts, including Used to extract features in the spatial domain, The features in wavelet domain are obtained by discrete wavelet transform DWT , and then in the wavelet domain Convolution is performed through two 3×3 convolution kernels to obtain the convolution result. ; Then use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The weighted results are then concatenated and the concatenated results are convolved with a convolution kernel of size 1. Add them; the result of the addition is then obtained by inverse wavelet transform ; and After concatenation, it passes through a 3×3 convolution kernel and a ReLU activation function, and then Add them together to get ; The spatial attention mechanism is: ； in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, Represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation; The channel attention mechanism is: ； in, represents the result of the channel attention mechanism, and Respectively represent the convolution operation using a 1×1 filter, represents the average pooling layer, and ReLU represents the ReLU activation function.

5. The method according to claim 3, characterized in that: In step S2, the multi-scale feature extraction attention mechanism includes: the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block; In the first stage, the feature map obtained after the residual block convolution operation is normalized, and then a multi-scale convolution architecture is applied: 1) Adjust the channel dimension through point convolution PWConv, and then perform 5×5 standard convolution to extract basic features; 2) Three depth-wise separable dilated convolutions are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields. The outputs of the three depth-wise separable dilated convolutions are concatenated along the channel dimension to double the feature depth. Finally, a nonlinear MLP fuses multi-scale features, and a residual connection is used to add the original input of the first stage and the output of the MLP to achieve cross-scale collaboration. In the second stage, the output of the first stage is first normalized; then, the weighting coefficients are obtained and spliced ​​through a three-branch attention architecture, namely the pixel attention mechanism, the channel attention mechanism, and the simple pixel attention mechanism; then, the spatial attention mechanism is used for weighting; finally, a nonlinear MLP is used to fuse multi-scale features, and the output of the MLP is added to the original input of the second stage to output the final result; The channel attention mechanism in the second stage is: ； in, represents the result of the second-stage channel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a given feature; GELU represents the GELU activation function; The pixel attention mechanism is: ； in, represents the result of the pixel attention mechanism, and They represent the convolution operation with the convolution kernel of 1 and the convolution kernel of 1 respectively; A simple pixel attention mechanism is: ； in, represents the result of a simple pixel attention mechanism, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a convolution operation with a convolution kernel of 3.

6. The method according to claim 1, characterized in that In step S2, the network predicts the intermediate features as follows: Multi-domain fusion object coordinate map of the object to be grasped , surface area attention map of multi-domain fusion , Dense correspondence graph of multi-domain fusion And the mask map of multi-domain fusion ; in It is estimated based on the spatial and wavelet domains by rendering the CAD model in the target pose; It is based on the spatial domain and wavelet domain, through the object coordinate map Generate by sampling the farthest point on; is through Each channel of the 3D bounding box size of the object in is normalized to [0,1], so that the side length of the 3D bounding box of the object in the object coordinate space is 1, and then Add 2D pixel coordinates to construct; It is obtained by performing semantic segmentation on the grasped objects based on the spatial domain and wavelet domain.

7. The method according to claim 6, characterized in that The step S3 comprises: Will and After splicing, the images are input into a neural network composed of several convolutional layers and fully connected layers as a pose regression module to regress the 6D pose including the position and posture of the object to be grasped. The translation amount t of the object to be grasped is calculated by the following process: First, we get , and : ； in, , , For the pose regression module , , The relative translation prediction results in the direction, , The center of the object to be grasped is , The coordinates in the direction, , The center of the known detection bounding box is , The coordinates in the direction, , is the width and height of the known detection bounding box; is the actual depth value to be solved, The scaling factor for the target area perception module to magnify the image; Then calculate the translation of the object to be grasped by back projection : ； in is the intrinsic parameter matrix of a given camera; Then calculate the rotation matrix of the object to be grasped according to the following formula: ； in, , , Three column vectors representing the rotation matrix; , To splice the and Input to the pose regression module to predict the first three values ​​and the last three values ​​of the rotation result. It is a normalization operation; Reconstruct L for constraining the rotation matrix information R , L used to constrain 2D center offset information center , L used to constrain depth information depth and L used to constrain geometric information geometry There are four loss functions: ； in and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, Indicates the first 3D models, L ce represents the cross entropy loss, ⊙ represents the Hadamard product operation, ‖·‖ represents the norm operation; and the overall loss function is established for: ； Then according to the pose regression module , , Relative translation prediction results in direction , , , translation , the calculation method of the rotation matrix and the overall loss function are used to regress the 6D pose information of the object to be grasped.

8. The method according to claim 7, characterized in that The step S4 comprises: Input crawled data into the CNN model as well as , , , , and , to evaluate the success probability of grasping in real time, and then decide whether to execute the current grasping or to perform a new grasping after randomly adjusting the end position of the robot: if the success probability is not lower than the preset threshold, the current grasping is performed; if the success probability is lower than the preset threshold, the robot randomly adjusts the end position and re-evaluates the success probability of grasping, and executes the cycle until the success probability of grasping is not lower than the preset threshold, and then the grasping is performed.

9. The method according to claim 8, characterized in that The perception model, backbone network and neural network in steps S1-S3 are obtained by joint training based on an RGB image dataset containing objects with 6D poses annotated, and the Ranger optimizer is used in the training, and when the preset training progress is reached, the cosine-based learning rate decay is started to complete the training; The CNN model in step S4 is obtained through pre-training, and the training process includes: Collect visual data from the crawl, including , , , , , and ; Then the result of each grasping is marked as a supervision signal, where a marking result of 1 indicates a successful grasping, and a marking result of 0 indicates a failed grasping; thus forming training sample data for training the CNN model; Then, a CNN network structure consisting of multiple convolutional layers, pooling layers, and fully connected layers is constructed as a model, and the model is trained to learn the mapping from 6D pose and mask map to the probability of successful grasping; The visual data F captured for the i-th time i It is expressed as: ； in , , , , , and Respectively represent the visual data in the i-th grasp , , , , , and ; Then the input of the CNN model is ,in is the total number of training samples; The output of the CNN model is the success probability of each grasping strategy: ； in, Indicates successful crawling. Indicates that the crawl failed; The model is trained using the Adam optimizer to obtain the trained CNN model, in which the cross entropy loss function is used as the loss function : ； in, It is The labeling result of the training sample is 0 or 1. The model is The prediction of the success probability of each sample.

10. A 6D pose estimation method for an object based on an attention mechanism and a robot intelligent grasping device, characterized in that: Used to perform the method according to any one of claims 1 to 9, comprising: The target area perception module performs perception processing on the RGB image according to the position of the object to be grasped in the RGB image, and obtains a target area image of a standard size with the object to be grasped as the target area; The backbone network module receives the output of the target area perception module, extracts the features of the target area image based on the fusion of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism, and obtains the network prediction intermediate features including the coordinate map, mask map, surface area attention map and dense correspondence map of the object to be grasped; The pose regression module receives the output of the backbone network module and regresses the 6D pose of the object to be grasped through the neural network structure; The robot-evaluable intelligent grasping module receives the output of the posture regression module and evaluates the success probability of the robot executing the grasping strategy on the grasped object. It executes the grasping or adjusts the robot end position to perform a new grasping according to the success probability.

Images