A robot object grasping method and device based on 6D pose estimation of the object

Abstract

The present invention discloses a robot object grasping method and device based on 6D pose estimation of objects. First, the RGB image is sensed and processed to obtain the target area image of the object to be grasped. Then, the features of the target area image are extracted based on a backbone network that integrates the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism. The features are then processed based on a neural network composed of multiple convolutional layers and fully connected layers, thereby regressing the 6D pose of the object to be grasped. Finally, the success probability of the robot grasping the object to be grasped is evaluated based on the CNN model, and the grasping is performed or the end position of the robot is adjusted to perform a new grasp according to the success probability. The present invention provides a new idea for solving the problem of occlusion robustness. More importantly, it constructs a new paradigm of spatial-frequency domain feature fusion and intelligent grasping, which greatly improves the robustness of 6D pose estimation in cluttered and occluded scenes, and has excellent application prospects in industrial vision.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and robotics, and in particular relates to a robot object grasping method and device based on 6D pose estimation of the object. Background Art

[0002] 6D pose estimation has become a cornerstone technology for robotic manipulation, augmented reality (AR) systems, and industrial automation. Emerging innovations in the multi-layer perceptron (MLP) paradigm have significantly improved the accuracy of RGB-based pose estimation frameworks, particularly through innovative network architectures and sophisticated feature representation methods. The LINEMOD dataset and its occluded variants have become standard benchmarks, driving continued progress in this field.

[0003] Current approaches primarily use convolutional neural networks (CNNs) for indirect or direct pose regression. Indirect approaches first establish correspondences between 2D positions and 3D object keypoints and then estimate the 6D pose using a PnP algorithm. Direct approaches employ a holistic end-to-end pose regression architecture, where a deep convolutional network computes the 6D pose of an object directly from raw visual input. Pioneering work such as PoseCNN has established baseline performance by learning global features, while PVNet introduced a vector field representation to improve occlusion handling. As demonstrated by CBAM, attention mechanisms have shown potential in improving feature extraction. The approach presented by PANet, using a pyramid pixel-level attention module to extract rich appearance features from RGB images, effectively improves the localization error tolerance of the 6D pose estimation task.

[0004] However, current 6D pose estimation schemes still have some key limitations. The PnP algorithm in indirect methods usually takes a long time to estimate the pose of an object and cannot be applied to many tasks that require differentiable pose estimation, while direct methods are sensitive to noise and require a large amount of labeled data. In addition, the existing attention mechanism mainly acts on the normal domain of the object, ignoring the more diverse complementary wavelet domain features. In addition, multi-scale feature integration usually adopts a simpler structure and cannot establish the best cross-scale relationship. The performance of some existing works has declined under severe occlusion, especially when the object is partially or completely occluded, and cannot fully meet the needs of robots for high-success-rate intelligent grasping in complex scenes. Summary of the Invention

[0005] In view of the significant defects of the 6D pose estimation scheme currently used in robot grasping in terms of occlusion robustness, feature diversity acquisition and multi-scale feature extraction, as well as the lack of intelligence in robot grasping, the present invention provides a robot object grasping method and device based on 6D pose estimation of objects with high accuracy, anti-occlusion and robustness.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the present invention include:

[0007] In one aspect, the present invention provides a robot object grasping method based on 6D pose estimation of the object, comprising the following steps:

[0008] S1, obtaining an RGB image including the object to be grasped, performing perception processing on the RGB image using a perception model, and obtaining a target area image of a standard size with the object to be grasped as the target area;

[0009] S2, extracts features of the target area image based on a backbone network that integrates a half-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism. The half-wavelet pyramid attention mechanism simultaneously collects and fuses spatial and wavelet domain features through a hierarchical pyramid structure; the multi-scale feature extraction attention mechanism uses differentiated receptive fields and multi-branch attention to extract multi-scale features. The spatial and wavelet domain features are then combined to obtain network prediction intermediate features, including a coordinate map, a mask map, a surface area attention map, and a dense correspondence map of the object to be grasped.

[0010] S3, based on a neural network composed of multiple convolutional layers and fully connected layers, processes the surface area attention map and the dense correspondence map to regress the 6D pose including the position and posture of the object to be grasped;

[0011] S4 inputs the 6D pose and mask image into the CNN model to evaluate the success probability of the robot grasping the object to be grasped, and executes the grasp or adjusts the robot end position to perform a new grasp based on the success probability.

[0012] Furthermore, in step S1, the step of perceiving and processing the image includes:

[0013] The existing two-dimensional target recognition model is used to perform real-time analysis on the image including the object to be grasped, thereby obtaining the regional coordinates of the object to be grasped in the image, and extracting the original image area in the image according to the regional coordinates;

[0014] Then, the original image area is normalized according to a preset size, so as to adjust the original image area to a preset uniform pixel size.

[0015] Furthermore, in step S2, the backbone network is a ResNet-34 structure, the half-wavelet pyramid attention mechanism is set between the residual blocks of ResNet-34, and the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block.

[0016] Furthermore, in step S2, the half-wavelet pyramid attention mechanism is:

[0017] The half-wavelet pyramid attention mechanism includes a multi-layer pyramid structure, and each layer of the pyramid structure receives the features obtained after the residual block convolution operation separately. , and the pyramid structure of each layer will along and The direction is evenly divided into feature vectors, where R represents the real space, C represents the number of channels, H represents the height, and W represents the width; each feature vector is processed by the half-wavelet attention mechanism and then de-segmented to merge to obtain the output of each layer, and finally the output of each layer is fused as the output of the half-wavelet pyramid attention mechanism ; where n is determined by the number of layers i of the pyramid structure where the segmentation is performed, n=i 2 ;

[0018] The half-wavelet attention mechanism is:

[0019] The feature vector input to the half-wavelet attention mechanism is ,Will Divided into spatial domain feature branches and wavelet domain feature branch Two parts, of which Used to extract features in the spatial domain, The features in the wavelet domain are obtained by discrete wavelet transform DWT , and then in the wavelet domain Convolution is performed through two 3×3 convolution kernels to obtain the convolution result ; Then use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The weighted results are then concatenated and the concatenated results are convolved with a convolution kernel of size 1. Add the result; the result of the addition is then obtained by inverse wavelet transform ; and After concatenation, it is convolved with a 3×3 convolution kernel and a ReLU activation function, and then Add them together to get ;

[0020] The spatial attention mechanism is:

[0021] ;

[0022] in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, Represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation;

[0023] The channel attention mechanism is:

[0024] ;

[0025] in, Represents the channel attention mechanism result, and Respectively represent the convolution operation using 1×1 filter, represents the average pooling layer, and ReLU represents the ReLU activation function.

[0026] Furthermore, in step S2, the multi-scale feature extraction attention mechanism includes: the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolutional layer in the residual block;

[0027] In the first stage, the feature map obtained after the residual block convolution operation is normalized, and then a multi-scale convolution architecture is applied:

[0028] 1) Adjust the channel dimension through point convolution PWConv, and then perform 5×5 standard convolution to extract basic features;

[0029] 2) Three depthwise separable dilated convolutions are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields. The outputs of the three depthwise separable dilated convolutions are concatenated along the channel dimension, tripling the feature depth. Finally, a nonlinear MLP fuses multi-scale features, and a residual connection adds the original input from the first stage and the output of the MLP to achieve cross-scale collaboration.

[0030] In the second stage, the output of the first stage is first normalized. Then, through a three-branch attention architecture, namely pixel attention mechanism, channel attention mechanism, and simple pixel attention mechanism, weighted coefficients are obtained and spliced. After that, spatial attention mechanism is used for weighting. Finally, a nonlinear MLP is used to fuse multi-scale features. The output of the MLP is added to the original input of the second stage to output the final result.

[0031] The channel attention mechanism in the second stage is:

[0032] ;

[0033] in, represents the result of the second-stage channel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a given feature; GELU represents the GELU activation function;

[0034] The pixel attention mechanism is:

[0035] ;

[0036] in, represents the result of the pixel attention mechanism, and They represent the convolution operation with the convolution kernel of 1 and the convolution kernel of 1 respectively;

[0037] The simple pixel attention mechanism is:

[0038] ;

[0039] in, represents the result of a simple pixel attention mechanism, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a convolution operation with a convolution kernel of 3.

[0040] Furthermore, in step S2, the network predicts the intermediate features as follows:

[0041] Multi-domain fusion object coordinate map of the object to be grasped , surface area attention map of multi-domain fusion , dense correspondence diagram of multi-domain fusion and multi-domain fusion mask ;

[0042] in It is estimated based on the spatial and wavelet domains by rendering the CAD model in the target pose;

[0043] It is based on the spatial domain and wavelet domain, through the object coordinate map Generate by sampling the farthest point on ;

[0044] is by Each channel of the object's 3D bounding box size is normalized to [0,1], so that the side length of the object's 3D bounding box in the object coordinate space is 1, and then Add 2D pixel coordinates to construct it;

[0045] It is obtained by performing semantic segmentation on the grasped object based on the spatial domain and wavelet domain.

[0046] Furthermore, step S3 includes:

[0047] Will and After splicing, the data is input into a neural network composed of several convolutional layers and fully connected layers as a pose regression module to regress the 6D pose including the position and posture of the object to be grasped.

[0048] The translation t of the object to be grasped is calculated through the following process:

[0049] First, we get the following formula 、 and :

[0050] ;

[0051] in, 、 、 For the pose regression module 、 、 The relative translation prediction results in the direction, , The center of the object to be grasped is 、 Coordinates in direction, , The center of the known detection bounding box is 、 Coordinates in direction, , is the width and height of the known detection bounding box; is the actual depth value to be solved, The scaling factor for the target area perception module;

[0052] Then calculate the translation of the object to be grasped by back projection :

[0053] ;

[0054] in is the intrinsic parameter matrix of a given camera;

[0055] Then calculate the rotation matrix of the object to be grasped according to the following formula:

[0056] ;

[0057] in, , , Three column vectors representing the rotation matrix; , To splice and Input the first three values ​​and the last three values ​​of the rotation result predicted by the pose regression module, is the normalization operation;

[0058] Reconstruct L for constraining the rotation matrix information R , L used to constrain 2D center offset information center , L used to constrain depth information depth and L for constraining geometric information geometry There are four loss functions:

[0059] ;

[0060] in and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, Indicates the first 3D models, L ce Represents cross entropy loss, ⊙ represents Hadamard product operation, ‖·‖ represents norm operation; and establishes the overall loss function for:

[0061] ;

[0062] Then according to the pose regression module 、 、 Relative translation prediction results in direction 、 、 , translation , the calculation method of the rotation matrix and the overall loss function are used to regress the 6D pose information of the object to be grasped.

[0063] Furthermore, the step S4 includes:

[0064] Input crawled data into the CNN model as well as 、 、 、 、 and , to evaluate the success probability of grasping in real time, and then decide whether to execute the current grasping or perform a new grasping after randomly adjusting the end position of the robot: if the success probability is not lower than the preset threshold, the current grasping is performed; if the success probability is lower than the preset threshold, the robot randomly adjusts the end position and re-evaluates the success probability of grasping, and executes the cycle until the success probability of grasping is no lower than the preset threshold, and then the grasping is performed.

[0065] Furthermore, the perception model, backbone network, and neural network in steps S1-S3 are jointly trained on an RGB image dataset containing objects annotated with 6D poses. The Ranger optimizer is used in the training, and a cosine-based learning rate decay is initiated when a preset training progress is reached to complete the training.

[0066] The CNN model in step S4 is obtained through pre-training, and the training process includes:

[0067] Collect visual data from the crawl, including 、 、 、 、 、 and ; Then, the result of each grasp is labeled as a supervisory signal, where a labeling result of 1 indicates a successful grasp, and a labeling result of 0 indicates a failed grasp; thus forming training sample data for training the CNN model;

[0068] Then, a CNN network structure consisting of multiple convolutional layers, pooling layers, and fully connected layers is constructed as a model, and the model is trained to learn the mapping from 6D pose and mask map to grasp success probability;

[0069] The visual data F captured for the i-th time i Expressed as:

[0070] ;

[0071] in 、 、 、 、 、 and Represents the visual data in the i-th grasp 、 、 、 、 、 and ;

[0072] Then the input of the CNN model is ,in is the total number of training samples;

[0073] The output of the CNN model is the success probability of each grasping strategy:

[0074] ;

[0075] in, Indicates successful crawling. Indicates that the crawl failed;

[0076] The model is trained using the Adam optimizer to obtain the CNN model after training, in which the cross entropy loss function is used as the loss function :

[0077] ;

[0078] in, It is The labeling result of the training sample is 0 or 1. The model is The prediction of the success probability of each sample.

[0079] On the other hand, the present invention also provides a method for estimating the 6D pose of an object based on an attention mechanism and a robot intelligent grasping device, which are used in the aforementioned method, including:

[0080] The target area perception module performs perception processing on the RGB image according to the position of the object to be grasped in the RGB image, and obtains a target area image of a standard size with the object to be grasped as the target area;

[0081] The backbone network module receives the output of the target region perception module and extracts the features of the target region image based on the fusion of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism. It also obtains the network prediction intermediate features including the coordinate map, mask map, surface area attention map and dense correspondence map of the object to be grasped.

[0082] The pose regression module receives the output of the backbone network module and regresses the 6D pose of the object to be grasped through the neural network structure;

[0083] The robot-evaluable intelligent grasping module receives the output of the pose regression module and evaluates the success probability of the robot's grasping strategy for the grasped object. Based on the success probability, it executes the grasp or adjusts the robot's end position to perform a new grasp.

[0084] The technical effects of the present invention are as follows: (1) The technical solution of the present invention provides a half-wavelet pyramid attention mechanism, which innovatively constructs a dual-stream feature interaction channel in the spatial domain and wavelet domain and integrates it into the ResNet-34 architecture. This mechanism uses the pyramid structure to simultaneously extract and fuse the spatial domain and wavelet domain features of the target, effectively improving the accuracy and anti-occlusion performance of 6D pose estimation.

[0085] (2) The technical solution of the present invention provides a multi-scale feature extraction attention mechanism, which is integrated into the residual block of ResNet-34, realizing multi-scale perception capabilities from microscopic local details to macroscopic overall structures, with excellent feature extraction capabilities, and greatly improving the robustness of 6D pose estimation in cluttered and occluded scenes.

[0086] (3) The end-to-end 6D pose estimation method based on the attention mechanism provided by the technical solution of the present invention can obtain reliable 6D pose results when dealing with occlusion, thereby further effectively improving the reliability of the robot grasping task. Benchmark evaluations on the LINEMOD and LM-O datasets verify that the method provided by the present invention achieves competitive performance.

[0087] (4) The technical solution of the present invention provides a robot-evaluable intelligent grasping module that can evaluate the current grasping operation and determine whether to execute the current grasping or randomly adjust the end position of the robot itself in a very small range to execute a new grasping strategy, so as to achieve intelligent grasping with a high success rate.

[0088] By processing visual images, the present invention realizes intelligent control of robot grasping that can be applied to fields such as industrial production, and has excellent application prospects in fields such as robot operation, augmented reality (AR) systems and industrial automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a flow chart of a method for estimating 6D pose of an object and a robot intelligent grasping system based on an attention mechanism in one embodiment of the present invention;

[0090] Figure 2 Schematic diagram of the network structure of a method for estimating 6D pose of an object based on an attention mechanism in one embodiment of the present invention;

[0091] Figure 3 Schematic diagram of the network structure of the half-wavelet pyramid attention mechanism in one embodiment of the present invention;

[0092] Figure 4 A schematic diagram of the network structure of a multi-scale feature extraction attention mechanism in one embodiment of the present invention;

[0093] Figure 5This is an experimental visualization diagram of an embodiment of the present invention on the LINEMOD-Occluded dataset;

[0094] Figure 6 Schematic diagram of the CNN network structure of the robot-evaluable intelligent grasping module in one embodiment of the present invention. DETAILED DESCRIPTION

[0095] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0096] See also Figure 1 In one embodiment of the present invention, a method for estimating 6D pose of an object based on an attention mechanism is provided, comprising the following steps:

[0097] S1: Obtain an RGB image containing the object to be grasped. Then, based on the perception model, perceive the object to be grasped in the RGB image, thereby magnifying the area where the object to be grasped is located as the target area. The processed target area image serves as the input of the backbone network. Specifically:

[0098] Utilize existing 2D object recognition models to analyze the input RGB image in real time and accurately locate the coordinates of the object of interest. These models can be based on target detection models such as the Detectron2 framework, Fast CNN models, Adelaidet framework, and YOLO models.

[0099] The detected target area image is then normalized to a preset size, adjusting it to a uniform pixel size. In this embodiment, the target area in the original RGB image is resized to a uniform 256×256 pixel size. This processing step helps ensure input consistency in the subsequent feature extraction process, improving target recognition accuracy and system efficiency.

[0100] S2, based on the input target area image, uses a backbone network that combines a half-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism to extract features of the object to be grasped. The half-wavelet pyramid attention mechanism simultaneously collects and fuses spatial and wavelet domain features through a hierarchical pyramid structure. The multi-scale feature extraction attention mechanism uses a differentiated receptive field and multi-branch attention design to effectively extract multi-scale features of the object. The spatial and wavelet domain features are then combined to obtain the network's predicted intermediate features, including the coordinate map, mask map, surface area attention map, and dense correspondence map of the object to be grasped.

[0101] In this step, the half-wavelet pyramid attention mechanism is integrated into the ResNet-34 architecture, specifically set between the residual blocks in ResNet-34, to use the pyramid structure to extract and fuse the wavelet domain and spatial domain features of the target in parallel, thereby improving the accuracy of the pose estimation results. Specifically:

[0102] See also Figure 3 , the half-wavelet pyramid attention mechanism takes the input features Through the pyramid structure Segmentation is performed, where R represents the real space, C represents the number of channels, H represents the height, and W represents the width. That is, each layer of the pyramid structure will input features along and Direction is divided equally feature vectors, the number of segments Based on the number of pyramid levels It shows quadratic growth, that is ,in Indicates the Layer pyramid. As the number of pyramid layers increases, more refined feature vectors can be obtained through segmentation, while the first layer of the pyramid structure does not segment. , so the first layer of the pyramid structure obtains a global feature map, and the subsequent pyramid layers divide the feature map into smaller dimensions, that is, local feature maps are obtained. Then, each layer of the pyramid structure applies the half-wavelet attention mechanism to the feature map obtained by itself. The pyramid structure that has been segmented then performs an inverse segmentation operation based on the spatial position encoding information at the time of segmentation. The inverse segmentation is specifically as follows: first, the position of each local feature map, that is, the small block, is recorded at the time of segmentation, and the sequential index of each small block is recorded in row priority. Then, after the half-wavelet attention mechanism is applied to each small block separately, the small blocks are rearranged according to the recorded sequential index, so as to be merged back to the original size. Then, in Generated , and generate the Pyramid Attention Map Finally, the pyramid attention maps of different layers are Fusion to obtain the final . This embodiment Figure 3 A two-layer pyramid structure is adopted in the present invention, and a pyramid structure with more layers can be constructed as needed during specific implementation.

[0103] The half-wavelet attention mechanism used in the half-wavelet pyramid attention mechanism is specifically:

[0104] The single feature map in each pyramid is the minimum dimension feature It is divided into two parts, namely the spatial domain feature branch and wavelet domain feature branch .in Used to extract features in the spatial domain. Then the features in the wavelet domain are obtained by discrete wavelet transform DWT , used to extract features in the wavelet domain. In the wavelet domain, Convolution is performed through two 3×3 convolution kernels to obtain the convolution result Then, we use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The two weighted results are then added together, and the result of the addition is convolved with a convolution kernel of size 1. Then the result of the addition is obtained by inverse wavelet transform , Reconciliation After splicing, it is convolved with a 3×3 convolution kernel and then activated with a ReLU function. Add them together and finally get .

[0105] The discrete wavelet transform decomposes the feature sequence (feature map) into low-frequency components of different scales through multi-resolution analysis and high-frequency components , the expression is:

[0106] Low frequency component:

[0107] ;

[0108] High frequency components:

[0109] ;

[0110] in, is the input feature sequence, represents the number of eigenvectors, is the scaling function, defined as , used to extract low-frequency information of features, is a wavelet function, defined as , used to extract high-frequency information of features, is the number of decomposition layers, indicating the scale level of wavelet transform, is the translation parameter, which represents the position of the wavelet in time or space.

[0111] The spatial attention mechanism mentioned above is specifically:

[0112] ;

[0113] in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, Represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation.

[0114] The channel attention mechanism mentioned above is specifically:

[0115] ;

[0116] in, Represents the channel attention mechanism result, represents the sigmoid activation function, and represents two convolution operations using 1×1 filters, represents the average pooling layer, and ReLU represents the ReLU activation function.

[0117] The inverse wavelet transform mentioned above can reconstruct the original feature sequence (feature map), which is expressed as:

[0118] ;

[0119] in, is the maximum number of decomposition levels, For the The approximation coefficient of the layer represents the low-frequency global characteristics of the feature, For the The detail coefficient of the layer represents the local variation of features at different resolutions.

[0120] In this step, the multi-scale feature extraction attention mechanism mentioned above is integrated into the first three residual blocks of ResNet-34, replacing the original third convolutional layer in the residual block. The multi-scale feature extraction attention mechanism of this embodiment adopts differentiated receptive field and multi-branch attention design to effectively extract the multi-scale features of the object, specifically:

[0121] See also Figure 4In the first stage, the feature maps obtained after the residual block convolution operation are normalized by BatchNorm preprocessing. Then a multi-scale convolutional architecture is applied: (1) the channel dimension is adjusted by point convolution (PWConv), followed by a 5×5 standard convolution to extract basic features; (2) three depth-wise separable dilated convolutions (DWDConv7×7, DWDConv5×5, and DWDConv3×3) are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields. The outputs of the three convolutions are then concatenated along the channel dimension to double the feature depth. Finally, a nonlinear MLP (including two PWConv layers using the GELU activation function) fuses the multi-scale features, while the residual connection coordinates the original and processed features across scales.

[0122] In the second stage, the output of the first stage is first processed by BatchNorm. Then, through the three-branch attention architecture, namely the pixel attention mechanism, the channel attention mechanism and the simple pixel attention mechanism, the weighted coefficients are obtained and spliced. After that, the spatial attention mechanism is used for weighting. The spatial attention mechanism here is the same as the one in the half-wavelet pyramid attention mechanism. Finally, a nonlinear MLP is used to fuse the multi-scale features and add the output of the MLP to the original input of the second stage to output the final result.

[0123] The channel attention mechanism in the second stage of the multi-scale feature extraction attention mechanism is specifically:

[0124] ;

[0125] in, represents the result of the channel attention mechanism in the second stage, represents the sigmoid activation function, represents the average pooling layer, and Represents two convolution operations, with a convolution kernel of 1; Represents a given feature; GELU represents the GELU activation function.

[0126] The pixel attention mechanism is specifically:

[0127] ;

[0128] in, represents the result of the pixel attention mechanism, represents the sigmoid activation function, and Represents two convolution operations with a convolution kernel of 1.

[0129] The simple pixel attention mechanism is as follows:

[0130] ;

[0131] in, represents the result of a simple pixel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and Represents two convolution operations, the convolution kernel is 1, and Represents a convolution operation with a convolution kernel of 3.

[0132] Then, the spatial domain and wavelet domain features are combined to obtain the network prediction intermediate features, that is, the backbone network predicts the surface area attention map of the multi-domain fusion of the object to be grasped based on the RGB image. , multi-domain fusion object coordinate map , dense correspondence diagram of multi-domain fusion and multi-domain fusion mask .

[0133] The surface area attention map of multi-domain fusion It is based on the spatial domain and wavelet domain, through the object coordinate map The farthest point on is sampled.

[0134] Object coordinate map of multi-domain fusion It is estimated by rendering the CAD model in the target pose based on the spatial domain and wavelet domain.

[0135] Dense correspondence graph for multi-domain fusion is based on The 3D bounding box size of the object to be grasped , , , normalize each channel to [0,1], and then pass Add 2D pixel coordinates to construct .

[0136] Mask map for multi-domain fusion It is obtained by performing semantic segmentation on the grasped objects based on the spatial domain and wavelet domain.

[0137] S3: The obtained multi-domain fusion surface area attention map and the multi-domain fusion dense correspondence map are input into the pose regression module. Based on the neural network structure of the pose regression module, the 6D pose information of the object to be grasped is directly regressed. Specifically:

[0138] The resulting multi-domain fused surface area attention map and the multi-domain fused dense correspondence map are concatenated and input into the pose regression module. The pose regression module in this embodiment is a neural network structure composed of several convolutional layers and fully connected layers, which is used to regress the 6D pose information of the object to be grasped, including its position and posture.

[0139] First, we get the following formula 、 and :

[0140] ;

[0141] in, 、 、 For the pose regression module 、 、 The relative translation prediction results in the direction, 、 The center of the object to be grasped is 、 Coordinates in direction, 、 The center of the known detection bounding box is 、 Coordinates in direction, 、 are the width and height of the known detection bounding box. is the actual depth value to be solved, It is the scaling factor of the target area perception module.

[0142] Then calculate the translation of the object to be grasped by back projection :

[0143] ;

[0144] in is the intrinsic parameter matrix of a given camera, thereby regressing the position of the object to be grasped.

[0145] According to the multi-domain fused surface area attention map and the multi-domain fused dense correspondence map of the object to be grasped, the rotation matrix of the object to be grasped can be calculated according to the following formula:

[0146] ;

[0147] in, , , Three column vectors representing the rotation matrix, , To splice and Input the first three values ​​and the last three values ​​of the rotation result predicted by the pose regression module, It is a normalization operation.

[0148] At the same time, this embodiment constructs the corresponding loss function, which constrains the rotation information , 2D center offset information , depth information and geometric information , the loss functions of the four are specifically:

[0149] ;

[0150] in, and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, , Indicates the first 3D models, L ce represents the cross entropy loss, ⊙ represents the Hadamard product operation, and ‖·‖ represents the norm operation.

[0151] In particular, for symmetrical objects, Expands to:

[0152] ;

[0153] in, represents all valid rotations in the symmetric case.

[0154] Furthermore, based on the above loss function, the overall loss function constructed in this embodiment is:

[0155] ;

[0156] The target translation prediction result is obtained by the calculation method 、 、 The 6D pose information of the object to be grasped is regressed based on the calculation method of t, the rotation matrix, and the overall loss function.

[0157] S4, see Figure 6 The obtained 6D pose information is sent to the robot's evaluable intelligent grasping module to enable the robot to grasp the object to be grasped. In addition, the module can predict the success probability of grasping based on the geometric features, position information and pose information of the object to be grasped, so as to help the robot make a more intelligent grasping strategy.

[0158] Specifically, we collect visual data for each grasping task: object mask map (reflects the geometric characteristics of the object), the predicted value of the object translation result 、 、 (reflects the position information of the object) and the predicted value of the object rotation result , , (Reflects the relative relationship between the object and the environment), annotates the result of each grasp (success or failure) as a supervision signal, success (1): grasping is successful, the object is stably grasped and the task is completed, failure (0): grasping fails, the object is not successfully grasped, or the task is not completed.

[0159] Using a CNN network architecture comprised of multiple convolutional, pooling, and fully connected layers, the model is trained to learn the mapping from object masks, predicted translations, and predicted rotations to grasp success probabilities. The trained CNN model evaluates the success probability of grasping strategies in real time and makes intelligent decisions, determining whether to proceed with the grasp or to randomly adjust the robot's end-point position within a very small range to form a new grasping strategy to further assess success probability.

[0160] The visual data F captured each time i It can be expressed as:

[0161] ;

[0162] The subscript i of each symbol in the above formula represents the corresponding data during the i-th crawl.

[0163] Then the input of the CNN model is:

[0164] ;

[0165] The output of the CNN model is the success probability of each grasping strategy:

[0166] ;

[0167] in, Indicates that the crawl is successful, otherwise, Indicates that the crawl failed.

[0168] The present invention uses the Adam optimizer for training in the robot-evaluable intelligent grasping module, and adopts the cross entropy loss function as the loss function, specifically:

[0169] ;

[0170] in, It is The true label of each sample is 0 or 1 (failure or success). The model is The prediction of the success probability of each sample, is the number of training samples.

[0171] In the crawling task, the system will The trained CNN model is used to predict the success probability of a grasping strategy. If the success probability is at least a set threshold, the robot executes the strategy. If the success probability is below the threshold, the robot randomly adjusts its end position within a very small range and predicts the success probability again. This cycle continues until the success probability is at least a set threshold, at which point the corresponding grasping strategy is executed.

[0172] Furthermore, in steps S1-S3, the perception model, backbone network and neural network are jointly trained based on an RGB image dataset containing objects with 6D pose annotations. Specifically, in this embodiment, the widely used LINEMOD dataset and LINEMOD-Occluded dataset are selected for training, which can also be replaced by self-made private datasets in specific implementations. The LINEMOD dataset contains 15 objects with different colors, shapes and sizes, and 6D pose annotations are provided for 13 of them. The dataset is divided into 85% real test images and 15% real training images, and synthetic images are enhanced for each object. The LINEMOD-Occluded dataset is an expansion of the LINEMOD dataset, containing 8 objects with enhanced occlusion. During the training process, the real annotations are combined with synthetic samples of each object.

[0173] The model trained in steps S1-S3 of this example was developed on PyTorch. The training configuration of this example uses the Ranger optimizer (batch size = 24, learning rate = 0.0001), and cosine-based learning rate decay is initiated after the training progress reaches 72%.

[0174] For comparison, all models mentioned below in this example are trained, verified, and tested using the PyTorch framework, CUDA=11.1, on an RTX4090 GPU. To verify the superiority of the method provided in this example, the following experiments were conducted:

[0175] (1) Comparative experiments between the method provided in this example and other models:

[0176] Table 1 lists the quantitative results of the method provided by the present invention and six SOTA algorithms in the LINEMOD benchmark test based on the ADD(-S) indicator under the same environment. Table 2 shows the performance of the method provided by the present invention and six SOTA algorithms on the LINEMOD-Occluded occlusion dataset under the same environment, evaluated using the ADD(-S) standard. The experimental results show that the method provided by the present invention surpasses all SOTA methods in ADD(-S) performance under standard conditions and occlusion conditions.

[0177] Table 1 Comparison results of the method provided by the present invention and other SOTA algorithms on LINEMOD.

[0178] method PoseCNN Pix2Pose DPOD PVNet CDPN GDR-Net Method of the present invention Ape Model - 58.10 53.28 43.62 - - 81.62 Bench - 91.00 95.34 99.90 - - 99.03 camera - 60.90 90.36 86.86 - - 95.78 kettle - 84.40 94.10 95.47 - - 98.13 Cat Model - 65.00 60.38 79.34 - - 94.81 electric drill - 76.30 97.72 96.43 - - 96.93 duck model - 43.80 66.01 52.58 - - 83.29 egg carton - 96.80 99.72 99.15 - - 99.53 glue - 79.40 93.83 95.66 - - 99.42 punching machine - 74.80 65.83 81.92 - - 88.68 iron - 83.40 99.80 98.88 - - 97.96 desk lamp - 82.00 88.11 99.33 - - 99.23 Telephone - 45.00 74.24 92.41 - - 93.58 average value 62.7 72.38 82.98 86.27 89.86 93.70 94.46

[0179] (“-” means the result is not available).

[0180] Table 2 Comparison results between the method provided by the present invention and other SOTA algorithms on LINEMOD-Occluded.

[0181] method PoseCNN Pix2Pose DPOD PVNet GeoPose GDR-Net Method of the present invention Ape Model 9.60 22.00 - 15.18 19.07 46.80 50.94 kettle 45.20 44.70 - 63.30 65.46 90.80 90.89 Cat Model 0.90 22.70 - 16.68 18.83 40.50 40.94 electric drill 41.40 44.70 - 65.65 65.86 82.60 85.83 duck model 19.60 15.00 - 25.24 28.47 46.90 52.06 egg carton 22.00 25.20 - 50.17 54.73 54.20 61.36 glue 38.50 32.40 - 49.62 56.77 75.80 78.85 punching machine 22.10 49.50 - 39.67 49.21 60.10 75.37 average value 24.90 32.00 32.79 40.77 44.80 62.20 67.03

[0182] (“-” means the result is not available).

[0183] The verification results of the LINEMOD-Occluded dataset are visualized as follows Figure 5 As shown in Table 1 and Table 2, the method provided by the present invention improves ADD(-S) by 0.76% on the LINEMOD dataset compared to the baseline model GDR-Net, and improves it by 4.83% on the LINEMOD-Occluded dataset.

[0184] The above experimental data fully verify the breakthrough performance of the model framework provided by this invention in solving complex 6D object pose estimation tasks.

[0185] (2) Ablation experiment of the method provided by the present invention:

[0186] Table 3 lists the ablation experimental results of the method provided by the present invention on the LINEMOD dataset under different levels of ADD(-S) under the same environment.

[0187] Table 3 Ablation experiment results of the method provided by the present invention.

[0188]

[0189] The experimental results in Table 3 reveal the functional complementarity between the Half-Wavelet Pyramid Attention mechanism and the Multi-Scale Feature Extraction Attention mechanism. When implemented independently, each mechanism exhibits unique performance advantages: the Half-Wavelet Pyramid Attention mechanism improves accuracy to 93.83% (0.1 ADD(-S)). This is due to its ability to simultaneously collect and integrate spatial and wavelet domain features of an object, enabling the network to obtain more diverse feature information for 6D pose estimation. Meanwhile, the Multi-Scale Feature Extraction Attention mechanism captures the multi-scale features and contextual relationships of an object, achieving an accuracy of 93.89% (0.1 ADD(-S)), making it particularly effective in resolving occlusion ambiguities common in complex scenes.

[0190] It is worth noting that the combination of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism produces an excellent synergistic effect. The combination of the two achieves a peak performance of 94.46%, which cannot be explained by a simple additive effect. This synergistic effect stems from two interdependent mechanisms: (1) the multi-receptive field features of the multi-scale feature extraction attention mechanism provide rich input for the attention weight calculation of the half-wavelet pyramid attention mechanism; (2) the multi-domain features extracted by the half-wavelet pyramid attention mechanism also provide diverse input for the multi-scale feature extraction attention mechanism.

[0191] On the other hand, see Figure 2 In one embodiment, the present invention further provides a robot object grasping device based on 6D pose estimation of an object, comprising:

[0192] The target area perception module magnifies the local area of ​​the image according to the position of the object to be grasped in the RGB image.

[0193] The backbone network module extracts multi-domain (spatial domain and wavelet domain) and multi-scale features of the grasped object based on the input RGB image and the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism, and predicts the intermediate features.

[0194] The pose regression module directly regresses the 6D pose of the object to be grasped based on the intermediate features through a neural network structure.

[0195] The robot's evaluable intelligent grasping module receives the 6D pose information of the object to be grasped and grasps the object. At the same time, it can evaluate the grasping operation, predict the success rate of the grasping, and then determine whether to execute the grasping or randomly adjust the end position of the robot within a very small range to implement a new grasping strategy.

[0196] The device operates on a structure consisting of a robot, an end effector, and an industrial camera, and employs an "eye-in-hand" vision model. Robot vision calibration is used to obtain a transformation matrix from the camera coordinate system to the robot end effector coordinate system. Given an RGB image, the system first acquires the 6D pose information of the object to be grasped. This information is then converted using the robot's hand-eye calibration matrix. Finally, the converted 6D pose is transmitted to the robotic system to execute the grasping task. During the grasping task, the system can evaluate the grasping operation and predict the success rate. This allows the system to determine whether to proceed with the grasp or to implement a new grasping strategy by randomly adjusting the end effector position within a very small range.

[0197] The above is a detailed introduction to the object 6D pose estimation method based on the attention mechanism and the robot intelligent grasping system provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified. These improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A robot object grasping method based on 6D pose estimation of the object, characterized in that: The following steps are involved: S1, obtaining an RGB image including the object to be grasped, performing perception processing on the RGB image using a perception model, and obtaining a target area image of a standard size with the object to be grasped as the target area; S2, extracts features of the target area image based on a backbone network that integrates a half-wavelet pyramid attention mechanism and a multi-scale feature extraction attention mechanism. The half-wavelet pyramid attention mechanism simultaneously collects and fuses spatial and wavelet domain features through a hierarchical pyramid structure; the multi-scale feature extraction attention mechanism uses differentiated receptive fields and multi-branch attention to extract multi-scale features. The spatial and wavelet domain features are then combined to obtain network prediction intermediate features, including a coordinate map, a mask map, a surface area attention map, and a dense correspondence map of the object to be grasped. S3, based on a neural network composed of multiple convolutional layers and fully connected layers, processes the surface area attention map and the dense correspondence map to regress the 6D pose including the position and posture of the object to be grasped; S4 inputs the 6D pose and mask image into the CNN model to evaluate the success probability of the robot grasping the object to be grasped, and executes the grasp or adjusts the robot end position to perform a new grasp based on the success probability.

2. The method according to claim 1, characterized in that In step S1, the step of performing perception processing on the image includes: The existing two-dimensional target recognition model is used to perform real-time analysis on the image including the object to be grasped, thereby obtaining the regional coordinates of the object to be grasped in the image, and extracting the original image area in the image according to the regional coordinates; Then, the original image area is normalized according to a preset size, so as to adjust the original image area to a preset uniform pixel size.

3. The method according to claim 1, characterized in that In step S2, the backbone network is a ResNet-34 structure, the half-wavelet pyramid attention mechanism is set between the residual blocks of ResNet-34, and the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolution layer in the residual block.

4. The method according to claim 3, characterized in that In step S2, the half-wavelet pyramid attention mechanism is: The half-wavelet pyramid attention mechanism includes a multi-layer pyramid structure, and each layer of the pyramid structure receives the features obtained after the residual block convolution operation separately. , and the pyramid structure of each layer will along and The direction is evenly divided into feature vectors, where R represents the real space, C represents the number of channels, H represents the height, and W represents the width; each feature vector is processed by the half-wavelet attention mechanism and then de-segmented to merge to obtain the output of each layer, and finally the output of each layer is fused as the output of the half-wavelet pyramid attention mechanism ; where n is determined by the number of layers i of the pyramid structure where the segmentation is performed, n=i 2 ; The half-wavelet attention mechanism is: The feature vector input to the half-wavelet attention mechanism is ,Will Divided into spatial domain feature branches and wavelet domain feature branch Two parts, of which Used to extract features in the spatial domain, The features in the wavelet domain are obtained by discrete wavelet transform DWT , and then in the wavelet domain Convolution is performed through two 3×3 convolution kernels to obtain the convolution result ; Then use the parallel spatial attention mechanism and channel attention mechanism to simultaneously The weighted results are then concatenated and the concatenated results are convolved with a convolution kernel of size 1. Add the result; the result of the addition is then obtained by inverse wavelet transform ; and After concatenation, it is convolved with a 3×3 convolution kernel and a ReLU activation function, and then Add them together to get ; The spatial attention mechanism is: ； in, Represents the result of the spatial attention mechanism, represents the sigmoid activation function, and To perform adjustable maximum pooling and average pooling channel functions, respectively, Represents the convolution operation with the 5x5 convolution kernel, and Concat represents concatenation; The channel attention mechanism is: ； in, Represents the channel attention mechanism result, and Respectively represent the convolution operation using 1×1 filter, represents the average pooling layer, and ReLU represents the ReLU activation function.

5. The method according to claim 3, characterized in that In step S2, the multi-scale feature extraction attention mechanism includes: the multi-scale feature extraction attention mechanism is set in the first three residual blocks of ResNet-34 and replaces the original third convolutional layer in the residual block; In the first stage, the feature map obtained after the residual block convolution operation is normalized, and then a multi-scale convolution architecture is applied: 1) Adjust the channel dimension through point convolution PWConv, and then perform 5×5 standard convolution to extract basic features; 2) Three depthwise separable dilated convolutions are deployed in parallel to capture global spatial patterns and local textures through heterogeneous receptive fields. The outputs of the three depthwise separable dilated convolutions are concatenated along the channel dimension, tripling the feature depth. Finally, a nonlinear MLP fuses multi-scale features, and a residual connection adds the original input from the first stage and the output of the MLP to achieve cross-scale collaboration. In the second stage, the output of the first stage is first normalized. Then, through a three-branch attention architecture, namely pixel attention mechanism, channel attention mechanism, and simple pixel attention mechanism, weighted coefficients are obtained and spliced. After that, spatial attention mechanism is used for weighting. Finally, a nonlinear MLP is used to fuse multi-scale features. The output of the MLP is added to the original input of the second stage to output the final result. The channel attention mechanism in the second stage is: ； in, represents the result of the second-stage channel attention mechanism, represents the sigmoid activation function, represents the average pooling layer, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a given feature; GELU represents the GELU activation function; The pixel attention mechanism is: ； in, represents the result of the pixel attention mechanism, and They represent the convolution operation with the convolution kernel of 1 and the convolution kernel of 1 respectively; The simple pixel attention mechanism is: ； in, represents the result of a simple pixel attention mechanism, and They represent convolution operations with a convolution kernel of 1 respectively; Represents a convolution operation with a convolution kernel of 3.

6. The method according to claim 1, characterized in that In step S2, the network predicts the intermediate features as follows: Multi-domain fusion object coordinate map of the object to be grasped , surface area attention map of multi-domain fusion , dense correspondence diagram of multi-domain fusion and multi-domain fusion mask ; in It is estimated based on the spatial and wavelet domains by rendering the CAD model in the target pose; It is based on the spatial domain and wavelet domain, through the object coordinate map Generate by sampling the farthest point on ; is by Each channel of the object's 3D bounding box size is normalized to [0,1], so that the side length of the object's 3D bounding box in the object coordinate space is 1, and then Add 2D pixel coordinates to construct it; It is obtained by performing semantic segmentation on the grasped object based on the spatial domain and wavelet domain.

7. The method according to claim 6, characterized in that The step S3 comprises: Will and After splicing, the data is input into a neural network composed of several convolutional layers and fully connected layers as a pose regression module to regress the 6D pose including the position and posture of the object to be grasped. The translation t of the object to be grasped is calculated through the following process: First, we get the following formula 、 and : ； in, 、 、 For the pose regression module 、 、 The relative translation prediction results in the direction, , The center of the object to be grasped is 、 Coordinates in direction, , The center of the known detection bounding box is 、 Coordinates in direction, , is the width and height of the known detection bounding box; is the actual depth value to be solved, The scaling factor for the target area perception module; Then calculate the translation of the object to be grasped by back projection : ； in is the intrinsic parameter matrix of a given camera; Then calculate the rotation matrix of the object to be grasped according to the following formula: ； in, , , Three column vectors representing the rotation matrix; , To splice and Input the first three values ​​and the last three values ​​of the rotation result predicted by the pose regression module, is the normalization operation; Reconstruct L for constraining the rotation matrix information R , L used to constrain 2D center offset information center , L used to constrain depth information depth and L for constraining geometric information geometry There are four loss functions: ； in and Represent the predicted value and the true value respectively, B represents the 3D model set of the object to be grasped, Indicates the first 3D models, L ce Represents cross entropy loss, ⊙ represents Hadamard product operation, ‖·‖ represents norm operation; and establishes the overall loss function for: ； Then according to the pose regression module 、 、 Relative translation prediction results in direction 、 、 , translation , the calculation method of the rotation matrix and the overall loss function are used to regress the 6D pose information of the object to be grasped.

8. The method according to claim 7, characterized in that The step S4 comprises: Input crawled data into the CNN model as well as 、 、 、 、 and , to evaluate the success probability of grasping in real time, and then decide whether to execute the current grasping or perform a new grasping after randomly adjusting the end position of the robot: if the success probability is not lower than the preset threshold, the current grasping is performed; if the success probability is lower than the preset threshold, the robot randomly adjusts the end position and re-evaluates the success probability of grasping, and executes the cycle until the success probability of grasping is no lower than the preset threshold, and then the grasping is performed.

9. The method according to claim 8, characterized in that The perception model, backbone network, and neural network in steps S1-S3 are jointly trained based on an RGB image dataset containing objects labeled with 6D poses. The Ranger optimizer is used in the training, and a cosine-based learning rate decay is initiated when a preset training progress is reached to complete the training. The CNN model in step S4 is obtained through pre-training, and the training process includes: Collect visual data from the crawl, including 、 、 、 、 、 and ; Then, the result of each grasp is labeled as a supervisory signal, where a labeling result of 1 indicates a successful grasp, and a labeling result of 0 indicates a failed grasp; thus forming training sample data for training the CNN model; Then, a CNN network structure consisting of multiple convolutional layers, pooling layers, and fully connected layers is constructed as a model, and the model is trained to learn the mapping from 6D pose and mask map to grasp success probability; The visual data F captured for the i-th time i Expressed as: ； in 、 、 、 、 、 and Represents the visual data in the i-th grasp 、 、 、 、 、 and ; Then the input of the CNN model is ,in is the total number of training samples; The output of the CNN model is the success probability of each grasping strategy: ； in, Indicates successful crawling. Indicates that the crawl failed; The model is trained using the Adam optimizer to obtain the CNN model after training, in which the cross entropy loss function is used as the loss function : ； in, It is The labeling result of the training sample is 0 or 1. The model is The prediction of the success probability of each sample.

10. A 6D pose estimation method for an object based on an attention mechanism and a robot intelligent grasping device, characterized in that: Used to perform the method according to any one of claims 1 to 9, comprising: The target area perception module performs perception processing on the RGB image according to the position of the object to be grasped in the RGB image, and obtains a target area image of a standard size with the object to be grasped as the target area; The backbone network module receives the output of the target region perception module and extracts the features of the target region image based on the fusion of the half-wavelet pyramid attention mechanism and the multi-scale feature extraction attention mechanism. It also obtains the network prediction intermediate features including the coordinate map, mask map, surface area attention map and dense correspondence map of the object to be grasped. The pose regression module receives the output of the backbone network module and regresses the 6D pose of the object to be grasped through the neural network structure; The robot-evaluable intelligent grasping module receives the output of the pose regression module and evaluates the success probability of the robot's grasping strategy for the object to be grasped. Based on the success probability, it executes the grasp or adjusts the robot's end position to perform a new grasp.

Images