Grabbing pose detection method based on attention mechanism

Abstract

The invention belongs to the field of computer vision, and particularly relates to a grabbing pose detection method based on an attention mechanism. In order to solve the problems that point cloud feature capability extraction is insufficient and generated pose estimation is incomplete, the invention provides a dual-channel dual-branch extrusion and incentive attention fusion residual module (DDSER), the module is fused with UNet to serve as a backbone network, and point cloud channel features are extracted; in order to reduce channel feature loss caused by obtaining point vectors through farthest point sampling (FPS), a multi-head self-attention channel module (MSCAM) fused with an affinity matrix is provided, redundant features are avoided, global feature representation in the point vector channel direction is enhanced, the overall detection precision of the network is improved, and the method is widely suitable for industrial mechanical arm grabbing.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a grasping posture detection method based on an attention mechanism. Background Art

[0002] Grasping posture detection is one of the research hotspots in the field of computer vision and is crucial to improving the grasping efficiency of industrial robotic arms. Robots still face many challenges in grasping objects in unstructured environments, such as difficulty in object recognition, insufficient grasping adaptability, and the balance between real-time performance and accuracy. In recent years, with the rise of deep learning technology, especially the introduction of convolutional neural networks (CNNs) and attention mechanisms, these problems have been alleviated to a certain extent. By learning a large amount of data, robotic arms can more accurately identify objects of different shapes and materials, and focus on key parts through attention mechanisms to improve grasping accuracy.

[0003] With the development of depth sensor technology and the application trend of deep learning in it, data-driven methods for robot grasping have made great progress. They can continuously learn based on data in actual scenes, improve their adaptability to various object shapes and environments, and achieve a wider range of grasping tasks. Recently, some scholars have studied the extension of six-degree-of-freedom grasping detection methods to cluttered scenes to generate grasping postures of objects, but they cannot learn instance-level grasping. Due to the lack of instance information, the model cannot perform target-driven grasping. To solve this problem, in 2021, Li et al. proposed a simultaneous multi-task learning method for six-degree-of-freedom grasping pose estimation in structured cluttered scenes, and combined instance semantic segmentation to predict point-by-point six-degree-of-freedom grasping poses, but the mentioned method is difficult to generate pose estimates for small and medium-sized objects or parts. In 2023, Ma et al. proposed a new six-degree-of-freedom scale-balanced grasping detection method, which introduced a multi-scale cylinder grouping (MsCG) module to combine multi-scale cylinder features into a global feature and embed additional context to more comprehensively describe the local shape. In addition, the Scale Balanced Learning (SBL) loss and Object Balanced Sampling (OBS) strategy were designed for the training and inference stages respectively. This method solves the problem of poor grasping effect for small objects.

[0004] However, the above methods still have some limitations. First, the point codec part does not use the attention mechanism, and the six-degree-of-freedom grasping detection method cannot fully focus on the important point cloud channel features, resulting in the problem of missing pose estimation; second, the farthest point sampling (FPS) operation only uses local information, resulting in the loss of point vector channel features. Therefore, this field still needs the introduction of more ideas to fully combine the attention mechanism to realize feature utilization. Summary of the invention

[0005] In order to solve the problem of incomplete generated pose estimation, the present invention proposes a grasping pose detection method based on the attention mechanism. By introducing a dual-channel dual-branch squeezing and excitation attention fusion residual module (DDSER), this module is fused with UNet as the backbone network to extract point cloud channel features; secondly, in order to reduce the channel feature loss caused by obtaining point vectors through FPS, a multi-head self-attention channel module (MSCAM) fused with affinity matrix is ​​proposed to avoid redundant features, enhance the global feature representation of the point vector channel direction, improve the detection capability of the network model, and achieve accurate and complete technical effects of pose estimation prediction.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a grasping posture detection method based on an attention mechanism, comprising the following steps:

[0008] Step 1: Collect the RGB-D image of the object to be grasped and convert the RGB-D image into a point cloud form;

[0009] Step 2, input the point cloud into the trained grasping posture detection model, perform posture estimation, and output the grasping posture; the grasping posture detection model includes: point cloud feature extraction, grasping area and final grasping suggestion; the point cloud feature extraction uses DDSER-UNet as the backbone network, introduces dual-channel dual-branch attention, integrates the jump connection mechanism to extract the point cloud channel features, generates the graspable area, and uses the multi-layer perceptron to calculate the graspable area score; the grasping area uses the farthest point sampling FPS to sample seed points from the grasping area, uses the MSCAM module to extract the channel feature vector of the sampled seed points, generates and selects the most suitable graspable view, uses the cylinder to combine the feature sum of the point vector and the view vector for grouping and sampling, and generates grasping candidate points; the final grasping suggestion uses the maximum pooling layer and the multi-layer perceptron to screen and evaluate the grasping candidate points and generate the best graspable posture.

[0010] Further, the point cloud feature extraction is specifically as follows:

[0011] The point cloud is taken as input, and the DDSER-UNet point encoder-decoder architecture is designed as the backbone network. The input channel is 3, and the output channel is 512 feature vectors. The first convolution layer of the encoder uses batch normalization and ReLU function. After a convolution layer of feature extraction, the output channel is 32. A downsampling layer is added after each DDSER module of the encoder, and the final output channel size is 256. The input of the decoder includes the output channel feature Q of the last layer and the channel feature P in the encoder. First, upsampling is performed on Q and cropping operations are performed on P, and then the two outputs are spliced ​​in the channel dimension. The upsampling operation is performed on the basis of the DDSER module, and finally the grasping area score is calculated by a multi-layer perceptron.

[0012] Further, the DDSER module is specifically:

[0013] First, enter u c After three convolutional layers, the residual layer output result r is obtained c , where the size of the convolution kernel is 3×3;

[0014] Secondly, the squeezing operation uses the average pooling layer to c Perform the extrusion operation to obtain z c , as follows:

[0015]

[0016] In the formula, F sq (r c ) represents the squeezing function, N represents the point set, r c (i) represents the value of the cth channel at the i-th point;

[0017] Finally, the excitation operation, for the output of the squeeze operation, uses the splicing of two excitation processes to adjust the channel weights in the local branch, completes the dimensionality reduction and dimensionality increase operations through two convolutional layers and ReLU activation functions, and sets two different reduction ratios, r and 2r, respectively, as shown in the following formula:

[0018] F r =W 21 δ(W 11 ,z c )

[0019] F 2r =W 22 δ(W 12 ,z c )

[0020] Where W 11 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of r, W 21 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of r, W12 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of 2r, W 22 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of 2r, and δ represents the ReLU activation function;

[0021] The output results of these two different reduction ratios are spliced ​​to obtain the local feature L c , the shape is C×N, as follows:

[0022] L c =Concat(F r ,F 2r )

[0023] In the formula, Concat represents the concatenation operation;

[0024] For the output of the squeeze operation, in the global branch, z c After being processed by the fully connected layer and the Softmax function, the global feature G is obtained. c , as follows:

[0025] G c =φ(W 13 ,z c )

[0026] In the formula, φ represents the Softmax function, W 13 Represents the weight parameters of the fully connected layer in the global branch;

[0027] The local feature L c With the global feature G c Multiply them and use the Sigmoid function to generate the final weight. The output s value is normalized and the value range becomes 0 to 1, as shown in the following formula:

[0028] s c =F ex (z,W)=σ(L c ·G c )

[0029] In the formula, F ex (z,W) represents the activation function, σ represents the Sigmoid function;

[0030] Will s c The result r of the residual layer output c Perform a reweighted scale operation and finally sum it with the input u c Perform the sum operation to get the final output of the DDSER module As follows:

[0031] F scale (r c ,sc )=r c ·s c

[0032]

[0033] In the formula, Addition operation.

[0034] Furthermore, the multilayer perceptron calculates the grasping area score according to the point set channel features extracted by DDSER-UNet, and the output result includes the graspable scene prediction value of N×1 dimension, the point vector score of N×1 dimension and the view vector score of N×1 dimension, thereby forming a total output of size N×3. After obtaining the graspable scene divided by points, the points with grasping scores greater than the threshold ω are selected according to the estimated point vector scores.

[0035] Further, the grabbing area is specifically:

[0036] According to the graspable scene, M seed points are selected by the farthest point sampling FPS. The (3+C)-dimensional feature M seed points are obtained through the MSCAM module. The view is generated by ApproachNet using the Fibonacci grid, and the M×V view vector is generated through its internal multi-layer perceptron structure. The best view is selected for subsequent prediction, and then the Cylinder Group cylinder combines the point vector and the view vector to perform grouping and sampling to generate grasping candidate points.

[0037] Further, the MSCAM module is specifically:

[0038] Construct multi-head channel self-attention and calculate the strategy of channel-by-channel attention as follows:

[0039]

[0040] Where Y represents the number of self-attention blocks, F l ∈R M×C Indicates that the channel feature map is the lth layer, AT(·) is as follows:

[0041]

[0042] Based on M c Calculate affinity matrix Measure the difference between channels as follows:

[0043]

[0044]

[0045] Where M crepresents the mth head channel attention matrix, m represents the index of the attention head, represents the affinity matrix, φ represents the Softmax function, and expand(·) repeats the max pooling (M c ) is extended to M c Matrices of the same size are subtracted, maxpooling (M c )∈R d×1 Extract M c The maximum value of each row in are the learnable weight parameters of the three linear layers, d c =C / Y.

[0046] Further, the Cylinder Group cylinder is specifically:

[0047] For each pair of M seed points and views, a cylinder with a fixed height d and radius r is used to group and sample K points among the M seed points. After the cylinder is aligned with the fixture frame, the coordinates of the seed points are normalized according to the radius of the cylinder and connected with the feature vector to obtain a grasping candidate point of size M×K×(3+C). The feature vector is obtained by summing the features of the point vector of the seed point and the view vector.

[0048] Furthermore, the final crawling suggestion is specifically:

[0049] The candidate points are processed by multi-layer perceptron and maximum pooling layer, and the output is a feature vector of size M×C. The SWAD Net output contains the scores and widths of different in-plane rotations and proximity depth combinations, that is, the output size is M×(A×D×2), where A represents the number of in-plane rotation angles, D represents the depth number, and 2 represents the score and width. The combination with the highest score is selected as the grasp prediction, and the friction coefficient μ is used. i The estimated grasping score is as follows:

[0050]

[0051] In the formula, μ max Indicates the maximum friction coefficient, μ min Indicates the minimum friction coefficient.

[0052] Furthermore, the overall loss function L of the grasping posture detection model is as follows:

[0053] L=L o +α(L p +λL v )+β(L s +L w )

[0054] Where, L o is the cross entropy loss function for object classification, L p is the smooth-l1 loss function of the point vector graspable scene, L v is the smooth-l1 loss function of the scene that the view vector can grasp, L s is the smooth-l1 loss function of the grasping score, L w is the smooth-l1 loss function of the gripper width, α and β are hyperparameters used to balance the importance of different tasks, and λ is a hyperparameter used to balance the two regression tasks of point vector graspable scenes and view vector graspable scenes.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] The method of the invention is suitable for posture estimation in complex scenes. By designing a DDSER module in the point encoder-decoder part, important channel features related to the grasping information are extracted. Secondly, after the farthest point sampling technology in the grasping area, the designed MSCAM module is proposed to avoid redundant features and enhance the global representation of the channel features of the seed point vector, thereby improving the detection capability of the grasping posture and improving the overall detection accuracy of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of grasping posture detection based on attention mechanism of the present invention;

[0058] Figure 2 It is a schematic diagram of the backbone network DDSER-UNet of the present invention;

[0059] Figure 3 is a schematic diagram of a DDSE module of the present invention;

[0060] Figure 4 It is a schematic diagram of the MSCAM module of the present invention;

[0061] Figure 5 is a schematic diagram of the affinity matrix of the present invention;

[0062] Figure 6 It is a schematic diagram of the grasping posture representation of the present invention;

[0063] Figure 7 This is a visualization of the pose estimation results of different algorithms. DETAILED DESCRIPTION

[0064] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0065] Example 1

[0066] A grasping posture detection method based on an attention mechanism in this embodiment includes the following steps:

[0067] A grasping posture detection method based on an attention mechanism comprises the following steps:

[0068] Step 1: Collect the RGB-D image of the object to be grasped and convert the RGB-D image into a point cloud form;

[0069] Step 2: Input the point cloud into the trained grasping posture detection model, perform posture estimation, and output the grasping posture; the grasping posture detection model is as follows: Figure 1 As shown, it includes: point cloud feature extraction, grasping area and final grasping suggestion. The point cloud feature extraction uses DDSER-UNet as the backbone network, introduces dual-channel dual-branch attention, integrates the jump connection mechanism to extract point cloud channel features, generates a graspable area, and uses a multi-layer perceptron to calculate the graspable area score. The introduction of dual-channel dual-branch attention can allow the network to consider different feature representations more comprehensively, thereby improving the network's representation ability and performance; the grasping area uses the farthest point sampling FPS to sample seed points from the grasping area, uses the MSCAM module to extract the channel feature vector of the sampled seed points, generates and selects the most suitable graspable view, uses a cylinder combined with the feature sum of the point vector and the view vector for grouping and sampling, and generates grasping candidate points; the final grasping suggestion uses a maximum pooling layer and a multi-layer perceptron to screen and evaluate grasping candidate points and generate the best graspable posture.

[0070] The camera is used to capture the RGB-D image and provide the position and orientation information of the object relative to the camera, so that the grasping detection model can accurately estimate the pose according to the camera perspective. The grasping pose G is defined as follows:

[0071] G=[R,t,W]

[0072] In the formula, Indicates the grabbing direction. Indicates the grab position, Indicates the grasping width suitable for grasping the target object.

[0073] 1) Point cloud feature extraction:

[0074] Taking the point cloud as input, the DDSER-UNet point encoder-decoder architecture is designed as the backbone network, as shown in Figure 2As shown in the figure, the feature vector of the input channel is 3 and the output channel is 512. The first convolutional layer of the encoder uses batch normalization and ReLU function to simplify feature extraction. After a convolutional layer of feature extraction, the output channel is 32. The encoder adds a downsampling layer after each DDSER module, and the final output channel size is 256. The input of the decoder includes the output channel feature Q of the last layer and the channel feature P in the encoder. For example, if the channel sizes of Q and P are 192 and 64, upsampling is first performed on Q and cropping is performed on P, and then the two outputs are spliced ​​in the channel dimension with a channel size of 256. Upsampling is performed on the basis of the DDSER module, and the output channel size is 192. Finally, the grasping area score is calculated by a multi-layer perceptron.

[0075] The DDSER module is as Figure 3 As shown, specifically: First, enter u c After three convolutional layers, the residual layer output result r is obtained c , where the size of the convolution kernel is 3×3; secondly, the squeezing operation uses the average pooling layer (AveragePooling) to c Perform the extrusion operation to obtain z c , as follows:

[0076]

[0077] In the formula, F sq (r c ) represents the squeezing function, N represents the point set, r c (i) represents the value of the cth channel at the i-th point;

[0078] Finally, the excitation operation, for the output of the squeeze operation, in the local branch, uses the splicing of two excitation processes to adjust the channel weights, and completes the dimensionality reduction and dimensionality increase operations through two convolutional layers and ReLU activation functions. In order to avoid suboptimal results, two different reduction ratios are set, r and 2r, respectively, as shown in the following formula:

[0079] F r =W 21 δ(W 11 ,z c )

[0080] F 2r =W 22 δ(W 12 ,z c )

[0081] Where W 11 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of r, W 21 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of r, W12 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of 2r, W 22 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of 2r, and δ represents the ReLU activation function;

[0082] The output results of these two different reduction ratios are spliced ​​to obtain the local feature L c , the shape is C×N, as follows:

[0083] L c =Concat(F r ,F 2r )

[0084] In the formula, Concat represents the concatenation operation;

[0085] For the output of the squeeze operation, in the global branch, z c After being processed by the fully connected layer and the Softmax function, the global feature G is obtained. c , as follows:

[0086] G c =φ(W 13 ,z c )

[0087] In the formula, φ represents the Softmax function, W 13 Represents the weight parameters of the fully connected layer in the global branch;

[0088] The local feature L c With the global feature G c Multiply them and use the Sigmoid function to generate the final weight. The output s value is normalized and the value range becomes 0 to 1, as shown in the following formula:

[0089] s c =F ex (z,W)=σ(L c ·G c )

[0090] In the formula, F ex (z,W) represents the activation function, σ represents the Sigmoid function;

[0091] Will s c The result r of the residual layer output c The re-weighted scale operation is performed so that the irrelevant feature information values ​​tend to 0, and the relevant feature information is retained, and finally combined with the input u c Perform the sum operation to get the final output of the DDSER module As follows:

[0092] Fscale (r c ,s c )=r c ·s c

[0093]

[0094] In the formula, Addition operation.

[0095] The Multi-Layer Perceptron (MLP) calculates the grasping area score according to the point set channel features extracted by DDSER-UNet. The output result includes the graspable scene prediction value of N×1 dimension, the point vector score of N×1 dimension and the view vector score of N×1 dimension, thereby forming a total output of size N×3. After obtaining the graspable scene divided by points, the points with grasping scores greater than the threshold ω are selected according to the estimated point vector scores. Although these initial points may contain duplicate points or too many points to be effectively calculated, the farthest point sampling FPS is used to select M seed points to cover the entire graspable scene. This sampling method can maximize the distance between each selected point, thereby reducing computing resources; finally, the (3+C) dimensional feature M seed points are obtained through the MSCAM module, where 3 represents the point coordinate dimension and C represents the dimensional feature.

[0096] 2) Grab area:

[0097] According to the graspable scene, M seed points are selected by the farthest point sampling FPS. The (3+C)-dimensional feature M seed points are obtained through the MSCAM module. The view is generated by ApproachNet using the Fibonacci grid, and the M×V view vector is generated through its internal multi-layer perceptron structure. The best view is selected for subsequent prediction, and then the Cylinder Group cylinder combines the point vector and the view vector to perform grouping and sampling to generate grasping candidate points.

[0098] The MSCAM module is as follows Figure 4 and Figure 5 Specifically, in order to highlight the interaction between different channel features, enhance the global feature representation of the seed point vector channel direction, avoid generating redundant features when FPS is aggregated, construct multi-head channel self-attention, and calculate the channel-by-channel attention strategy, as shown in the following formula:

[0099]

[0100] In the formula, Y represents the number of self-attention blocks, F l ∈R M×C Indicates that the channel feature map is the lth layer, AT(·) is as follows:

[0101]

[0102] Based on M c Calculate affinity matrix Measure the difference between channels as follows:

[0103]

[0104]

[0105] Where M c represents the mth head channel attention matrix, m represents the index of the attention head, represents the affinity matrix, φ represents the Softmax function, and exp and (·) repeats the max pooling (M c ) is extended to M c Matrices of the same size are subtracted, and max pooling (M c )∈R d×1 Extract M c The maximum value of each row in are the learnable weight parameters of the three linear layers, d c = C / Y. The larger the magnitude of the element in the affinity matrix, the lower the similarity of the corresponding two channels. Therefore, the MSCAM module tends to focus on channels with significant differences and avoid aggregating similar and redundant information.

[0106] The MLP of the ApproachNet internal structure is applied to the seed point after the MSCAM module operation and outputs a vector of size M×V for generating the view vector. At the same time, it generates features of size M×C for the seed point for subsequent grasp generation. V represents the number of views. To obtain the view grasp score, V views are sampled from the unit sphere using the Fibonacci grid. From these views, the view with the best grasp score is selected for subsequent prediction. Then the point view score and the extracted features can be used to estimate the grasp score, grasp width, and grasp pose.

[0107] The Cylinder Group cylinder is specifically: the position and direction of the cylinder space are determined by the seed point coordinates and the view vector respectively. For each pair of M seed points-views, a cylinder with a fixed height d and radius r is used to group and sample the K points in the M seed points. After the cylinder is aligned with the fixture frame, the coordinates of the seed points are normalized according to the radius of the cylinder and connected with the feature vector to obtain a grasping candidate point of size M×K×(3+C). The feature vector is obtained by summing the feature of the seed point vector and the view vector.

[0108] 3) Final crawling suggestions:

[0109] The candidate points are processed by a multi-layer perceptron and a maximum pooling layer, and the output is a feature vector of size M×C. The SWAD Net output contains the scores and widths of different in-plane rotations and proximity depth combinations, that is, the output size is M×(A×D×2), where A represents the number of in-plane rotation angles, which is divided into 12 categories (15° / category), D represents the depth number, which is divided into 4 categories (0.01m, 0.02m, 0.03m, 0.04m), and 2 represents the score and width. The combination with the highest score is selected as the grasp prediction, and the friction coefficient μ is used. i The estimated grasping score is as follows:

[0110]

[0111] In the formula, μ max Indicates the maximum friction coefficient, μ min represents the minimum friction coefficient. The grasping posture is represented as Figure 6 shown.

[0112] In addition, the overall loss function L of the grasping posture detection model described in this embodiment is as follows:

[0113] L=L o +α(L p +λL v )+β(L s +L w )

[0114] Where, L o is the cross entropy loss function for object classification, L p is the smooth-l1 loss function of the point vector graspable scene, L v is the smooth-l1 loss function of the scene that the view vector can grasp, L s is the smooth-l1 loss function of the grasping score, L w is the smooth-l1 loss function of the gripper width, α and β are hyperparameters used to balance the importance of different tasks, λ is a hyperparameter used to balance the two regression tasks of point vector graspable scenes and view vector graspable scenes, α, β, λ are set to 10.

[0115] The smooth-l1 loss function is a loss function used for regression tasks. It combines the advantages of the L1 loss function (absolute error) and the L2 loss function (square error). It is usually used for robust regression because it exhibits the smoothness of the L2 loss function when the error is small, and the robustness of the L1 loss function when the error is large. p and L sWhen L v Calculate the view at the seed point; use L w Compute the grasp poses with scores greater than zero.

[0116] Example 2

[0117] GraspNet-1Billion is used as a data set for the grasping posture detection task based on the attention mechanism of the present invention. GraspNet-1Billion is a large data set for grasping posture detection, which contains 190 scenes and 256 different views. These scenes are captured by two cameras (RealSense / Kinect). It is divided into training set, test seen set, test similar set and test unseen set, with 100, 30, 30, and 30 scenes respectively. In this data set, 88 objects provide accurate 3D mesh models, and the objects in each scene are densely labeled, including 6D posture and grasping posture, with a total of more than 1 billion grasping postures. Each frame of data is associated with the posture of the camera, which is convenient for the fusion of multi-viewpoint clouds. This embodiment is mainly aimed at the RealSense camera test scene, and is divided into three parts according to the object category (Seen / Similar / Novel).

[0118] In order to perform training and prediction, it is necessary to generate corresponding label files for the input data. By generating label files, this article can match the label data with the input data in order to achieve task objectives such as object classification or posture estimation. The label file provides label information related to each data sample for supervised learning or evaluating algorithm performance. The optimizer is Adam. The advantages of this optimizer are simple implementation, efficient calculation, low memory requirements, and the ability to automatically adjust the learning rate. It is suitable for scenarios with large-scale data and parameters. The batch size is 4 and the number of epochs is 10. During the training process, the initial learning rate is 0.001, and the learning rate is multiplied by 0.95 every other epoch. It takes about 1 day for the network model to converge, and the effectiveness of the model is verified in real scenarios. The results are obtained through multiple tests. See Table 1 and Figure 7 .

[0119] Table 1 Comparison of different algorithms for grasping and detection on GraspNet-1Billion (unit: %)

[0120]

[0121]

[0122] Table 1 lists the comparison of the method of the present invention with the recent methods on the GraspNet-1Billion dataset, and finds that the method ARGDNet proposed by the present invention is superior to the related methods, with an average accuracy of 71.89% for seen objects, 61.29% for similar objects, and 26.71% for new objects, which is more than 4% higher than the previous most advanced method GSNet in terms of evaluation indicators. Since GS Net does not introduce attention and does not combine FPS with global information, which affects the missing of graspable points and inaccurate pose estimation, the method of the present invention can solve the problem of incomplete and unreliable pose estimation production, thereby helping to improve the average grasping accuracy.

[0123] Figure 7 The visualization results of the method of the present invention are displayed. In order to intuitively verify the ability of the method of the present invention to grasp pose estimation, three scenes with viewing angles of 106, 108, and 121 are selected from the Seen scenes, one scene with a viewing angle of 148 is selected from the Similar scenes, and one scene with a viewing angle of 175 is selected from the Novel scenes. Figure 7 Qualitative results of the grasping network are presented, the resulting output of the grasping posture is visualized, and it is compared with the GSNet method. Some important results are marked in the figure, and black circles indicate failed grasps. Figure 7 It can be seen that the GSNet method did not generate effective grasping postures for the white skin care product bottles, blue buckets, and red strawberries in the Seen class 106, 108, and 121 scenes, the pink soap in the Similar class 148 scenes, and the white clip in the Novel class 175 scenes. However, the method of the present invention generates correct grasping postures for these objects and obtains reasonable grasping.

[0124] The above shows and describes the main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0125] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A grasping posture detection method based on attention mechanism, characterized in that: The following steps are involved: Step 1: Collect the RGB-D image of the object to be grasped and convert the RGB-D image into a point cloud form; Step 2: Input the point cloud into the trained grasping posture detection model, perform posture estimation, and output the grasping posture; The grasping posture detection model includes: point cloud feature extraction, grasping area and final grasping suggestion; the point cloud feature extraction uses DDSER-UNet as the main network, introduces dual-channel dual-branch attention, integrates the jump connection mechanism to extract point cloud channel features, generates a graspable area, and uses a multi-layer perceptron to calculate the graspable area score; the grasping area uses the farthest point sampling FPS to sample seed points from the grasping area, uses the MSCAM module to extract the channel feature vector of the sampled seed points, generates and selects the most suitable graspable view, uses a cylinder to combine the feature sum of the point vector and the view vector for grouping and sampling, and generates grasping candidate points; the final grasping suggestion uses a maximum pooling layer and a multi-layer perceptron to screen and evaluate grasping candidate points and generate the best graspable posture.

2. According to the method for grasping posture detection based on attention mechanism in claim 1, it is characterized in that: The point cloud feature extraction is specifically as follows: The point cloud is taken as input, and the DDSER-UNet point encoder-decoder architecture is designed as the backbone network. The input channel is 3, and the output channel is 512 feature vectors. The first convolution layer of the encoder uses batch normalization and ReLU function. After a convolution layer of feature extraction, the output channel is 32. A downsampling layer is added after each DDSER module of the encoder, and the final output channel size is 256. The input of the decoder includes the output channel feature Q of the last layer and the channel feature P in the encoder. First, upsampling is performed on Q and cropping operations are performed on P, and then the two outputs are spliced ​​in the channel dimension. The upsampling operation is performed on the basis of the DDSER module, and finally the grasping area score is calculated by a multi-layer perceptron.

3. According to the method of grasping posture detection based on attention mechanism in claim 2, it is characterized in that: The DDSER module is specifically: Enter u c After three convolutional layers, the residual layer output result r is obtained c , where the size of the convolution kernel is 3×3; Squeeze operation, use the average pooling layer to r c Perform the extrusion operation to obtain z c , as follows: In the formula, F sq (r c ) represents the squeezing function, N represents the point set, r c (i) represents the value of the cth channel at the i-th point; The excitation operation, for the output of the squeeze operation, in the local branch, uses the splicing of two excitation processes to adjust the channel weights, and completes the dimensionality reduction and dimensionality increase operations through two convolutional layers and ReLU activation functions, setting two different reduction ratios, r and 2r, respectively, as shown in the following formula: F r =W 21 δ(W 11 ,z c ) F 2r =W 22 δ(W 12 ,z c ) Where W 11 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of r, W 21 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of r, W 12 represents the weight parameter of the first fully connected layer of the branch with a reduction ratio of 2r, W 22 represents the weight parameter of the second fully connected layer of the branch with a reduction ratio of 2r, and δ represents the ReLU activation function; The output results of these two different reduction ratios are spliced ​​to obtain the local feature L c , the shape is C×N, as follows: L c =Concat(F r ,F 2r ) In the formula, Concat represents the concatenation operation; For the output of the squeeze operation, in the global branch, z c After being processed by the fully connected layer and the Softmax function, the global feature G is obtained. c , as follows: G c =φ(W 13 ,With c ) In the formula, φ represents the Softmax function, W 13 Represents the weight parameters of the fully connected layer in the global branch; The local feature L c With the global feature G c Multiply them together and use the Sigmoid function to generate the final weight. The output s value is normalized and the value range becomes 0 to 1, as shown in the following formula: s c =F ex (z,W)=σ(L c ·G c ) In the formula, F ex (z,W) represents the activation function, σ represents the Sigmoid function; Will s c The result r of the residual layer output c Perform a reweighted scale operation and finally sum it with the input u c Perform the sum operation to get the final output of the DDSER module As follows: F scale (r c ,s c )=r c ·s c In the formula, Addition operation.

4. According to the method of grasping posture detection based on attention mechanism in claim 2, it is characterized in that: The multilayer perceptron calculates the grasping area score according to the point set channel features extracted by DDSER-UNet. The output result includes the graspable scene prediction value of N×1 dimension, the point vector score of N×1 dimension and the view vector score of N×1 dimension, thereby forming a total output of size N×3. After obtaining the graspable scene divided by points, the points with grasping scores greater than the threshold ω are selected according to the estimated point vector scores.

5. According to the method of grasping posture detection based on attention mechanism in claim 4, it is characterized in that: The grabbing area is specifically: According to the graspable scene, M seed points are selected by the farthest point sampling FPS. The (3+C)-dimensional feature M seed points are obtained through the MSCAM module. The view is generated by ApproachNet using the Fibonacci grid, and the M×V view vector is generated through its internal multi-layer perceptron structure. The best view is selected for subsequent prediction, and then the Cylinder Group cylinder combines the point vector and the view vector to perform grouping and sampling to generate grasping candidate points.

6. The method for grasping posture detection based on attention mechanism according to claim 5, characterized in that: The MSCAM module is specifically: Construct multi-head channel self-attention and calculate the strategy of channel-by-channel attention as follows: In the formula, Y represents the number of self-attention blocks, F l ∈R M×C Indicates that the channel feature map is the lth layer, AT(·) is as follows: Based on M c Calculate affinity matrix Measure the difference between channels as follows: Where M c represents the mth head channel attention matrix, m represents the index of the attention head, represents the affinity matrix, φ represents the Softmax function, and expand(·) converts maxpooling(M c ) is extended to M c Matrices of the same size are subtracted, maxpooling (M c )∈R d×1 Extract M c The maximum value of each row in are the learnable weight parameters of the three linear layers, d c =C / Y.

7. The method for grasping posture detection based on attention mechanism according to claim 5, characterized in that: The Cylinder Group cylinder is specifically: For each pair of M seed points and views, a cylinder with a fixed height d and radius r is used to group and sample K points among the M seed points. After the cylinder is aligned with the fixture frame, the coordinates of the seed points are normalized according to the radius of the cylinder and connected with the feature vector to obtain a grasping candidate point of size M×K×(3+C). The feature vector is obtained by summing the features of the point vector of the seed point and the view vector.

8. The method for grasping posture detection based on attention mechanism according to claim 1, characterized in that: The final crawling suggestions are as follows: The candidate points are processed by multi-layer perceptron and maximum pooling layer, and the output is a feature vector of size M×C. The SWAD Net output contains the scores and widths of different in-plane rotations and proximity depth combinations, that is, the output size is M×(A×D×2), where A represents the number of in-plane rotation angles, D represents the depth number, and 2 represents the score and width. The combination with the highest score is selected as the grasp prediction, and the friction coefficient μ is used. i The evaluation grasping score is as follows: In the formula, μ max Indicates the maximum friction coefficient, μ min Indicates the minimum friction coefficient.

9. A grasping posture detection method based on an attention mechanism according to any one of claims 1 to 8, characterized in that: The overall loss function L of the grasping posture detection model is as follows: L=L o +α(L p +λL v )+β(L s +L w ) Where, L o is the cross entropy loss function for object classification, L p is the smooth-l1 loss function of the point vector graspable scene, L v is the smooth-l1 loss function of the scene that the view vector can grasp, L s is the smooth-l1 loss function of the grasping score, L w is the smooth-l1 loss function of the gripper width, α and β are hyperparameters used to balance the importance of different tasks, and λ is a hyperparameter used to balance the two regression tasks of point vector graspable scenes and view vector graspable scenes.

Images