Robotic arm grasping posture detection method based on multimodal fusion of RGB-D images
Through the multimodal fusion method based on RGB-D images, the key features in the robotic arm grab position detection are extracted using CA attention and hollow convolution branches, and the problems of poor real-time performance and incomplete feature information in the prior art are solved, and the problem of incomplete real-time information is realized. Effective grasping position detection in a multi-object environment.
Patent Information
- Application Number
- CN202410162069.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-02-05
AI Technical Summary
The prior art has problems in the detection of robotic arm grab position poses with poor real-time performance, incomplete feature information extraction, and inability to generate multiple grab position poses, making it difficult to complete grab position in multiple object environments.
Using a multimodal fusion method based on RGB-D images, the images are cropped by random inversion and rotation, combined with sigmoid activation and average pooling operations, and the area, angle and width information are extracted using the CA attention mechanism and the hollow convolution branch to generate thermal forces. Figure to determine the final grab point, rectangle, and angle.
It improves the richness and fine-grainedness of feature information, enhances the generalization ability of the model, can acquire and capture target features faster, reduce the impact of environmental background, and achieve stronger feature expression and generalization ability.
Smart Images

Figure CN118052874B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot vision technology, and in particular to a robot arm grasping posture detection method based on multi-modal fusion of RGB-D images. Background Art
[0002] The progress of science and technology promotes the development of the robot industry. With its highly automated advantages, the application scale of robotic grasping detection in the fields of medical care, education, public services, etc. is expanding year by year. Robotic grasping detection refers to finding a feasible grasping posture that meets the requirements of the grasping task on a given object through an algorithm. Robot grasping detection technology is divided into two categories. One is the traditional analysis method, also known as the hard-coding method, which refers to the grasping detection problem based on object geometry, physical model, kinematics and mechanical analysis. The objects faced by traditional analysis methods usually refer to objects with complete geometric three-dimensional models and grasping posture libraries. The robot can access a geometric object model database containing many good candidate grasping postures. This database is usually built offline and is called a grasping experience database. The typical process of object grasping is to process each object in the object model database in the offline stage, generate candidate grasping postures, score and sort them based on certain selection criteria, store the grasping postures in the grasping experience database, and correspond one-to-one with the object model database for retrieval. In the online stage, the scene is segmented, objects are identified and the corresponding objects in the object model database are found. Then the object posture is estimated, and the grasping posture in the grasping experience database is found based on the posture retrieval. Finally, grasping selection and reachability filtering are performed to execute grasping.
[0003] The other type is the grasping method based on experience, also known as the data-driven method. It imitates human grasping strategies to perform grasping detection, avoiding the calculation of mathematical and physical models. The analytical method needs to satisfy some kinematic and dynamic formulas when determining grasping. Because there are many conditions to be satisfied, the calculation is complicated. The analytical method is a general strategy used in grasping detection. First, grasping detection is performed based on environmental constraints and the manipulator and object model. Based on certain selection criteria and task orientation, a set of grasping postures output by the scene is selected, and the best grasping posture is output, and finally passed to the robot to perform grasping.
[0004] The analytical method must set some assumptions to predict the grasping posture, such as being able to accurately obtain the model parameters of the object to be grasped, the environmental parameters, and being able to reasonably simplify the fixture model. However, these assumptions are often difficult to establish in real grasping scenarios. In addition, the analytical method requires three-dimensional modeling of known objects, but it is often difficult to completely and accurately reconstruct the actual object seen for the first time through vision. The empirical method is highly dependent on previous grasping experience. Using the empirical method to detect the grasping posture must establish a complete grasping experience database. However, it is impractical to establish such a database, and it is impossible to collect all grasping postures. In addition, with the reality of increasingly complex environments and diversified grasping tasks, the computational complexity of the empirical method is also increasing, and it is difficult to meet the task requirements in actual scenarios.
[0005] The existing technology applies deep learning to the field of grasping posture detection. It uses a sliding window to traverse the entire image, inputs each image block into the first-level network for learning, obtains the approximate grasping position, and then learns a more accurate position through a second-level deeper network. However, this two-stage detection method requires traversing the entire image, which is very time-consuming and has poor real-time performance in actual application.
[0006] The existing technology also uses two parallel network structures to perform multimodal fusion grasping detection models, with the inputs being RGB images and three-channel depth images. First, two deep convolutional neural networks ResNet-50 are used to extract features from the scene respectively, and then the feature maps obtained by the two parallel networks are fused, and finally a shallow convolutional neural network is used to predict the grasping position of a given object. Compared with previous models, this method greatly improves the running speed and realizes real-time detection. The model uses two deep convolutional neural networks to extract image RGB and depth information to extract features from the scene respectively. However, the network ignores the feature maps generated by the hidden layers of the two deep convolutional neural networks. The features of the intermediate layers may imply other important information of the detected object.
[0007] Existing techniques have also overcome the limitations of individual models by combining convolutional neural network structures with feature fusion and generating grasps with confidence scores at different levels of the image hierarchy. However, the algorithm is unable to generate multiple grasp poses, which makes it difficult to complete grasps in environments with multiple objects.
[0008] In order to solve the above problems, the present invention proposes a robot arm grasping posture detection method based on multimodal fusion of RGB-D images. Summary of the invention
[0009] The purpose of the present invention is to propose a robot arm grasping posture detection method based on multimodal fusion of RGB-D images to solve the problems raised in the background technology:
[0010] The existing technology needs to traverse the entire image, which is very time-consuming and has poor real-time performance in actual application; the extraction of implicit information of feature maps is not comprehensive; and it is unable to generate multiple grasping postures, which makes it difficult to complete grasping in an environment with multiple objects.
[0011] In order to achieve the above object, the present invention adopts the following technical solutions:
[0012] The robot grasping posture detection method based on multimodal fusion of RGB-D images includes:
[0013] S1: Randomly reverse and rotate the cropped image to a 320*320 input image with the object to be grasped as the center, and collect the RGB image and depth image of the input image;
[0014] S2: Perform sigmoid activation and average pooling operations on the RBG image and depth image respectively to generate weights corresponding to the two feature information, and then assign the weight values to the respective feature information through element-by-element multiplication operations, and then add the two features element-by-element, and finally perform CA attention operations before output;
[0015] S3: After the RGB image and the depth image are fused, the dilated convolution branches with different dilation rates are used to capture the region information, angle information, and width information of the image features output in S2, and then the output of each branch is reduced in dimension and integrated;
[0016] S4: The output results in S3 are passed through four convolutional layers to generate four heat maps with a resolution of 320*320, which correspond to the score map, width map, rotation angle sin component map and rotation angle cos component map respectively, and the final grasping point, grasping rectangle and grasping angle are determined based on each heat map.
[0017] Preferably, the weights corresponding to the two types of feature information in S2 are as follows:
[0018]
[0019]
[0020] Among them, conv() represents the convolution operation; sigmoid() represents the sigmoid function activation operation; Avg() represents the average pooling operation; f i r 、f i d Represents two kinds of feature information, Corresponding to f i r 、f i d The weights of the two feature information;
[0021] The CA attention operation outputs the result f i rd The details are as follows:
[0022]
[0023] Among them, + represents element-by-element addition; × represents element-by-element multiplication; CA represents CA attention operation.
[0024] Preferably, the step of capturing the regional information in S3 is as follows:
[0025] S3.1.1: Perform BatchNorm and Relu operations on the 1×1 convolution kernel f1 with a dilation of 1 and the 1×1 convolution kernel f6 with a dilation of 6 respectively, and then concatenate them;
[0026] S3.1.2: Then change the convolution kernel f to the same size as the low-dimensional fusion feature by bilinear interpolation upsampling l Perform splicing operations;
[0027] S3.1.3: After convolution operation, the dimension is reduced and the 3×3 convolution kernel f with dilation 12 is used. 12 and a 3×3 convolution kernel f with a dilation of 18 18 The spliced feature information is spliced;
[0028] S3.1.4: Finally, the output is passed through a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU. The specific output results are as follows:
[0029] f i s =conv(conv(up(cat(f1,f6))+f l )+up(cat(f 12 ,f 18 )))
[0030] Among them, f i d Indicates the result of regional information capture; up indicates bilinear interpolation upsampling; conv() indicates convolution operation; cat() indicates concatenation operation;
[0031] The angle information grabbing steps are as follows:
[0032] S3.2.1: First, obtain the 3×3 convolution kernel f6 with a dilation of 6 and the 3×3 convolution kernel f with a dilation of 12 12Perform BatchNorm and Relu operations respectively and then perform concatenation operation;
[0033] S3.2.2: Then change the bilinear interpolation upsampling method to the same size convolution kernel f as the low-dimensional fusion feature l Perform splicing operations;
[0034] S3.2.3: Pass through a convolutional layer;
[0035] S3.2.4: After a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU, the specific output results are as follows:
[0036] f i θ =conv(conv(up(cat(f6,f 12 ))+f l ))
[0037] Among them, f i θ Indicates the angle information capture result;
[0038] The steps for capturing width information are as follows:
[0039] S3.3.1: First, obtain the 3×3 convolution kernel f with a dilation of 12 12 and a 3×3 convolution kernel f with a dilation of 18 18 The results of the BatchNorm and Relu operations are concatenated, and the concatenation is performed with the 1×1 convolution kernel f1 with a dilation of 1 and the 1×1 convolution kernel f6 with a dilation of 6 to supplement the feature information.
[0040] S3.3.2: After passing through a convolutional layer, a bilinear interpolation upsampling operation is performed;
[0041] S3.3.3: After a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU, the specific output results are as follows:
[0042] f i w =conv(up(conv(cat(f 12 ,f 18 ))+cat(f1,f6)))
[0043] Among them, f i w Indicates the result of fetching width information.
[0044] Preferably, in S4, the grasping angle information is replaced by sine and cosine, and the grasping angle is restored by calculation, and the output result is:
[0045]
[0046]
[0047]
[0048]
[0049] Where g represents the grasping posture predicted by the grasping detection model, g score , g sin , g cos and g width They represent the confidence score of the predicted grasped g, the sin and cos components of the rotation angle, and the width of the grasped prediction box; max score Indicates the maximum confidence score; x n Represents the data input from the grasp detection network to the grasp prediction module; Represents the four convolution operations that the grasp detection network processes on the input data.
[0050] Preferably, in S4, the grasping point is determined by the pixel with the maximum pixel value in the grasping component map, the width of the grasping rectangle is determined by the pixel value of the width map, and the grasping angle θ is determined by the following formula:
[0051]
[0052] Among them, sin2θ and cos2θ are represented by the pixel values of the sin component and cos component heat map.
[0053] Preferably, in S4, the grasping detection problem is transformed into an end-to-end regression problem, and a mapping relationship F:I→G is obtained through model training, where I is the detection image and G is the correct grasping position and posture, which is used to minimize the prediction The distance between the actual value G, where S represents the grasping score range of each pixel in the image, W represents the grasping width of each pixel, and the network loss function L is defined as follows:
[0054]
[0055]
[0056] Where n represents the number of samples that can be captured, and Smooth(x,y) represents the smoothing function.
[0057] Preferably, in S4, if the grasping posture predicted by the model and the target satisfy both of the following two conditions, the prediction result is judged to be correct:
[0058] |G θ -G θ '|<30°
[0059]
[0060] Among them, G θ Represents the angle of the predicted grab box G; G θ ' represents the angle of the labeled rectangular box G'; the Jaccard index represents the intersection-over-union ratio of the predicted grab box and the labeled rectangular box.
[0061] Compared with the prior art, the present invention provides a robot arm grasping posture detection method based on multimodal fusion of RGB-D images, which has the following beneficial effects:
[0062] 1. This paper proposes a new RGB-D feature fusion method based on attention mechanism. The convolutional neural network and CA attention mechanism are used to fuse the two features, so that the useful feature information obtained by each level module when the subsequent network extracts features is richer and retains more fine-grained features.
[0063] 2. The present invention proposes a hybrid atrous pyramid module method. Multiple atrous convolution branches are used, each branch uses a different dilation rate, so as to capture contextual information at different scales. In order to make the information captured by each task clearer, each branch is combined according to the dilation coefficient. Then the output of each branch is subjected to appropriate dimensionality reduction and integration operations. Such operations can make the grasping detection model pay more attention to the grasping target, enable the model to obtain the characteristic information of the grasping target faster, reduce the influence of environmental background factors on grasping detection, and enhance the generalization ability of the model.
[0064] 3. The present invention proposes a new grasping detection model. The detection model uses a hybrid dilated convolutional architecture to extract detection images, so that the model has excellent global feature association and modeling capabilities. Compared with the traditional grasping detection model based on convolutional neural networks, the grasping detection model proposed in the present invention has stronger feature expression and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is a general schematic diagram of the network structure mentioned in Example 1 of the present invention;
[0066] Figure 2 This is a schematic diagram of feature fusion mentioned in Example 1 of the present invention;
[0067] Figure 3 This is a schematic diagram of the mixed hollow pyramid module mentioned in Example 1 of the present invention;
[0068] Figure 4 Schematic diagram of the posture prediction convolutional layer structure mentioned in Example 1 of the present invention;
[0069] Figure 5 This is a schematic diagram of grabbing mentioned in Example 1 of the present invention;
[0070] Figure 6 This is a visualization diagram of some experimental results on the Jacquard dataset mentioned in Example 2 of the present invention. DETAILED DESCRIPTION
[0071] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0072] The present invention introduces a robot gripping posture detection method based on multimodal fusion of RGB-D images. First, the original image is cropped to retain the input RGB-D image with a size of 320*320. After the two formats of images are input into the network, a feature cross-fusion module is used to fuse the feature outputs of each sub-module of the depth image and the RGB image, retaining the feature information of each level, and then input into the improved gripping hole convolution module, and finally the gripping posture prediction is performed. Specifically include the following contents.
[0073] Embodiment 1:
[0074] See also Figure 1-4 The present invention provides a robot arm grasping posture detection method based on multimodal fusion of RGB-D images, comprising the following steps:
[0075] S1: Randomly reverse and rotate the cropped image to a 320*320 input image with the object to be grasped as the center, and collect the RGB image and depth image of the input image; the details are as follows:
[0076] Randomly invert and rotate the cropped image with the object to be grasped as the center to be a 320*320 input image. Preprocessing not only enriches the training data of the model, but also increases the generalization ability of the model. Then collect the RGB image and depth image of the input image as the input image. The depth image records the distance or depth information from each pixel to the camera. The depth image can be used to calculate information such as the distance, shape, and position of objects in the scene. There are many ways to obtain depth images, including using lidar, structured light sensors, RGB-D cameras, and binocular or multi-camera parallax information to calculate depth. By combining the RGB image with the depth image as input, the grasping posture can be accurately predicted and the grasping distance can be mechanically transmitted.
[0077] S2: Perform sigmoid activation and average pooling operations on the RBG image and the depth image respectively to generate weights corresponding to the two feature information, and then assign the weight values to the respective feature information through element-by-element multiplication operations, and then add the two features element-by-element, and finally perform CA attention operations before output; the details are as follows:
[0078] Reference Figure 2 , perform sigmoid activation and average pooling operations on the RGB image and the depth image respectively, and then generate the weights of the features. The output result can be expressed as:
[0079]
[0080]
[0081] Among them, conv() represents the convolution operation; sigmoid() represents the sigmoid function activation operation; Avg() represents the average pooling operation; f i r 、f i d Represents two kinds of feature information, Corresponding to f i r 、f i d The weights of the two feature information;
[0082] Next, the weight values are assigned to the respective feature information through element-by-element multiplication, and then the two features are added. Finally, a CA attention operation is performed before the output to capture the relationship between the two and improve the accuracy of feature expression. The result can be expressed as
[0083]
[0084] Among them, + represents element-by-element addition; × represents element-by-element multiplication; CA represents CA attention operation. The attention mechanism is an attention mechanism used to strengthen the deep learning model's understanding of the spatial structure of the input data. The core idea of the CA attention mechanism is to introduce coordinate information so that the model can better understand the relationship between different positions.
[0085] S3: After the RGB image and the depth image are fused, the dilated convolution branches with different dilation rates are used to capture the region information, angle information, and width information of the image features output in S2, and then the output of each branch is reduced in dimension and integrated; the details are as follows:
[0086] Reference Figure 3 After the RGB image and the depth image are fused, the dilated convolution with dilation rate of 1, 6, 12, and 18 is used to obtain the feature information of different receptive fields.
[0087] In the prediction of the captured area, the obtained 1×1 convolution kernel, dilation is 1 and 1×1 convolution kernel, dilation is 6, are first subjected to BatchNorm and Relu operations respectively, and then cat spliced together. Then, the bilinear interpolation upsampling method is changed to the same size as the low-dimensional fusion feature for splicing. Then, the dimension is reduced through convolution operation and the 3×3 convolution kernel, dilation is 12 and 3×3 convolution kernel, dilation is 18, and the spliced feature information is spliced (to ensure the same size, the features are also bilinearly interpolated and upsampled). Finally, it is output through a three-layer convolution layer consisting of Conv2d, BatchNorm, and ReLU. The specific output results are as follows:
[0088] f i s =conv(conv(up(cat(f1,f6))+f l )+up(cat(f 12 ,f 18 )))
[0089] Among them, f i d Indicates the result of regional information capture; up indicates bilinear interpolation upsampling; conv() indicates convolution operation; cat() indicates concatenation operation;
[0090] In the captured angle feature information, the obtained 3×3 convolution kernel with dilation of 6 and 3×3 convolution kernel with dilation of 12 are first subjected to BatchNorm and Relu operations respectively, and then cat-joined together. Then, the bilinear interpolation upsampling method is changed to the same size as the low-dimensional fusion feature for splicing. Then, it passes through a convolution layer. Finally, it passes through a three-layer convolution layer consisting of Conv2d, BatchNorm, and ReLU for output. The specific output results are as follows:
[0091] f i θ =conv(conv(up(cat(f6,f 12 ))+f l ))
[0092] Among them, f i θ Indicates the angle information capture result;
[0093] In capturing width feature information, the obtained 3×3 convolution kernel, dilation 12 and 3×3 convolution kernel, dilation 18 are first subjected to BatchNorm and Relu operations, and then 1×1 convolution kernel is spliced with dilation 1 and 1×1 convolution kernel, dilation 6 for cat operation to supplement their feature information. Then, a bilinear interpolation upsampling operation is performed after passing through a convolution layer. Finally, a three-layer convolution layer consisting of Conv2d, BatchNorm, and ReLU is output. The result can be expressed as:
[0094] f i w =conv(up(conv(cat(f 12 ,f 18 ))+cat(f1,f6)))
[0095] Among them, f i w Indicates the result of fetching width information.
[0096] Each dilated convolution branch can capture contextual information at different scales by using different dilation rates. In order to make the information captured by each task clearer, each branch is combined according to the dilation coefficient. The output of each branch is then subjected to appropriate dimensionality reduction and integration operations. The hybrid dilated pyramid module design can improve the model's perception of objects of different scales and achieve better performance in grasping detection.
[0097] S4: The output results in S3 are passed through four convolutional layers to generate four heat maps with a resolution of 320*320, which correspond to the score map, width map, rotation angle sin component map and rotation angle cos component map respectively, and the final grasping point, grasping rectangle and grasping angle are determined based on each heat map. The details are as follows:
[0098] Reference Figure 4-Figure 5 , the grasping angle information is replaced by sine and cosine, and the grasping angle is restored by calculation. The output result is:
[0099]
[0100]
[0101]
[0102]
[0103] Where g represents the grasping posture predicted by the grasping detection model, g score , g sin , g cos and g width They represent the confidence score of the predicted grasped g, the sin and cos components of the rotation angle, and the width of the grasped prediction box; max score Indicates the maximum confidence score; x n Represents the data input from the grasp detection network to the grasp prediction module; Represents the four convolution operations that the grasp detection network processes on the input data.
[0104] The grasping point is determined by the pixel with the maximum pixel value in the grasping component map, the width of the grasping rectangle is determined by the pixel value of the width map, and the grasping angle θ is determined by the following formula:
[0105]
[0106] Among them, sin2θ and cos2θ are represented by the pixel values of the sin component and cos component heat map;
[0107] Through model training, we get a mapping relationship F:I→G, where I is the detection image and G is the correct grasping position and posture.
[0108] To minimize the prediction The distance between the actual value G, where w1 and h1 represent the width and height of the input grasp detection network image, respectively; S represents the grasp score range of each pixel in the image; W represents the grasp width of each pixel. The design of the convolutional layer is shown below. Its purpose is to map the input features to a lower-dimensional representation and ultimately generate an output related to the grasping task.
[0109] The 2D plane grasping pose is expressed as:
[0110] G={x,y,w,h,θ}
[0111] Where (x, y) represents the coordinates of the center point of the grab rectangle; w and h represent the width and height of the grab rectangle; θ represents the rotation angle of the grab rectangle relative to the horizontal direction of the image; the simplified grab representation of this embodiment is: The network's loss function L is defined as follows:
[0112]
[0113]
[0114] Where n represents the number of samples that can be captured; Smooth(x,y) represents the smoothing function.
[0115] If the grasping pose predicted by the model and the target meet both of the following conditions, the prediction is considered correct:
[0116] |G θ -G θ '|<30°
[0117]
[0118] Among them, G θ Represents the angle of the predicted grab box G; G θ ' represents the angle of the labeled rectangle G'; the Jaccard index represents the intersection-over-union ratio of the predicted grasping box and the labeled rectangle. The first condition indicates that the angle difference between the predicted grasping box and the labeled rectangle is less than 30°, which means that the difference in the grasping angles of the two can be in [-30°, 30°]. The second condition indicates that the intersection-over-union ratio (IOU) of the predicted grasping box and the labeled rectangle is greater than 25%; in grasping posture detection, this indicator is also called the Jaccard index. For robotic arm grasping, the goal of grasping detection is to detect a valid grasping rectangle in the image that meets the grasping task.
[0119] Embodiment 2:
[0120] Reference Figure 6The grasping detection network proposed in this embodiment is trained and tested on the Cornell grasping dataset and the Jacquard grasping dataset. The model is verified on the Cornell dataset using the cross-validation method. Image segmentation (IW) and object segmentation (OW) are used to prove the accuracy of the prediction. For the Jacquard dataset, 90% of the data was used as a training set, and the remaining 10% was used as a validation set, achieving grasping accuracy rates of 98.8% and 97.7% respectively. Tables 1 and 2 show the comparison of grasping detection results of the present invention and some current grasping detection methods with better detection effects on the Cornell and Jacquard grasping datasets.
[0121] Table 1 Cornell dataset crawling detection results
[0122]
[0123] Table 2 Jacquard dataset crawling detection results
[0124]
[0125]
[0126] In addition, the model of the present invention is also compared with other models in terms of image segmentation, object segmentation and detection rate. As can be seen from the table, this model is superior to the previous models. Compared with Lenz's model, which first introduced deep learning methods into the field of grasping posture detection, the model of the present invention has improved by 24.9% in image segmentation and 22.2% in object segmentation. Compared with the model proposed by Kumra, the model in this section has improved by 1.1% in image segmentation and 1.2% in object segmentation.
[0127] The present invention visualizes part of the detection results. Figure 6 The point with the highest confidence detected by the method on a given target and the grab rectangle composed of other parameters corresponding to the point are shown in FIG. It can be seen that the detection result is basically consistent with the label using the method of the present invention.
[0128] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A robotic arm grasping posture detection method based on multimodal fusion of RGB-D images. It is characterized in that The steps include: S1: Randomly reverse and rotate the cropped image to a 320*320 input image with the object to be grasped as the center, and collect the RGB image and depth image of the input image; S2: Perform sigmoid activation and average pooling operations on the RBG image and depth image respectively to generate weights corresponding to the two feature information, and then assign the weight values to the respective feature information through element-by-element multiplication operations, and then add the two features element-by-element, and finally perform CA attention operations before output; S3: After the RGB image and the depth image are fused, the dilated convolution branches with different dilation rates are used to capture the region information, angle information, and width information of the image features output in S2, and then the output of each branch is reduced in dimension and integrated; S4: The output results in S3 are passed through four convolutional layers to generate four heat maps with a resolution of 320*320, which correspond to the score map, width map, rotation angle sin component map and rotation angle cos component map respectively, and the final grasping point, grasping rectangle and grasping angle are determined based on each heat map; The weights corresponding to the two types of feature information in S2 are as follows: in, Represents the convolution operation; Represents the sigmoid function activation operation; represents the average pooling operation; , Represents two kinds of feature information, , Corresponding to , The weights of the two feature information; The CA attention operation outputs the result The details are as follows: in, represents element-wise addition; represents element-wise multiplication; represents the CA attention operation; The specific steps for capturing the regional information described in S3 are as follows: S3.1.1: Get a 1×1 convolution kernel with a dilation of 1 and a 1×1 convolution kernel with a dilation of 6 Perform BatchNorm and Relu operations respectively and then concatenate; S3.1.2: Then change the convolution kernel to the same size as the low-dimensional fusion feature by bilinear interpolation upsampling Perform splicing operations; S3.1.3: Dimensionality reduction after convolution operation and 3×3 convolution kernel with dilation 12 and a 3×3 convolution kernel with a dilation of 18 The spliced feature information is spliced; S3.1.4: Finally, the output is passed through a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU. The specific output results are as follows: in, Indicates the result of regional information crawling; Represents bilinear interpolation upsampling; Represents the convolution operation; Represents a splicing operation; The angle information grabbing steps are as follows: S3.2.1: First, obtain the 3×3 convolution kernel with a dilation of 6 and a 3×3 convolution kernel with a dilation of 12 Perform BatchNorm and Relu operations respectively and then perform concatenation operation; S3.2.2: Then change the bilinear interpolation upsampling method to the same size convolution kernel as the low-dimensional fusion feature Perform splicing operations; S3.2.3: Pass through a convolutional layer; S3.2.4: After a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU, the specific output results are as follows: in, Indicates the angle information capture result; The steps for capturing width information are as follows: S3.3.1: First, obtain the 3×3 convolution kernel with a dilation of 12 and a 3×3 convolution kernel with a dilation of 18 After performing BatchNorm and Relu operations respectively, they are concatenated with a 1×1 convolution kernel with a dilation of 1. and a 1×1 convolution kernel with a dilation of 6 The results of the splicing operation after performing BatchNorm and Relu operations are spliced to supplement the feature information; S3.3.2: After passing through a convolutional layer, a bilinear interpolation upsampling operation is performed; S3.3.3: After a three-layer convolutional layer consisting of Conv2d, BatchNorm, and ReLU, the specific output results are as follows: in, Indicates the result of fetching width information.
2. According to the method for detecting the gripping posture of a robot arm based on multimodal fusion of RGB-D images according to claim 1, It is characterized in that In S4, the grasping angle information is replaced by sine and cosine, and the grasping angle is restored by calculation. The output result is: in, represents the grasping posture predicted by the grasping detection model, , , and Respectively represent the predicted crawl The confidence score, the sin and cos components of the rotation angle, and the width of the captured prediction box; Indicates the maximum confidence score; Represents the data input from the grasp detection network to the grasp prediction module; Represents the four convolution operations that the grasp detection network processes on the input data.
3. According to the method for detecting the gripping posture of a robotic arm based on multimodal fusion of RGB-D images according to claim 2, It is characterized in that In S4, the grasping point is determined by the pixel with the maximum pixel value in the grasping component map, the width of the grasping rectangle is determined by the pixel value of the width map, and the grasping angle is determined by the pixel value of the width map. Determined by the following formula: in, and It is represented by the pixel values of the sin component and cos component heat map.
4. According to the method for detecting the gripping posture of a manipulator based on multimodal fusion of RGB-D images according to claim 3, It is characterized in that In S4, the grasp detection problem is transformed into an end-to-end regression problem, and a mapping relationship is obtained through model training. ,in, To detect the image, is the correct grasping position and posture, used to minimize the prediction With actual value The distance between Indicates the capture score range of each pixel in the image. Represents the capture width of each pixel, the loss function of the network The definition is as follows: in, Indicates the number of samples that can be captured; Represents a smooth function.
5. According to the method for detecting the gripping posture of a manipulator based on multimodal fusion of RGB-D images according to claim 4, It is characterized in that In S4, if the grasping posture predicted by the model and the target satisfy both of the following two conditions, the prediction result is judged to be correct: in, Represents the predicted grab box Angle Represents a label rectangle Angle The index represents the intersection-over-union ratio of the predicted grab box and the labeled rectangle.
