A grasping posture detection method based on multimodal fusion features

Abstract

The present invention specifically relates to a grasping posture detection method based on multimodal fusion features, including: obtaining an RGB image and a depth image of an object to be grasped; performing multi-layer feature extraction on the RGB image and the depth image respectively to obtain RGB features and depth features corresponding to multiple levels; projecting and splicing the RGB features and depth features corresponding to each level to obtain multimodal fusion features of each level; performing feature extraction on the multimodal fusion features of each level respectively to generate a corresponding feature map to be predicted; performing grasping posture prediction based on the feature map to be predicted, and calculating the corresponding grasping position information according to the result of the grasping posture prediction. The grasping posture detection method in the present invention can effectively improve the multimodal fusion features and accurately detect the grasping position based on the multimodal fusion features, thereby ensuring the accuracy and stability of grasping.

Description

Technical Field

[0001] The present invention relates to the field of robot vision technology, and in particular to a grasping posture detection method based on multimodal fusion features. Background Art

[0002] With the continuous development of computer technology, mechanical structures such as robots are constantly being used in various industries. For example, in some companies, robots are used to check the working conditions of equipment in extreme environments, repair equipment, or sort, grab and move some industrial products. Among them, the operation of autonomous grasping of objects by robotic arms is a very important task for robots in the working environment. However, there are still many problems that need to be solved in the field of autonomous grasping by robotic arms, and grasping posture detection is one of the challenging problems.

[0003] Due to the influence of factors such as the shape and mass of the object, it is difficult for the robot arm to accurately determine the grasping position when grasping the object, resulting in the phenomenon that the object cannot be grasped or the object falls during the movement after grasping the object, making the efficiency of the robot arm's successful grasping low. For this reason, the Chinese patent with publication number CN108044627B discloses "A grasping position detection method, device and robot arm", and the method includes: the robot arm obtains the first image information and the second image information of the target object at the first position and the second position respectively, and determines the three-dimensional geometric figure of the shape of the target object; if the three-dimensional geometric figure is a regular geometric figure, the grasping position of the target object is determined according to its geometric center; otherwise, at least two feature points of the irregular geometric figure are obtained, and the grasping position of the target object is determined according to the feature points.

[0004] The grasping posture detection method in the above-mentioned existing scheme uses different methods to determine the grasping position for target objects of different shapes, so as to find a suitable grasping position according to the characteristics of the target object itself. However, how to quickly detect the grasping position is itself a difficult problem. Most of the existing methods detect the grasping position by extracting the RGB features of the object, but the information reflected by the RGB features of the object is very limited and is not enough to ensure the accuracy of the grasping. To this end, the applicant thinks of fusing the depth features on the basis of the RGB features to generate multimodal fusion features, and then detects the grasping position based on the multimodal fusion features. The existing method usually sets two feature extraction networks of the same structure to extract RGB features and depth features respectively, and then combines the outputs of the last layer of the two networks to obtain the final multimodal fusion features. However, the feature abstraction level extracted by the existing method is relatively single, and some important information is easily lost during the processing process, resulting in poor generation of multimodal fusion features, which in turn affects the accuracy and stability of the robot arm grasping.

[0005] Therefore, how to design a grasping posture detection method that can effectively improve the multimodal fusion features and accurately detect the grasping position based on the multimodal fusion features is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a grasping posture detection method that can effectively enhance the multimodal fusion features and accurately detect the grasping position based on the multimodal fusion features, thereby ensuring the accuracy and stability of grasping.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A grasping posture detection method based on multimodal fusion features comprises the following steps:

[0009] S1: Obtain the RGB image and depth image of the object to be grasped;

[0010] S2: Perform multi-layer feature extraction on the RGB image and the depth image respectively to obtain RGB features and depth features corresponding to multiple levels;

[0011] S3: Project and concatenate the RGB features and depth features corresponding to each level to obtain multimodal fusion features of each level;

[0012] S4: extracting features from the multimodal fusion features at each level respectively, and generating corresponding feature maps to be predicted;

[0013] S5: Predict the grasping posture based on the feature map to be predicted, and calculate the corresponding grasping position information according to the result of the grasping posture prediction.

[0014] Preferably, in step S2, the RGB image and the depth image are firstly input into a convolutional neural network having the same structure, and then the RGB features and the depth features of the corresponding levels are respectively extracted through each layer of the convolutional neural network.

[0015] Preferably, in step S3, the RGB features and depth features corresponding to each level are projected and concatenated through two convolutional layers and one maximum pooling layer; wherein the multimodal fusion feature is expressed by the following formula:

[0016]

[0017]

[0018] In the above formula: represents the RGB features extracted from the i-th layer of the convolutional neural network; represents the deep features extracted from the i-th layer of the convolutional neural network; Gi () represents a set of nonlinear operations, namely, projection processing; Represents the RGB features after projection processing; represents the depth feature after projection processing; p i Represents the multimodal fusion features corresponding to the RGB features and depth features extracted by the i-th layer of the convolutional neural network.

[0019] Preferably, in step S5, the result of the grasping posture prediction includes the coordinates of the center point of the grasping box prediction Grab box prediction offset vector Grab box prediction size and And the predicted rotation angle of the grab box

[0020] Preferably, the grab position information (x, y, w, h, θ) includes the grab box center point coordinates (x, y), the grab box sizes w and h, and the grab box rotation angle θ;

[0021] in:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] In the above formula: Indicates that Activation function constrained to the range (0,1).

[0028] Preferably, the grasping pose prediction is to find an optimal grasping box: firstly, a center point heat map is generated based on the feature map to be predicted; then, based on the peak point of the center point heat map Determine the center point of the maximum grab box, that is, predict the center point coordinates Finally, the predicted offset vector used to correct the center point is output Predicted size and And the predicted rotation angle

[0029] Preferably, when predicting the coordinates of the center point of the grab box: first, generate a corresponding center point heat map based on the feature map to be predicted; then use the pixel points in the center point heat map whose values ​​are greater than a preset threshold as candidate grab box prediction center points; finally, use the coordinates of the candidate grab box prediction center point with the largest value as the coordinates of the grab box prediction center point.

[0030] Preferably, in steps S4 and S5, a feature map to be predicted is generated and a grasping pose prediction is performed by a grasping pose detection model; the grasping pose detection model includes a feature map extraction model for extracting features of multimodal fusion features at each level, and three prediction networks for respectively predicting the coordinates of the predicted center point of the grasping box, the predicted offset vector of the grasping box, the predicted size of the grasping box, and the predicted rotation angle;

[0031] During training, the feature map extraction model is first pre-trained, and then the grasping posture detection model is end-to-end trained.

[0032] Preferably, the total loss function of the grasping posture detection model is expressed as:

[0033] L=λ1L1+λ w L w +λ h L h +λ θ L θ ;

[0034] in:

[0035]

[0036]

[0037]

[0038]

[0039] In the above formula: L represents the total loss function; L1 represents the loss function of the prediction center point coordinate network; L w , L h , L θ Respectively represent the loss functions of the network for predicting the length, width, and rotation angle of the grab box; λ1, λ w , h , θ They represent the proportional coefficients of the loss function in the total loss function, and their values ​​are 0.1, 0.2, 0.2, and 1 respectively; represents the center point heat map; Y represents the preprocessed label; N represents the number of predicted center points of the candidate grab box; α and β represent hyperparameters; Indicates the predicted size of the grab box; represents the predicted rotation angle of the grab box; θ represents the rotation angle of the grab box; w and h represent the size of the grab box.

[0040] Preferably, when calculating the loss, the labels of the grab boxes are preprocessed by the following steps:

[0041] For the real center point p of the center point heat map, downsample it to get the corresponding key point

[0042] For each pixel of the center point heat map, the Gaussian kernel Mapping to labels superior;

[0043] Among them, σ p represents the standard deviation, σ p The value is radius;

[0044] In the above formula: the key point after the real center point p is downsampled Expressed as R represents the downsampling factor, which is 4; x and y represent the coordinates of the center point of the grab box; w and h represent the size of the grab box; λ represents the IOU threshold, which is 0.25.

[0045] Compared with the prior art, the grasping posture detection method in the present invention has the following beneficial effects:

[0046] The present invention combines RGB features and depth features at different abstraction levels by means of multi-level feature extraction and fusion, thereby effectively generating compact and highly distinguishable multimodal fusion features, which enables important information to be retained, thereby improving the detection accuracy of the grasping posture.

[0047] The present invention generates multimodal fusion features of each level by feature projection and splicing, so as to eliminate the problem that the elements of the feature vectors extracted in different hidden layers of the same network cannot correspond one to one, and then project the features from different abstract levels into the same common space, thereby ensuring the generation effect of multimodal fusion features.

[0048] The present invention generates a feature map to be predicted based on multi-level multimodal fusion features, and then realizes grasping posture prediction and grasping position information calculation through the feature map to be predicted, so that grasping posture detection can be effectively realized based on multimodal fusion features to accurately determine the grasping position, thereby effectively ensuring the accuracy and stability of grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0050] Figure 1 It is the logic block diagram of the grasping posture detection method;

[0051] Figure 2 This is the network architecture diagram of the grasping posture detection model;

[0052] Figure 3 Schematic diagram of some results of grasping posture detection in the experiment. DETAILED DESCRIPTION

[0053] The following is a further detailed description through specific implementation methods:

[0054] Example:

[0055] This embodiment discloses a grasping posture detection method based on multimodal fusion features.

[0056] like Figure 1 As shown, the grasping posture detection method based on multimodal fusion features includes the following steps:

[0057] S1: Obtain the RGB image and depth image of the object to be grasped;

[0058] S2: Perform multi-layer feature extraction on the RGB image and the depth image respectively to obtain RGB features and depth features corresponding to multiple levels;

[0059] S3: Project and concatenate the RGB features and depth features corresponding to each level to obtain multimodal fusion features of each level;

[0060] S4: extracting features from the multimodal fusion features at each level respectively, and generating corresponding feature maps to be predicted;

[0061] S5: Predict the grasping posture based on the feature map to be predicted, and calculate the corresponding grasping position information according to the result of the grasping posture prediction.

[0062] In the present invention, RGB features and depth features at different abstract levels are combined by multi-level feature extraction and fusion, thereby effectively generating compact and highly differentiated multimodal fusion features, which can retain important information, thereby improving the detection accuracy of the grasping posture. At the same time, the present invention generates multimodal fusion features of each level by feature projection and splicing, so that the problem that the elements of the feature vectors extracted from different hidden layers of the same network cannot correspond one to one can be eliminated, and then the features from different abstract levels are projected into the same public space, thereby ensuring the generation effect of the multimodal fusion features. In addition, the present invention generates a feature map to be predicted based on multi-level multimodal fusion features, and then realizes the grasping posture prediction and grasping position information calculation through the feature map to be predicted, so that the grasping posture detection can be effectively realized based on the multimodal fusion features to accurately determine the grasping position, thereby effectively ensuring the accuracy and stability of the grasping.

[0063] In the specific implementation process, the RGB image and the depth image are first input into the convolutional neural network with the same structure, and then the RGB features and depth features of the corresponding levels are extracted through each layer of the convolutional neural network. Considering the computational efficiency and memory, this embodiment selects the residual network ResNet-18 with fewer parameters as the multi-level feature extraction network. ResNet-18 has a total of 18 convolutional layers, which are deployed in 5 residual blocks. The corresponding RGB features and depth features are extracted from the two layers of each residual block and sent to the set projection block (for projection and splicing).

[0064] In the present invention, RGB features and depth features at different abstraction levels are combined by multi-level feature extraction and fusion, thereby effectively generating compact and highly differentiated multimodal fusion features, so that important information can be retained, thereby improving the detection accuracy of the grasping posture.

[0065] During the specific implementation process, the RGB features and depth features corresponding to each level are projected and spliced ​​through the set projection blocks; since the elements of the feature vectors extracted from different hidden layers of the same network are difficult to correspond one to one, the present invention sets a projection block to eliminate this obstacle and project the features from different abstract levels into the same common space.

[0066] The projection block consists of two convolutional layers and a maximum pooling layer. The first convolutional layer is a 7×7 filter, batch normalization, and ReLU nonlinear function, focusing on the spatial dimensions, height and width; the second convolutional layer is a 1×1 filter, batch normalization, and ReLU nonlinear function, focusing on depth; the maximum pooling layer is used to calculate the maximum value of each depth slice (plane at the same depth).

[0067] Among them, the multimodal fusion feature is expressed by the following formula:

[0068]

[0069]

[0070] In the above formula: represents the RGB features extracted from the i-th layer of the convolutional neural network; represents the deep features extracted from the i-th layer of the convolutional neural network; G i () represents a set of nonlinear operations, namely, projection processing; Represents the RGB features after projection processing; represents the depth feature after projection processing; p i Represents the multimodal fusion features corresponding to the RGB features and depth features extracted by the i-th layer of the convolutional neural network.

[0071] In the present invention, based on the above-mentioned projection block, multimodal fusion features of each level can be generated by feature projection and splicing, so that the problem that the elements of the feature vectors extracted in different hidden layers of the same network are difficult to correspond one to one can be eliminated, and then the features from different abstract levels can be projected into the same common space, thereby ensuring the generation effect of multimodal fusion features.

[0072] In the specific implementation process, the results of the grasp pose prediction include the coordinates of the center point of the grasp box prediction Grab box prediction offset vector Grab box prediction size and And the predicted rotation angle of the grab box Grasp pose prediction is to find an optimal grasping box: first generate a center point heat map based on the feature map to be predicted; then generate a center point heat map based on the peak point of the center point heat map Determine the center point of the maximum grab box, that is, predict the center point coordinates Finally, the predicted offset vector used to correct the center point is output Predicted size and And the predicted rotation angle

[0073] The grasp pose prediction stage is divided into three parts, and three different two-layer convolutional neural networks are used for prediction. The first part is the center point of the grasp box prediction. First, a 128×128 center point heat map is generated based on the feature map to be predicted. The value of each pixel represents the weight of the point, and the weight range is [0,1]; then the pixels in the center point heat map with a value greater than the preset threshold are used as candidate grasp box prediction center points; finally, the coordinates of the candidate grasp box prediction center point with the largest value are used as the coordinates of the grasp box prediction center point. The second part is the grasp box prediction offset vector, which represents the offset of the candidate grasp box prediction center point in the x and y directions, and outputs a vector of size 128×128×2. The third part is the predicted (length and width) size and predicted rotation angle of the grasp box centered on each candidate grasp box prediction center point, and outputs a vector of size 128×128×3.

[0074] Specifically, the grab position information (x, y, w, h, θ) includes the grab box center point coordinates (x, y), the grab box sizes w and h, and the grab box rotation angle θ;

[0075] in:

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] In the above formula: Indicates that Activation function constrained to the range (0,1).

[0082] In the present invention, the above steps are used to well generate a feature map to be predicted based on multi-level multimodal fusion features, and then the grasping posture prediction and grasping position information calculation are realized through the feature map to be predicted, so that the grasping posture detection can be effectively realized based on the multimodal fusion features to accurately determine the grasping position, thereby effectively ensuring the accuracy and stability of grasping.

[0083] In the specific implementation process, Figure 2As shown, a grasping posture detection model is used to generate a feature map to be predicted and to perform grasping posture prediction; the grasping posture detection model includes a feature map extraction model for extracting features of multimodal fusion features at each level, and three prediction networks for respectively predicting the coordinates of the predicted center point of the grasping box, the predicted offset vector of the grasping box, the predicted size of the grasping box, and the predicted rotation angle; during training, the feature map extraction model is first pre-trained using an existing training method, and then the grasping posture detection model is end-to-end trained using an existing training method.

[0084] Specifically, the feature map extraction model uses the Hourglass-104 with stronger feature extraction capability as the backbone network for extracting the feature map to be predicted, inputs the multi-level multimodal fusion features into the feature map extraction model, and obtains the feature map to be predicted with a size of 128×128×256 after feature extraction.

[0085] The total loss function of the grasping posture detection model is expressed as:

[0086] L=λ1L1+λ w L w +λ h L h +λ θ L θ ;

[0087] in:

[0088]

[0089]

[0090]

[0091]

[0092] In the above formula: L represents the total loss function; L1 represents the loss function of the prediction center point coordinate network; L w , L h , L θ Respectively represent the loss functions of the network for predicting the length, width, and rotation angle of the grab box; λ1, λ w , h , θ They represent the proportional coefficients of the loss function in the total loss function, and their values ​​are 0.1, 0.2, 0.2, and 1 respectively; represents the center point heat map; Y represents the preprocessed label; N represents the number of predicted center points of the candidate grab box; α and β represent hyperparameters; Indicates the predicted size of the grab box; represents the predicted rotation angle of the grab box; θ represents the rotation angle of the grab box; w and h represent the size of the grab box.

[0093] When calculating the loss, the labels of the grab boxes are preprocessed by the following steps:

[0094] For the real center point p of the center point heat map, downsample it to get the corresponding key point

[0095] For each pixel of the center point heat map, the Gaussian kernel Mapping to labels superior;

[0096] Among them, σ p represents the standard deviation, σ p The value is radius;

[0097] In the above formula: the key point after the real center point p is downsampled Expressed as R represents the downsampling factor, which is 4; x and y represent the coordinates of the center point of the grab box; w and h represent the size of the grab box; λ represents the IOU threshold, which is 0.25.

[0098] In the present invention, the optimal grasping posture detection model can be trained through the above steps, and then the feature map to be predicted can be effectively generated based on the multi-level multimodal fusion features, and then the grasping posture prediction and grasping position information calculation can be realized through the feature map to be predicted, so that the grasping posture detection can be effectively realized based on the multimodal fusion features to accurately determine the grasping position, thereby effectively ensuring the accuracy and stability of grasping.

[0099] In order to better illustrate the advantages of the grasping posture detection method of the present invention, this embodiment discloses the following experiment.

[0100] This experiment uses a five-fold cross-validation method to evaluate the effect of the grasping detection method of the present invention. There are two different ways to divide the training set and the test set, namely image segmentation and object segmentation. Image segmentation refers to randomly dividing all images in the data set into five folds, and the image ratio of the training set and the test set is 4:1, which helps to evaluate the performance of the model for object detection in different positions and postures. Object segmentation refers to randomly dividing all object instances in the data set into five folds, and the object ratio of the training set and the test set is 4:1, which helps to evaluate the performance of the model for detecting unseen objects.

[0101] The results are evaluated according to the rectangle metric on the Cornell grasping dataset: if the predicted grasp box G and any positive label G' simultaneously meet the following two conditions, then the grasp box is considered to be a correct grasp location.

[0102] 1)|G θ -G' θ |<30°

[0103] 2)

[0104] Among them, |G θ -G' θ |<30° means the angle between the predicted grab box and the positive label is less than 30°; condition The Jaccard similarity coefficient between the predicted grab box and the label is greater than 25%.

[0105] Table 1 shows the grasping detection results of the method of the present invention in image segmentation and object segmentation, and its comparison with some previous algorithms.

[0106] Table 1

[0107]

[0108] Among them, method 1 comes from Lenz I, Lee H, Saxena A. "Deep learning for detecting robotic grasps".

[0109] Method 2 comes from Redmon J, Angelova A. "Real-time grasp detection using convolutional neural networks".

[0110] Method 3 comes from Chen L, Huang PF, Meng Z J. "Convolutional Multi-Grasp Detection using Grasp Path for RGBD'. Robotics and Autonomous Systems".

[0111] Method 4 comes from D.Guo, T.Kong, F.Sun and H.Liu, "Object discovery and grasp detection with a shared convolutional neural network".

[0112] Method 5 comes from S.Kumra and C.Kanan. "Robotic grasp detection using deep convolutional neural networks".

[0113] Method 6 comes from Z.Luo, B.Tang, S.Jiang, M.Pang, K.Xiang. "Grasp Detection Based on Faster Region CNN".

[0114] Combined with Table 1, we can see that:

[0115] This experiment gives the detection results with and without multimodal fusion features in the network architecture. If the multimodal fusion feature method is not added, the prediction method using only the Hourglass-104 network for feature extraction and key point estimation has an accuracy of 88.2% and 86.3% in image segmentation and object segmentation respectively. After adding the multimodal fusion feature extraction method, the results are 92.0% and 92.8%, which are 3.8% and 6.5% higher respectively.

[0116] Methods 1 and 3 both use anchor-free methods. It can be seen that the accuracy of the method of the present invention is improved by 18.1% and 17.2% respectively compared with the model in method 1; it is improved by 5.6% and 8.1% compared with the model in method 3. Methods 2 and 4 use multimodal fusion feature fusion methods. It can be seen that the method of the present invention is improved by 4% and 5.7% compared with the model in method 2; it is 3.7% higher in object segmentation accuracy than the model in method 4. Methods 5 and 6 use anchor-based methods. The accuracy of the method of the present invention is improved by 2.8% and 3.8% respectively compared with method 5; it is 0.3% higher in object segmentation accuracy than method 6.

[0117] In addition, if Figure 3 As shown in Figure 2, this experiment visualizes some of the results of grasping posture detection, namely the corresponding dataset annotations, where: Figure 3 (a) The dataset label for a given target, Figure 3 (b) The grasping rectangle composed of the point with the highest confidence detected by this method on a given target and other parameters corresponding to the point. Figure 3 It can be seen that, by using the method of the present invention, the grasping posture detection result is basically consistent with the label.

[0118] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims. At the same time, common sense such as the well-known specific structures and characteristics in the embodiments are not described in detail here. Finally, the scope of protection claimed by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A grasping posture detection method based on multimodal fusion features, characterized in that: The following steps are involved: S1: Obtain the RGB image and depth image of the object to be grasped; S2: Perform multi-layer feature extraction on the RGB image and the depth image respectively to obtain RGB features and depth features corresponding to multiple levels; S3: Project and concatenate the RGB features and depth features corresponding to each level to obtain multimodal fusion features of each level; S4: extracting features from the multimodal fusion features at each level respectively, and generating corresponding feature maps to be predicted; S5: Predict the grasping posture based on the feature map to be predicted, and calculate the corresponding grasping position information according to the result of the grasping posture prediction.

2. The grasping posture detection method based on multimodal fusion features as claimed in claim 1, characterized in that: In step S2, the RGB image and the depth image are firstly input into a convolutional neural network with the same structure, and then the RGB features and the depth features of the corresponding levels are extracted through each layer of the convolutional neural network.

3. The grasping posture detection method based on multimodal fusion features as described in claim 2 is characterized in that: In step S3, the RGB features and depth features corresponding to each level are projected and concatenated through two convolutional layers and one maximum pooling layer; wherein the multimodal fusion feature is expressed by the following formula: In the above formula: f i rgb represents the RGB features extracted from the i-th layer of the convolutional neural network; f i depth represents the deep features extracted from the i-th layer of the convolutional neural network; G i () represents a set of nonlinear operations, namely, projection processing; Represents the RGB features after projection processing; represents the depth feature after projection processing; p i Represents the multimodal fusion features corresponding to the RGB features and depth features extracted by the i-th layer of the convolutional neural network.

4. The grasping posture detection method based on multimodal fusion features as claimed in claim 1, characterized in that: In step S5, the result of the grasping pose prediction includes the coordinates of the center point of the grasping box prediction Grab box prediction offset vector Grab box prediction size and And the predicted rotation angle of the grab box 5. The grasping posture detection method based on multimodal fusion features as claimed in claim 4, characterized in that: The grab position information (x, y, w, h, θ) includes the grab box center coordinates (x, y), the grab box sizes w and h, and the grab box rotation angle θ; in: In the above formula: Indicates that Activation function constrained to the range (0,1).

6. The grasping posture detection method based on multimodal fusion features as claimed in claim 4, characterized in that: Grasp pose prediction is to find an optimal grasping box: first generate a center point heat map based on the feature map to be predicted; then generate a center point heat map based on the peak point of the center point heat map Determine the center point of the maximum grab box, that is, predict the center point coordinates Finally, the predicted offset vector used to correct the center point is output Predicted size and And the predicted rotation angle 7. The grasping posture detection method based on multimodal fusion features as claimed in claim 6, characterized in that: When predicting the coordinates of the center point of the grab box: first generate the corresponding center point heat map based on the feature map to be predicted; then use the pixel points in the center point heat map whose values ​​are greater than the preset threshold as candidate grab box prediction center points; finally, use the coordinates of the candidate grab box prediction center point with the largest value as the coordinates of the grab box prediction center point.

8. The grasping posture detection method based on multimodal fusion features as claimed in claim 7, characterized in that: In steps S4 and S5, a feature map to be predicted is generated and a grasping pose prediction is performed by using a grasping pose detection model; the grasping pose detection model includes a feature map extraction model for extracting features of multimodal fusion features at each level, and three prediction networks for respectively predicting the coordinates of the grasping box prediction center point, the grasping box prediction offset vector, the grasping box prediction size, and the predicted rotation angle; During training, the feature map extraction model is first pre-trained, and then the grasping posture detection model is end-to-end trained.

9. The grasping posture detection method based on multimodal fusion features as claimed in claim 8, characterized in that: The total loss function of the grasping posture detection model is expressed as: L=λ1L1+λ w L w +λ h L h +λ θ L θ ; in: In the above formula: L represents the total loss function; L1 represents the loss function of the prediction center point coordinate network; L w , L h , L θ Respectively represent the loss functions of the network for predicting the length, width, and rotation angle of the grab box; λ1, λ w , h , θ They represent the proportional coefficients of the loss function in the total loss function, and their values ​​are 0.1, 0.2, 0.2, and 1 respectively; represents the center point heat map; Y represents the preprocessed label; N represents the number of predicted center points of the candidate grab box; α and β both represent hyperparameters; Indicates the predicted size of the grab box; represents the predicted rotation angle of the grab box; θ represents the rotation angle of the grab box; w and h represent the size of the grab box.

10. The grasping posture detection method based on multimodal fusion features according to claim 9, characterized in that: When calculating the loss, the labels of the grab boxes are preprocessed by the following steps: For the real center point p of the center point heat map, downsample it to get the corresponding key point For each pixel of the center point heat map, the Gaussian kernel Mapping to labels superior; Among them, σ p represents the standard deviation, σ p The value is radius; In the above formula: the key point after the real center point p is downsampled Expressed as R represents the downsampling factor, which is 4; x and y represent the coordinates of the center point of the grab box; w and h represent the size of the grab box; λ represents the IOU threshold, which is 0.25.

Images