Method and device for predicting gripping angle and width of robotic arm based on multiple features

Abstract

The invention discloses a method and device for predicting a grasping angle and width of a robot arm based on multiple features, and relates to the field of grasping of a robot arm. The method and device include: constructing a grasping angle and width prediction model of a robot arm and training it to obtain a trained grasping angle and width prediction model of the robot arm, acquiring an RGB-D image of an object to be grasped and inputting it into the trained grasping angle and width prediction model of the robot arm, inputting the RGB-D image into an initial feature extraction module, sequentially performing feature extraction through two depthwise separable convolution units, and performing downsampling through a maximum pooling layer to obtain initial features, and inputting the initial features into a feature enhancement module, firstly performing multi-scale features through two multi-scale feature extraction modules to extract multi-scale features, and performing fusion and enhancement of multi-scale features through a bidirectional feature pyramid network to obtain enhanced features, and performing feature decoding module to predict the grasping angle, grasping width and grasping quality score of the enhanced features, so as to solve the problems of low grasping accuracy and success rate.

Description

Technical Field

[0001] The present invention relates to the field of robot arm grasping, and in particular to a method and device for predicting a robot arm grasping angle and width based on multiple features. Background Art

[0002] With the rapid development of science and technology, robotics has made breakthrough progress in various fields. Among them, robotic grasping, as a key technology for robots to physically interact with the external environment, has been widely used in many scenarios such as industrial manufacturing, warehousing and logistics, and home services. However, in practical applications, robotic grasping technology still faces many challenges. The first challenge comes from the diversity of the grasped objects themselves. Different objects have significant differences in shape, size, material, etc., which requires the system to be able to adopt corresponding grasping strategies for objects with different characteristics. Secondly, the complex and changeable operating environment, such as changes in lighting conditions and background confusion, further increases the difficulty of target detection. In addition, the accuracy deviation of grasping prediction often directly affects the accuracy of subsequent grasping actions. These problems together make it difficult for grasping detection to achieve the ideal accuracy, thereby reducing the success rate of the overall grasping task.

[0003] In the prior art, traditional robotic grasping methods mainly rely on the analysis of geometric features of objects, physical modeling and mechanical calculations. However, these methods are often difficult to adapt to and meet actual needs when facing unstructured complex environments. In recent years, with the vigorous development of deep learning and computer vision technology, researchers have begun to introduce these advanced technologies into the field of robotic grasping. Deep learning-based grasping methods have quickly become the mainstream technical route in the field of robot grasping due to their advantages such as strong automatic feature extraction capabilities, no need for manual feature matching, and good system robustness. Despite this, it is still difficult for existing grasping detection algorithms to achieve an ideal balance between accuracy and processing speed. Although some algorithms perform well in detection accuracy, their calculation time is long and it is difficult to meet the needs of real-time operation of robots; other algorithms, despite their fast processing speed, lack accuracy and cannot guarantee accurate grasping of target objects. Summary of the invention

[0004] The purpose of this application is to propose a method and device for predicting the gripping angle and width of a robotic arm based on multiple features in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for predicting the gripping angle and width of a robotic arm based on multiple features, comprising the following steps:

[0006] A robot arm grasping angle and width prediction model is constructed and trained to obtain a trained robot arm grasping angle and width prediction model, wherein the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module, and a feature decoding module connected in sequence; the initial feature extraction module includes a first depthwise separable convolution unit, a second depthwise separable convolution unit, and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence;

[0007] The RGB-D image of the object to be grasped is obtained and input into the trained robot arm grasping angle and width prediction model. The RGB-D image is input into the initial feature extraction module, and is successively subjected to the first depthwise separable convolution unit and the second depthwise separable convolution unit for feature extraction to obtain the basic features. The basic features are downsampled through the maximum pooling layer to obtain the initial features. The initial features are input into the feature enhancement module, and are first subjected to two multi-scale feature extraction modules to extract multi-scale features, and the multi-scale features are fused and enhanced through the bidirectional feature pyramid network to obtain the enhanced features. The enhanced features are subjected to the feature decoding module to predict the grasping angle, grasping width and grasping quality score, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robot arm.

[0008] Preferably, the first depthwise separable convolution unit comprises a depthwise separable convolution layer with a convolution kernel size of 9×9, a batch normalization layer, and a Mish activation function layer connected in sequence; the second depthwise separable convolution unit comprises a depthwise separable convolution layer with a convolution kernel size of 5×5, a batch normalization layer, and a Mish activation function layer connected in sequence, and the step size of the maximum pooling layer is 2;

[0009] The input features of the depthwise separable convolutional layer are input into the depthwise separable convolutional layer, and depthwise convolution is performed first, as shown in the following formula:

[0010] ;

[0011] in, Indicates The input features of the depth-wise separable convolutional layer are channels, represents the corresponding depth convolution kernel, Indicates The output features of the depthwise convolution of channels, Represents the convolution operation;

[0012] Then perform point-by-point convolution, as shown below:

[0013] ;

[0014] in, represents the corresponding point-by-point convolution kernel, Indicates the number of channels, Indicates channels, Represents the output features of the depthwise separable convolutional layer.

[0015] Preferably, the hybrid attention module includes a channel attention module and a spatial attention module connected in sequence, and the input features of the hybrid attention module pass through the channel attention module to obtain the channel attention weight, as shown in the following formula:

[0016] ;

[0017] ;

[0018] ;

[0019] in, and represents the height and width of the input features of the hybrid attention module, Represents the input feature of the hybrid attention module at position The eigenvalue at and Respectively represent the average pooling results and maximum pooling results of the channel dimension, Indicates the maximum value operation. represents a multi-layer perceptron, represents the sigmoid activation function, represents the channel attention weight;

[0020] The channel attention weight is multiplied by the input feature of the hybrid attention module to obtain the channel feature. The channel feature passes through the spatial attention module to obtain the spatial attention weight, as shown in the following formula:

[0021] ;

[0022] ;

[0023] ;

[0024] in, Indicates The channel characteristics of the channels, and Respectively represent the average pooling results and maximum pooling results of the spatial dimension, represents the convolution operation, represents the concatenation of channel dimensions, represents the spatial attention weight;

[0025] The spatial attention weights are multiplied by the channel features to obtain the output features of the hybrid attention module.

[0026] Preferably, the atrous convolution module includes three convolution layers with different dilation rates and a convolution kernel size of 3×3. The input features of the atrous convolution module are respectively input into three convolution layers with different dilation rates and a convolution kernel size of 3×3 in parallel to perform three convolution operations to obtain multi-scale features, as shown in the following formula:

[0027] ;

[0028] ;

[0029] ;

[0030] in, represents the input features of the dilated convolution module, , , Represent the convolution kernels with expansion rates of 1, 2, and 4 respectively. represents the convolution operation, , , They represent the output features of the input features of the dilated convolution module after the convolution kernels with dilation rates of 1, 2, and 4, respectively, and constitute multi-scale features.

[0031] Preferably, the bidirectional feature pyramid network includes a top-down path and a bottom-up path; , , As the original input features of the first level in the top-down path and the bottom-up path respectively , the original input features of the second level And the original input features of the third level ;

[0032] In the top-down path, the original input features of the third level are As the output features of the third level in the top-down pathway ; The calculation process of the second level and the first level in the top-down path is as follows:

[0033] ;

[0034] In the bottom-up path, the output features of the first level in the top-down path are combined with the input features of the first level Add together to get the output features of the first level in the bottom-up path. The calculation process of the second and third levels in the bottom-up path is as follows:

[0035] ;

[0036] in, Indicates the first The output features of the level, Indicates the first The output features of the level, Indicates The original input features of the level, is 1, 2 or 3, Conv represents the convolution operation, represents the upsampling operation, Represents a downsampling operation, which converts the output features of the third level in the bottom-up path As an enhanced feature.

[0037] Preferably, the feature decoding module includes a first convolutional layer, a progressive upsampling module and three output branches connected in sequence, the progressive upsampling module includes a three-stage upsampling module, the upsampling module includes a second convolutional layer and a bilinear upsampling layer connected in sequence, and the three output branches include a quality score branch; the enhanced feature is first subjected to feature dimensionality reduction by a first convolutional layer with a convolution kernel size of 1×1 to obtain a reduced dimensionality feature, as shown in the following formula:

[0038] ;

[0039] in, Represents enhanced features, represents the first convolution layer with a convolution kernel size of 1×1, represents the dimension reduction feature, Represents the convolution operation;

[0040] The reduced dimension features are input to the progressive upsampling module to perform three-stage upsampling operations, including:

[0041] Phase 1, Execution Convolution and bilinear upsampling:

[0042] ;

[0043] In the second stage, 3×3 convolution and bilinear upsampling are performed:

[0044] ;

[0045] In the third stage, 3×3 convolution and bilinear upsampling are performed:

[0046] ;

[0047] in, , , Represent the features after upsampling in three stages respectively. , , Represent the convolution kernels of size 3×3 in three stages respectively. represents a bilinear upsampling operation, Indicates the upsampling scaling factor;

[0048] The features after upsampling in the third stage are input into the three output branches for the following calculation process:

[0049] Crawl quality scoring branch:

[0050] ;

[0051] Grab the angular branch:

[0052] ;

[0053] Grab the width branch:

[0054] ;

[0055] in, , , Represent the convolutional layers of the three output branches respectively, represents the features after upsampling in the third stage, represents the sigmoid activation function, represents the linear rectification function, , , They represent the crawl quality score, crawl angle, and crawl width respectively.

[0056] In a second aspect, the present invention provides a robot arm grasping angle and width prediction device based on multiple features, comprising:

[0057] A model building module is configured to build and train a robot arm grasping angle and width prediction model to obtain a trained robot arm grasping angle and width prediction model, wherein the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module, and a feature decoding module connected in sequence; the initial feature extraction module includes a first depthwise separable convolution unit, a second depthwise separable convolution unit, and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence;

[0058] The prediction module is configured to obtain an RGB-D image of the object to be grasped and input it into a trained prediction model of the grasping angle and width of the robotic arm. The RGB-D image is input into the initial feature extraction module, and is sequentially subjected to feature extraction by the first depthwise separable convolution unit and the second depthwise separable convolution unit to obtain basic features. The basic features are downsampled through the maximum pooling layer to obtain initial features. The initial features are input into the feature enhancement module, and multi-scale features are first extracted through two multi-scale feature extraction modules, and multi-scale features are fused and enhanced through a bidirectional feature pyramid network to obtain enhanced features. The enhanced features are subjected to the feature decoding module to predict the grasping angle, grasping width and grasping quality score, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robotic arm.

[0059] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0060] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0061] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The multi-feature based robotic arm grasping angle and width prediction method proposed in this paper achieves accurate capture of the multi-dimensional features of the object to be grasped by applying deep feature extraction of RGB-D images and a hybrid attention mechanism, thereby improving the grasping accuracy and robustness of the robotic arm in complex scenes; through the combined use of depthwise separable convolution and dilated convolution, as well as the design of a bidirectional feature pyramid network, it can efficiently extract multi-scale feature information from the input image, while significantly reducing the computational complexity and improving the real-time processing capability of the system.

[0064] (2) The multi-feature based robotic arm grasping angle and width prediction method proposed in the present invention adopts a progressive upsampling module and a multi-scale convolution combination to achieve accurate reconstruction of the feature map, enhance the model's adaptability to objects of different sizes and shapes, and improve the generalization performance of the system in practical application scenarios.

[0065] (3) The multi-feature based robot arm grasping angle and width prediction method proposed in the present invention adopts a three-branch parallel output structure to simultaneously predict the grasping quality score, grasping angle and grasping width, so that the robot arm can comprehensively evaluate the feasibility of the grasping plan and avoid performing unreliable grasping actions, thereby improving the overall operating efficiency and safety of the system. It can significantly improve the accuracy and reasoning speed of the robot arm's grasping detection, improve operational flexibility, enhance the ability to adapt to different grasping tasks, and reduce the operational error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A schematic diagram of a flow chart of a method for predicting a gripping angle and width of a robotic arm based on multiple features according to an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a robot arm grasping angle and width prediction model of a robot arm grasping angle and width prediction method based on multiple features according to an embodiment of the present application;

[0069] Figure 3 A schematic diagram of an initial feature extraction module of a method for predicting a gripping angle and width of a robotic arm based on multiple features according to an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a feature enhancement module of a method for predicting a gripping angle and width of a robotic arm based on multiple features according to an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a feature decoding module of a method for predicting a gripping angle and width of a robotic arm based on multiple features according to an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a robot arm grasping angle and width prediction device based on multiple features according to an embodiment of the present application;

[0073] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Figure 1 A method for predicting the gripping angle and width of a robotic arm based on multiple features is shown in an embodiment of the present application, comprising the following steps:

[0076] S1, construct and train a robot arm grasping angle and width prediction model to obtain a trained robot arm grasping angle and width prediction model, the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module and a feature decoding module connected in sequence; the initial feature extraction module includes a first depth-separable convolution unit, a second depth-separable convolution unit and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence.

[0077] Specifically, first, collect the RGB-D image data of the object as the robot grasping dataset, which includes RGB-D images of objects at multiple angles and grasping labels. Secondly, build a prediction model for the grasping angle and width of the robot arm, referring to Figure 2 The robot grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module and a feature decoding module connected in sequence; the initial feature extraction module contains two depth-separable convolution units and a maximum pooling layer, which are used to extract the initial features of the RGB-D image; the feature enhancement module is composed of multiple groups of alternating hybrid attention modules and hole convolution modules and a bidirectional feature pyramid network, the hybrid attention module contains a channel attention module and a spatial attention module, the hole convolution module contains convolution layers with different expansion rates, which are used to extract multi-scale features, and at the end of the feature enhancement module, the multi-scale features are fused and enhanced through the bidirectional feature pyramid network to obtain enhanced features; the feature decoding module first performs dimensionality reduction processing through the first convolution layer with a convolution kernel size of 1×1, and then adopts three-stage progressive upsampling operations, each stage contains a second convolution layer with a convolution kernel size of 3×3 and a bilinear upsampling layer, which are used to restore the spatial resolution of the features, and finally outputs two key grasping parameters of grasping angle and grasping width and grasping quality score through three parallel output branches. Finally, the robot grasping data set is used to train the robot arm grasping angle and width prediction model to obtain the trained robot arm grasping angle and width prediction model.

[0078] S2, obtain the RGB-D image of the object to be grasped and input it into the trained robot arm grasping angle and width prediction model, the RGB-D image is input into the initial feature extraction module, and is sequentially subjected to feature extraction through the first depthwise separable convolution unit and the second depthwise separable convolution unit to obtain basic features, the basic features are downsampled through the maximum pooling layer to obtain initial features, the initial features are input into the feature enhancement module, multi-scale features are first extracted through two multi-scale feature extraction modules, and multi-scale features are fused and enhanced through a bidirectional feature pyramid network to obtain enhanced features, the enhanced features are subjected to the feature decoding module to predict the grasping angle, grasping width and grasping quality score, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robot arm.

[0079] Specifically, the trained robot gripping angle and width prediction model is deployed and applied, and the RGB-D image of the object to be grasped is input into the trained robot gripping angle and width prediction model to obtain the predicted gripping angle and width. The calculation process of each module in the robot gripping angle and width prediction model is described in detail below.

[0080] In a specific embodiment, the first depth-wise separable convolution unit includes a depth-wise separable convolution layer with a convolution kernel size of 9×9, a batch normalization layer, and a Mish activation function layer connected in sequence; the second depth-wise separable convolution unit includes a depth-wise separable convolution layer with a convolution kernel size of 5×5, a batch normalization layer, and a Mish activation function layer connected in sequence, and the step size of the maximum pooling layer is 2;

[0081] The input features of the depthwise separable convolutional layer are input into the depthwise separable convolutional layer, and depthwise convolution is performed first, as shown in the following formula:

[0082] ;

[0083] in, Indicates The input features of the depth-wise separable convolutional layer are channels, represents the corresponding depth convolution kernel, Indicates The output features of the depthwise convolution of channels, Represents the convolution operation;

[0084] Then perform point-by-point convolution, as shown below:

[0085] ;

[0086] in, represents the corresponding point-by-point convolution kernel, Indicates the number of channels, Indicates channels, Represents the output features of the depthwise separable convolutional layer.

[0087] Specifically, refer to Figure 3 In the initial feature extraction module, a depthwise separable convolution layer with a kernel size of 9×9 is used to extract features from the input RGB-D image, and then it is processed by a batch normalization layer and a Mish activation function layer in turn; then, the output features obtained in the previous step are input into a depthwise separable convolution layer with a kernel size of 5×5 for further feature extraction, and are also processed by a batch normalization layer and a Mish activation function layer in turn to obtain basic features; finally, a maximum pooling layer with a step size of 2 is used to downsample the basic features, reduce the spatial resolution of the features and retain the main feature information, and finally obtain the initial features.

[0088] The calculation process of the first depth-wise separable convolution unit and the second depth-wise separable convolution unit is as follows:

[0089] First, perform a depthwise separable convolution operation, including two steps: depthwise convolution and pointwise convolution. The depthwise convolution performs convolution operations on each channel of the input feature independently, and the pointwise convolution achieves the fusion of information between channels through a convolution layer with a convolution kernel size of 1×1. Then, batch normalization is performed on the features to standardize the feature distribution, as shown in the following formula:

[0090] ;

[0091] ;

[0092] in, represents the input features of the batch normalization layer, represents the batch mean, represents the batch standard deviation, represents the stability factor, and They represent the scaling and translation parameters respectively. represents the standardized features, Represents the features after batch normalization;

[0093] Finally, a nonlinear transformation is performed through the Mish activation function layer, as shown in the following formula:

[0094] ;

[0095] in, represents the base of natural logarithms, represents the natural logarithm, represents the hyperbolic tangent function, Represents the dot product operation.

[0096] In the maximum pooling layer, a 2×2 pooling window is used to slide and scan the input features of the maximum pooling layer with a step size of 2, and a maximum value selection operation is performed on each window area, as shown in the following formula:

[0097] ;

[0098] in, Indicates that the input feature map is at position The eigenvalue at Indicates by location The 2×2 pooling window area centered on Indicates that the output feature map is at position The eigenvalue at Indicates the maximum value operation;

[0099] The maximum pooling operation can reduce the spatial dimension of the feature map, reduce the amount of calculation, while retaining the significant features in the region and enhancing the translation invariance of the model. The size calculation formula of the output feature of the maximum pooling layer is:

[0100] ;

[0101] ;

[0102] in, and Respectively represent the height and width of the input features of the maximum pooling layer, represents the size of the pooling window, represents the step length, and Respectively represent the height and width of the output features of the maximum pooling layer, Indicates a floor operation.

[0103] In a specific embodiment, the hybrid attention module includes a channel attention module and a spatial attention module connected in sequence. The input features of the hybrid attention module pass through the channel attention module to obtain the channel attention weight, as shown in the following formula:

[0104] ;

[0105] ;

[0106] ;

[0107] in, and represents the height and width of the input features of the hybrid attention module, Represents the input feature of the hybrid attention module at position The eigenvalue at and Respectively represent the average pooling results and maximum pooling results of the channel dimension, Indicates the maximum value operation. represents a multi-layer perceptron, represents the sigmoid activation function, represents the channel attention weight;

[0108] The channel attention weight is multiplied by the input feature of the hybrid attention module to obtain the channel feature. The channel feature passes through the spatial attention module to obtain the spatial attention weight, as shown in the following formula:

[0109] ;

[0110] ;

[0111] ;

[0112] in, Indicates The channel characteristics of the channels, and Respectively represent the average pooling results and maximum pooling results of the spatial dimension, represents the convolution operation, represents the concatenation of channel dimensions, represents the spatial attention weight;

[0113] The spatial attention weights are multiplied by the channel features to obtain the output features of the hybrid attention module.

[0114] In a specific embodiment, the dilated convolution module includes three convolution layers with different dilation rates and a convolution kernel size of 3×3. The input features of the dilated convolution module are respectively input into three convolution layers with different dilation rates and a convolution kernel size of 3×3 in parallel to perform three convolution operations to obtain multi-scale features, as shown in the following formula:

[0115] ;

[0116] ;

[0117] ;

[0118] in, represents the input features of the dilated convolution module, , , Represent the convolution kernels with expansion rates of 1, 2, and 4 respectively. represents the convolution operation, , , They represent the output features of the input features of the dilated convolution module after the convolution kernels with dilation rates of 1, 2, and 4, respectively, and constitute multi-scale features.

[0119] In a specific embodiment, the bidirectional feature pyramid network includes a top-down path and a bottom-up path; , , As the original input features of the first level in the top-down path and the bottom-up path respectively , the original input features of the second level And the original input features of the third level ;

[0120] In the top-down path, the original input features of the third level are As the output features of the third level in the top-down pathway ; The calculation process of the second level and the first level in the top-down path is as follows:

[0121] ;

[0122] In the bottom-up path, the output features of the first level in the top-down path are combined with the input features of the first level Add together to get the output features of the first level in the bottom-up path. The calculation process of the second and third levels in the bottom-up path is as follows:

[0123] ;

[0124] in, Indicates the first The output features of the level, Indicates the first The output features of the level, Indicates The original input features of the level, is 1, 2 or 3, Conv represents the convolution operation, represents the upsampling operation, Represents a downsampling operation, which converts the output features of the third level in the bottom-up path As an enhanced feature.

[0125] Specifically, refer to Figure 4The feature enhancement module consists of two multi-scale feature extraction modules and a bidirectional feature pyramid network. The multi-scale feature extraction module consists of a hybrid attention module and a dilated convolution module. The feature enhancement is performed through the hybrid attention module, and then input into the dilated convolution module to perform three convolution operations with different dilation rates and a kernel size of 3×3 in parallel. The dilation rates are 1, 2, and 4 respectively. The output features of the convolution layers with three different dilation rates constitute multi-scale features ( , , ), which will be input into the Bidirectional Feature Pyramid Network (BiFPN) for feature fusion. The specific processing process is as follows:

[0126] The output features of the three convolutional layers with different expansion rates are used as input features of different levels in the bidirectional feature pyramid network. (rate=1) as the input feature of the first level , (rate=2) as the input feature of the second level , (rate=4) as the input feature of the third level .

[0127] The feature fusion process in the Bidirectional Feature Pyramid Network (BiFPN) specifically includes:

[0128] (1) Top-down path (from the third level to the first level):

[0129] Tier 3: ;

[0130] Tier 2: ;

[0131] Tier 1: ;

[0132] (2) Bottom-up path (from the first level to the third level):

[0133] Tier 1: ;

[0134] Tier 2: ;

[0135] Tier 3: ;

[0136] Finally, the output features of the third level in the bottom-up path are taken Enhanced features as output of bidirectional feature pyramid network , the enhanced feature contains information at multiple scales: Provides local details, Provides medium-range contextual information, Provides a wide range of contextual information.

[0137] In a specific embodiment, the feature decoding module includes a first convolutional layer, a progressive upsampling module and three output branches connected in sequence, the progressive upsampling module includes three stages of upsampling modules, the upsampling module includes a second convolutional layer and a bilinear upsampling layer connected in sequence, and the three output branches include a quality score branch; the enhanced feature is first subjected to feature dimensionality reduction by a first convolutional layer with a convolution kernel size of 1×1 to obtain a reduced dimensionality feature, as shown in the following formula:

[0138] ;

[0139] in, Represents enhanced features, represents the first convolution layer with a convolution kernel size of 1×1, represents the dimension reduction feature, Represents the convolution operation;

[0140] The reduced dimension features are input to the progressive upsampling module to perform three-stage upsampling operations, including:

[0141] Phase 1, Execution Convolution and bilinear upsampling:

[0142] ;

[0143] In the second stage, 3×3 convolution and bilinear upsampling are performed:

[0144] ;

[0145] In the third stage, 3×3 convolution and bilinear upsampling are performed:

[0146] ;

[0147] in, , , Represent the features after upsampling in three stages respectively. , , Represent the convolution kernels of size 3×3 in three stages respectively. represents a bilinear upsampling operation, Indicates the upsampling scaling factor;

[0148] The features after upsampling in the third stage are input into the three output branches for the following calculation process:

[0149] Crawl quality scoring branch:

[0150] ;

[0151] Grab the angular branch:

[0152] ;

[0153] Grab the width branch:

[0154] ;

[0155] in, , , Represent the convolutional layers of the three output branches respectively, represents the features after upsampling in the third stage, represents the sigmoid activation function, represents the linear rectification function, , , They represent the crawl quality score, crawl angle, and crawl width respectively.

[0156] Specifically, refer to Figure 5 In the feature decoding module, the first convolution layer with a convolution kernel size of 1×1 is used to reduce the feature dimension. Then, the feature map resolution is restored by a progressive upsampling module. Finally, the grasping angle and width as well as the grasping quality score are predicted through three parallel output branches. The grasping quality score indicates the reliability of the predicted grasping position, and its value range is [0,1], where 1 indicates the most reliable grasping position and 0 indicates the least reliable grasping position. The grasping angle indicates the rotation angle of the robot gripper relative to the horizontal direction. The grasping angle is encoded by the cosine function and the sine function, as shown in the following formula:

[0157] ;

[0158] ;

[0159] in, Indicates the predicted grasping angle, the value range is , and It represents the value after encoding the grasping angle through the cosine function and the sine function, that is, the angle actually grasped by the robot arm. It is multiplied by 2 to eliminate the periodic ambiguity of the angle.

[0160] The grip width indicates the size of the robot arm's gripper's opening and closing, in pixels, to accommodate different object sizes, as shown in the following formula:

[0161] ;

[0162] in, Indicates the actual grasping width of the robot arm. The scale factor representing the distance from pixel to actual value.

[0163] In actual scenarios, after obtaining the RGB-D image of the object to be grasped from the depth camera, the trained robotic arm grasping angle and width prediction model proposed in the embodiment of the present application is used to predict the grasping angle and grasping width, and generate the angle and width actually grasped by the robotic arm. The robotic arm controller receives the grasping angle and width, performs hand-eye calibration, and finally enables the robotic arm to grasp the object to be grasped.

[0164] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a robot arm grasping angle and width prediction device based on multiple features. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0165] The embodiment of the present application provides a robot arm grasping angle and width prediction device based on multiple features, including:

[0166] The model building module 1 is configured to build and train a robot arm grasping angle and width prediction model to obtain a trained robot arm grasping angle and width prediction model, wherein the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module, and a feature decoding module connected in sequence; the initial feature extraction module includes a first depthwise separable convolution unit, a second depthwise separable convolution unit, and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence;

[0167] The prediction module 2 is configured to obtain an RGB-D image of the object to be grasped and input it into a trained robot arm grasping angle and width prediction model. The RGB-D image is input into the initial feature extraction module, and is sequentially subjected to feature extraction by the first depthwise separable convolution unit and the second depthwise separable convolution unit to obtain basic features. The basic features are downsampled through the maximum pooling layer to obtain initial features. The initial features are input into the feature enhancement module, and are first subjected to multi-scale feature extraction modules to extract multi-scale features, and the multi-scale features are fused and enhanced through a bidirectional feature pyramid network to obtain enhanced features. The enhanced features are subjected to the feature decoding module to predict the grasping angle, grasping width and grasping quality score, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robot arm.

[0168] Figure 7Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7 As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0169] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0170] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0171] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0172] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0173] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0174] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0175] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0176] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0177] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0178] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0179] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0180] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0181] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0182] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the gripping angle and width of a robotic arm based on multiple features, characterized in that: The following steps are involved: A robot arm grasping angle and width prediction model is constructed and trained to obtain a trained robot arm grasping angle and width prediction model, wherein the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module, and a feature decoding module connected in sequence; the initial feature extraction module includes a first depthwise separable convolution unit, a second depthwise separable convolution unit, and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence; The RGB-D image of the object to be grasped is obtained and input into the trained robot grasping angle and width prediction model. The RGB-D image is input into the initial feature extraction module, and is sequentially subjected to the first depthwise separable convolution unit and the second depthwise separable convolution unit for feature extraction to obtain basic features. The basic features are downsampled through the maximum pooling layer to obtain initial features. The initial features are input into the feature enhancement module, and multi-scale features are first extracted through two multi-scale feature extraction modules, and multi-scale features are fused and enhanced through the bidirectional feature pyramid network to obtain enhanced features. The enhanced features are passed through the feature decoding module to predict the grasping angle, grasping width and grasping quality score. The feature decoding module includes a first convolution layer, a progressive upsampling module and three output branches connected in sequence. The progressive upsampling module includes three-stage upsampling modules. The upsampling module includes a second convolution layer and a bilinear upsampling layer connected in sequence. The three output branches include a quality score branch. The enhanced features are first subjected to feature dimension reduction through the first convolution layer with a convolution kernel size of 1×1 to obtain reduced dimension features, as shown in the following formula: Among them, F′ represents the enhanced feature, K 1×1 represents the first convolution layer with a convolution kernel size of 1×1, G represents the dimension reduction feature, Represents the convolution operation; The dimension reduction features are input to the progressive upsampling module to perform upsampling operations in three stages, including: In the first stage, 3×3 convolution and bilinear upsampling are performed: In the second stage, 3×3 convolution and bilinear upsampling are performed: In the third stage, 3×3 convolution and bilinear upsampling are performed: Among them, P1, P2, and P3 represent the features after upsampling in the three stages, K′1, K′2, and K′3 represent the convolution kernels of size 3×3 in the three stages, Upsample represents the bilinear upsampling operation, and scale represents the upsampling scaling factor; The features after upsampling in the third stage are input into the three output branches for the following calculation process: Crawl quality scoring branch: Q=σ(Conv Q (P3)); Grab the angular branch: θ=Conv θ (P3); Grab the width branch: W=ReLU(Conv W (P3)); Among them, Conv Q 、Conv θ 、Conv W They represent the convolutional layers of the three output branches respectively, P3 represents the features after upsampling in the third stage, σ represents the sigmoid activation function, ReLU represents the linear rectification function, Q, θ, and W represent the grasping quality score, grasping angle, and grasping width, respectively, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robot arm.

2. The method for predicting the gripping angle and width of a robotic arm based on multiple features according to claim 1, characterized in that: The first depth-wise separable convolution unit includes a depth-wise separable convolution layer with a convolution kernel size of 9×9, a batch normalization layer, and a Mish activation function layer connected in sequence; the second depth-wise separable convolution unit includes a depth-wise separable convolution layer with a convolution kernel size of 5×5, a batch normalization layer, and a Mish activation function layer connected in sequence, and the step size of the maximum pooling layer is 2; The input features of the depthwise separable convolutional layer are input into the depthwise separable convolutional layer, and depthwise convolution is first performed, as shown in the following formula: Among them, X k represents the input feature of the depthwise separable convolutional layer of the kth channel, D k represents the corresponding depth convolution kernel, Y k represents the output feature of the depthwise convolution of the kth channel, Represents the convolution operation; Then perform point-by-point convolution, as shown below: Among them, P k Represents the corresponding point-by-point convolution kernel, C represents the number of channels, k represents the kth channel, and Z represents the output feature of the depth-wise separable convolution layer.

3. The method for predicting the gripping angle and width of a robotic arm based on multiple features according to claim 1, characterized in that: The hybrid attention module includes a channel attention module and a spatial attention module connected in sequence. The input features of the hybrid attention module pass through the channel attention module to obtain the channel attention weight, as shown in the following formula: Wherein, H and W′ represent the height and width of the input feature of the hybrid attention module, F(i,j) represents the feature value of the input feature of the hybrid attention module at position (i,j), and They represent the average pooling result and the maximum pooling result of the channel dimension respectively, max represents the maximum value operation, MLP represents the multi-layer perceptron, σ represents the sigmoid activation function, and M c represents the channel attention weight; The channel attention weight is multiplied by the input feature of the hybrid attention module to obtain the channel feature, and the channel feature is passed through the spatial attention module to obtain the spatial attention weight, as shown in the following formula: Among them, F k represents the channel feature of the kth channel, and They represent the average pooling result and the maximum pooling result of the spatial dimension respectively, Conv represents the convolution operation, [;] represents the concatenation of the channel dimension, and M s represents the spatial attention weight; The spatial attention weight is multiplied by the channel feature to obtain the output feature of the hybrid attention module.

4. The method for predicting the gripping angle and width of a robotic arm based on multiple features according to claim 1, characterized in that: The atrous convolution module includes three convolution layers with different dilation rates and a convolution kernel size of 3×3. The input features of the atrous convolution module are respectively input into three convolution layers with different dilation rates and a convolution kernel size of 3×3 in parallel to perform three convolution operations to obtain multi-scale features, as shown in the following formula: Among them, X represents the input feature of the hole convolution module, K1, K2, and K4 represent the convolution kernels with expansion rates of 1, 2, and 4, respectively. represents a convolution operation, Y1, Y2, and Y4 represent the output features of the input features of the dilated convolution module after the convolution kernels with expansion rates of 1, 2, and 4, respectively, and constitute the multi-scale features.

5. The method for predicting the gripping angle and width of a robot arm based on multiple features according to claim 4, characterized in that: The bidirectional feature pyramid network includes a top-down path and a bottom-up path; Y1, Y2, and Y4 are used as the original input feature F1 of the first level, the original input feature F2 of the second level, and the original input feature F3 of the third level in the top-down path and the bottom-up path, respectively; In the top-down path, the original input feature F3 of the third level is used as the output feature of the third level in the top-down path The calculation process of the second level and the first level in the top-down path is as follows: In the bottom-up path, the output features of the first level in the top-down path are added to the input features F1 of the first level to obtain the output features of the first level in the bottom-up path. The calculation process of the second and third levels in the bottom-up path is as follows: in, represents the output features of the lth level in the top-down path, represents the output feature of the lth level in the bottom-up path, F l Represents the original input features of the lth level, l is 1, 2 or 3, Conv represents the convolution operation, U represents the upsampling operation, and D represents the downsampling operation. The output features of the third level in the bottom-up path are As the enhanced feature.

6. A robot arm grasping angle and width prediction device based on multiple features, characterized in that: A method for predicting the gripping angle and width of a robotic arm based on multiple features as claimed in any one of claims 1 to 5, comprising: A model building module is configured to build and train a robot arm grasping angle and width prediction model to obtain a trained robot arm grasping angle and width prediction model, wherein the robot arm grasping angle and width prediction model includes an initial feature extraction module, a feature enhancement module, and a feature decoding module connected in sequence; the initial feature extraction module includes a first depthwise separable convolution unit, a second depthwise separable convolution unit, and a maximum pooling layer connected in sequence, the feature enhancement module includes two multi-scale feature extraction modules connected in sequence and a bidirectional feature pyramid network, and the multi-scale feature extraction module includes a hybrid attention module and a hole convolution module connected in sequence; The prediction module is configured to obtain an RGB-D image of the object to be grasped and input it into the trained robotic arm grasping angle and width prediction model, the RGB-D image is input into the initial feature extraction module, and is sequentially subjected to feature extraction by a first depthwise separable convolution unit and a second depthwise separable convolution unit to obtain basic features, the basic features are downsampled by the maximum pooling layer to obtain initial features, the initial features are input into the feature enhancement module, multi-scale features are first extracted by two multi-scale feature extraction modules, and multi-scale features are fused and enhanced by the bidirectional feature pyramid network to obtain enhanced features, the enhanced features are passed through the feature decoding module to predict the grasping angle, grasping width and grasping quality score, and the grasping angle and grasping width are converted into the angle and width actually grasped by the robotic arm.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images