A lightweight grabbing detection method based on deep high-resolution convolutional neural network

Abstract

The application discloses a kind of light weight grabbing detection methods based on depth high-resolution convolutional neural network, belong to robot grabbing field.Method includes: the RGB image and depth image of the scene to be grabbed are collected;The RGB image and depth image of the scene to be grabbed are preprocessed respectively;Depth high-resolution convolutional neural network is constructed;Multi-modal data sample set is constructed to train and optimize the network;The preprocessed RGB image and depth image are input into the network after training, and finally output the feature map of three dimensions of grabbing quality Q, grabbing angle Angle and grabbing width Width related to grabbing;Robot executes grabbing action according to the feature map of three dimensions, if grabbing fails, multi-modal data of current failure scene is collected and added to multi-modal data sample set, for the incremental training of depth high-resolution convolutional neural network.The application improves the accuracy of grabbing detection.

Description

Technical Field

[0001] This invention belongs to the field of robot grasping. Specifically, this invention relates to a lightweight grasping and detection method based on a deep high-resolution convolutional neural network. Background Technology

[0002] With the rapid development of intelligent robots, robot grasping technology has been widely applied in industrial, agricultural, and home environments. For robot grasping in unstructured environments, it is necessary to generate a grasping configuration for the object to be grasped based on environmental perception, including the required gripper width and the object's posture. Two-dimensional grasping planning is a key issue in robot planar grasping technology, its core being the acquisition of the grasping configuration from input sensor data. Traditional two-dimensional grasping planning uses analytical methods, calculating the grasping configuration through geometry, kinematics, and dynamics. Such methods struggle to effectively handle uncertainties in the object's shape, size, and posture.

[0003] Traditional two-dimensional grasping planning relies on analytical calculations of geometry, kinematics, and dynamics. When faced with unstructured environments containing objects of irregular shapes (such as irregularly shaped industrial parts or natural agricultural products), large size differences (such as electronic components of different specifications), and random changes in posture (such as tilted or stacked objects), it cannot quickly and accurately generate suitable grasping configurations, easily leading to grasping failures or low efficiency. This makes it difficult to meet the demands for grasping flexibility and stability in real-world scenarios. Existing deep learning models are mostly trained on specific datasets (such as single object types or fixed scenes). When applied to unseen object categories (such as expanding from industrial parts to everyday household items) or new environmental disturbances (such as cluttered objects or complex background textures), the model's prediction accuracy for grasping configurations drops significantly, exhibiting weak generalization ability and limiting its application in diverse real-world scenarios.

[0004] Therefore, this invention proposes a lightweight grasping and detection method based on a deep high-resolution convolutional neural network. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and proposes a lightweight grasping detection method based on deep high-resolution convolutional neural networks to achieve the following objective: improve the accuracy of grasping detection.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a lightweight grasping and detection method based on a deep high-resolution convolutional neural network, the method comprising the following steps: Step S1: Acquire RGB and depth images of the scene to be captured; Step S2: Preprocess the RGB image and depth image of the scene to be captured, respectively; Step S3: Construct a deep, high-resolution convolutional neural network; Step S4: Construct a multimodal data sample set to train and optimize the deep high-resolution convolutional neural network; Step S5: Input the preprocessed RGB image and depth image into the trained deep high-resolution convolutional neural network, and finally output feature maps of three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width. Step S6: The robot performs a grasping action based on the feature maps of the three dimensions of grasping quality Q, grasping angle Angle, and grasping width Width. If the grasping fails, the robot collects multimodal data of the current failed scenario and adds it to the multimodal data sample set for incremental training of the deep high-resolution convolutional neural network.

[0007] Preferably, in step S1, an RGB camera and a depth sensor are used to simultaneously acquire RGB images and depth images of the scene to be captured, wherein the acquisition frame rates of the RGB camera and the depth sensor are kept consistent.

[0008] Preferably, in step S2, the preprocessing includes: normalizing the acquired RGB image to scale the pixel values ​​to the range of [0,1]; and normalizing the depth image after denoising to scale the pixel values ​​to the range of [0,1].

[0009] Preferably, in step S3, the deep high-resolution convolutional neural network includes an encoding layer, a residual layer, and a decoding layer. The deep high-resolution convolutional neural network takes the RGB image and the depth image as input to the encoding layer. After the input RGB image and depth image are processed by the encoding layer, the residual layer, and the decoding layer in sequence, the final output is a feature map with three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width.

[0010] Preferably, the coding layer includes multiple downsampling units connected in sequence, and each downsampling unit includes a convolution module (Conv), a batch normalization module (BN), and a Mish activation function module connected in sequence.

[0011] Preferably, each downsampling unit further includes an ECA module, which is connected after the Mish activation function module of each downsampling unit.

[0012] Preferably, the residual layer includes a plurality of residual units connected in sequence. Each residual unit includes a convolutional module Conv, a batch normalization module BN, and a Mish activation function module connected in sequence. In addition, each residual unit also includes a jump connection set between the input of the convolutional module Conv and the output of the Mish activation function module.

[0013] Preferably, the decoding layer includes multiple decoding units connected in sequence, and each decoding unit includes a transposed convolution module UpConvT, a batch normalization module BN, a convolution module Conv, a batch normalization module BN, and a Mish activation function module connected in sequence.

[0014] Preferably, step S4 includes: constructing a multimodal data sample set containing objects in different scenarios for training and optimization of a deep high-resolution convolutional neural network, wherein the grasping quality Q, grasping angle Angle, and grasping width Width of the grasping region are labeled for each sample.

[0015] Preferably, during the training and optimization process, a multi-task loss function is used to calculate and weight the loss of the prediction results of grasping quality Q, grasping angle Angle, and grasping width Width output by the deep high-resolution convolutional neural network. The loss of grasping quality Q adopts mean square error; the loss of grasping angle Angle adopts smoothed L1 loss; and the loss of grasping width Width adopts mean square error.

[0016] The technical effects of this invention are as follows: This invention improves the accuracy of target recognition by using multimodal input methods of RGB and depth images, compared to single-modal methods. The deep high-resolution convolutional neural network constructed in this invention, with the introduction of residual structures and attention mechanisms, enables the network to learn object grasping features more accurately, thereby improving the accuracy of predictions such as grasping angle and width. Simultaneously, the entire deep high-resolution convolutional neural network has a simple structure, reducing network complexity and achieving a lightweight design, facilitating rapid inference on embedded devices. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a deep high-resolution convolutional neural network structure provided in an embodiment of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. This is to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solutions of the present invention, and to facilitate its implementation. It should be noted that the terms "first," "second," etc., used in this application are only for the convenience of describing the technical solutions and to distinguish components; the corresponding component configurations may be the same or different, and are not intended to limit the scope of this application. To make the technical solutions of the present invention clearer, the present invention will be explained and illustrated through the following embodiments.

[0019] This embodiment provides a lightweight grasping and detection method based on a deep high-resolution convolutional neural network, the method including the following steps: Step S1: Acquire RGB and depth images of the scene to be captured; Step S2: Preprocess the RGB image and depth image of the scene to be captured, respectively; Step S3: Construct a deep, high-resolution convolutional neural network; Step S4: Construct a multimodal data sample set to train and optimize the deep high-resolution convolutional neural network; Step S5: Input the preprocessed RGB image and depth image into the trained deep high-resolution convolutional neural network, and finally output feature maps of three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width. Step S6: The robot performs a grasping action based on the feature maps of the three dimensions of grasping quality Q, grasping angle Angle, and grasping width Width. If the grasping fails, the robot collects multimodal data of the current failed scenario and adds it to the multimodal data sample set for incremental training of the deep high-resolution convolutional neural network.

[0020] Specifically, referring to step S1, this embodiment uses an RGB camera with a resolution of 224×224 and a depth sensor to simultaneously acquire RGB images and depth images of the scene to be captured. The RGB image acquisition frame rate is 30fps, and the depth image acquisition frame rate is the same as that of the RGB image. By keeping the acquisition frame rate consistent between the RGB camera and the depth sensor, the synchronization in time is ensured.

[0021] Referring to step S2, the acquired RGB image is normalized to scale the pixel values ​​to the range of [0,1]. The depth image is first denoised, for example, in this embodiment, median filtering is used for denoising with a window size of 3×3; after denoising, normalization is performed again, similarly scaling the pixel values ​​to the range of [0,1], so that its numerical range matches that of the RGB image, facilitating subsequent network processing.

[0022] See step S3, such as Figure 1 As shown, this embodiment constructs a deep high-resolution convolutional neural network for grasping detection. The deep high-resolution convolutional neural network includes an encoding layer, a residual layer, and a decoding layer. The deep high-resolution convolutional neural network takes an RGB image and a depth image as input to the encoding layer. The input RGB image and depth image are processed sequentially through the encoding layer, the residual layer, and the decoding layer, and finally output feature maps in three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width.

[0023] The coding layer in this embodiment includes multiple downsampling units connected in sequence. Each downsampling unit includes a convolutional module (Conv), a batch normalization module (BN), and a Mish activation function module connected in sequence. Multiple downsampling units progressively increase the number of channels to obtain feature maps with more channels, thereby extracting features at different levels. At this time, the downsampling units are used to extract features from the RGB image and the depth image. Controlling the number of downsampling operations during the downsampling process can maintain a medium-to-high spatial resolution feature map. In addition, each downsampling unit in this embodiment also includes an ECA module (Lightweight Channel Attention), which is connected after the Mish activation function module of each downsampling unit. The ECA module is used to adaptively weight the capture-related feature channels. By learning the attention weights of the channel dimensions, it strengthens the extraction of capture-related key features, suppresses irrelevant background information, and improves the discriminative power of the features.

[0024] The residual layer in this embodiment is constructed at the same spatial resolution as the output of the encoding layer, enabling deep representation of extracted features while maintaining the spatial resolution of the feature map. The residual layer in this embodiment includes multiple residual units connected in sequence. Each residual unit includes a convolutional module (Conv), a batch normalization module (BN), and a Mish activation function module connected in sequence. Additionally, each residual unit includes a skip connection between the input of the convolutional module (Conv) and the output of the Mish activation function module. Through these skip connections, the residual units effectively address the vanishing gradient problem during deep network training, improving the depth and accuracy of feature extraction.

[0025] The decoding layer of the residual unit includes multiple decoding units connected in sequence. Each decoding unit includes a transposed convolution module UpConvT, a batch normalization module BN, a convolution module Conv, a batch normalization module BN, and a Mish activation function module connected in sequence.

[0026] In this embodiment, after the RGB image and depth image are input, they first pass through three encoding units sequentially to obtain feature maps with 32, 64, and 128 channels respectively, to achieve feature extraction at different levels. Each encoding unit incorporates an ECA module to enhance the extraction of key features. Next, the 128-channel feature map is input to the residual layer. Multiple 128-channel residual units alleviate the gradient vanishing problem, allowing the network to be trained deeper while preserving the 128-channel features of the encoding layer. Then, the 128-channel feature map output from the residual layer is upsampled by a decoding layer. The decoding layer has two decoding units, progressively upsampling the feature map to the original image size, gradually reducing the number of channels from 128 to 64 and then 32. Finally, feature maps related to grasping—grasping quality Q, grasping angle Angle, and grasping width Width—are output.

[0027] See step S4. After the deep, high-resolution convolutional neural network is built, it still needs to be trained and optimized before it can be used for actual robot grasping tasks.

[0028] Specifically, this embodiment constructs a multimodal data sample set containing objects in different scenarios for training and optimization of a deep high-resolution convolutional neural network. For each sample, the grasping quality Q, grasping angle Angle, and grasping width Width of the grasping region are labeled. The grasping quality Q is represented by a continuous value between 0 and 1, the grasping angle Angle ranges from -90° to 90°, and the grasping width Width is labeled within the range of 0-200mm according to the object size.

[0029] During training and optimization, a multi-task loss function is employed to calculate and weightedly sum the prediction results of grasping quality Q, grasping angle Angle, and grasping width Width from the deep high-resolution convolutional neural network output. Specifically, the grasping quality Q loss uses mean squared error; the grasping angle Angle loss uses smoothed L1 loss to reduce the impact of outliers; and the grasping width Width loss uses mean squared error. After calculating the losses for grasping quality Q, grasping angle Angle, and grasping width Width, they are weighted and summed. In this embodiment, the weights for the losses of grasping quality Q, grasping angle Angle, and grasping width Width are set to 1.0, 1.5, and 1.0, respectively, to ensure balanced optimization of the model across the three tasks.

[0030] During the training optimization process, the Adam optimizer was used with an initial learning rate of 1e-4, which was reduced to 0.5 every 10 epochs. The batch size was set to 16 and the number of training epochs was set to 100. An early stopping strategy was adopted during training, and training was stopped when the validation set loss no longer decreased for 15 consecutive epochs to prevent overfitting.

[0031] Referring to steps S5 and S6, the preprocessed RGB image and depth image are input into the trained deep high-resolution convolutional neural network, which ultimately outputs feature maps in three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width. The robot performs grasping actions based on the output feature maps in the three dimensions of grasping quality Q, grasping angle Angle, and grasping width Width. If the grasping fails, the multimodal data of the current failure scene is collected and added to the multimodal data sample set constructed in step S4 for incremental training of the deep high-resolution convolutional neural network, thereby improving the model's adaptability in complex scenes.

[0032] In summary, this embodiment utilizes the texture and color information of RGB images and the spatial and geometric information of depth images. Features are extracted separately through the encoding layer and then fused in subsequent layers. This compensates for the information loss problem of single modalities in complex scenes, enabling the network to perceive objects more comprehensively and improving the accuracy of target recognition. The residual structure solves the gradient vanishing problem in deep networks, allowing the network to learn more in-depth features. The ECA attention mechanism weights channel features, strengthening key features relevant to grasping and suppressing irrelevant noise. The combination of these two mechanisms enhances the discriminative and expressive power of features, thereby improving the accuracy of predictions such as grasping angle and width. In practical applications, by rationally designing the structure and parameters of the encoding, residual, and decoding layers, the number of network parameters and computational complexity are controlled while ensuring feature extraction capabilities. This allows the network to perform fast inference on embedded devices while maintaining high grasping detection accuracy, balancing the model's accuracy and efficiency.

[0033] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.

Claims

1. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network, characterized in that: The method includes the following steps: Step S1: Acquire RGB and depth images of the scene to be captured; Step S2: Preprocess the RGB image and depth image of the scene to be captured, respectively; Step S3: Construct a deep, high-resolution convolutional neural network; Step S4: Construct a multimodal data sample set to train and optimize the deep high-resolution convolutional neural network; Step S5: Input the preprocessed RGB image and depth image into the trained deep high-resolution convolutional neural network, and finally output feature maps of three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width. Step S6: The robot performs a grasping action based on the feature maps of the three dimensions of grasping quality Q, grasping angle Angle, and grasping width Width. If the grasping fails, the robot collects multimodal data of the current failed scenario and adds it to the multimodal data sample set for incremental training of the deep high-resolution convolutional neural network.

2. The lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that: In step S1, an RGB camera and a depth sensor are used to simultaneously acquire RGB images and depth images of the scene to be captured, wherein the acquisition frame rates of the RGB camera and the depth sensor are kept consistent.

3. The lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that: In step S2, the preprocessing includes: normalizing the acquired RGB image to scale the pixel values ​​to the range of [0,1]; and normalizing the depth image after denoising to scale the pixel values ​​to the range of [0,1].

4. The lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that: In step S3, the deep high-resolution convolutional neural network includes an encoding layer, a residual layer, and a decoding layer. The deep high-resolution convolutional neural network takes the RGB image and the depth image as input to the encoding layer. After the input RGB image and depth image are processed by the encoding layer, the residual layer, and the decoding layer in sequence, the final output is a feature map with three dimensions related to grasping: grasping quality Q, grasping angle Angle, and grasping width Width.

5. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 4, characterized in that: The coding layer includes multiple downsampling units connected in sequence. Each downsampling unit includes a convolution module (Conv), a batch normalization module (BN), and a Mish activation function module connected in sequence.

6. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 5, characterized in that: Each downsampling unit also includes an ECA module, which is connected after the Mish activation function module of each downsampling unit.

7. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 4, characterized in that: The residual layer includes multiple residual units connected in sequence. Each residual unit includes a convolution module (Conv), a batch normalization module (BN), and a Mish activation function module connected in sequence. In addition, each residual unit also includes a jump connection set between the input of the convolution module (Conv) and the output of the Mish activation function module.

8. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 4, characterized in that: The decoding layer includes multiple decoding units connected in sequence. Each decoding unit includes a transposed convolution module UpConvT, a batch normalization module BN, a convolution module Conv, a batch normalization module BN, and a Mish activation function module connected in sequence.

9. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that: Step S4 includes: constructing a multimodal data sample set containing objects in different scenarios for training and optimization of a deep high-resolution convolutional neural network, wherein the grasping quality Q, grasping angle Angle, and grasping width Width of the grasping region are labeled for each sample.

10. A lightweight grasping and detection method based on a deep high-resolution convolutional neural network according to claim 9, characterized in that: During the training and optimization process, a multi-task loss function is used to calculate and weight the loss of the prediction results of grasping quality Q, grasping angle Angle, and grasping width Width output by the deep high-resolution convolutional neural network. The loss of grasping quality Q is calculated using mean squared error; the loss of grasping angle Angle is calculated using smoothed L1 loss; and the loss of grasping width Width is calculated using mean squared error.

Images