A robot real-time grabbing detection method based on an improved CNN
By combining an improved CNN model with an RGB-D camera and depth-separable convolution and iteratively refined pyramid modules, the robustness and multi-scale adaptability of robot grasping and detection in complex scenarios are solved, achieving high-precision real-time grasping and detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI POLYTECHNIC UNIV
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-02
AI Technical Summary
Existing robot grasping and detection methods lack robustness in complex scenarios, have poor adaptability to multi-scale objects, and struggle to balance detection accuracy and real-time performance.
An improved CNN model is adopted, which is combined with an RGB-D camera to acquire multimodal data. Feature extraction and fusion are performed through depthwise separable convolution and iterative refined pyramid modules. The model is trained using the SmoothL1 loss function and the Adam optimizer to achieve high-precision real-time grasping and detection.
It improves the robustness and positioning accuracy of grasping and detection in complex environments, adapts to objects of multiple scales, and achieves a balance between detection accuracy and speed, making it suitable for intelligent manufacturing and automated logistics scenarios.
Smart Images
Figure CN122125706A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot grasping, specifically, it relates to a real-time robot grasping and detection method based on an improved CNN. Background Technology
[0002] Robotic grasping and detection systems are a core technology for achieving autonomous robot operations, widely used in intelligent manufacturing, automated logistics, and unmanned systems. Their performance typically depends on various complex visual environmental factors, such as the geometry of objects, the clutter level of the scene, occlusion between objects, and varying lighting conditions. Traditionally, robotic grasping and detection relies on geometric modeling or manually designed visual features, but these methods often suffer from inaccurate grasping and localization due to weak model generalization ability and difficulty in handling the complexity of real-world scenes. With the development of deep learning technology, especially techniques represented by Convolutional Neural Networks (CNNs), powerful capabilities have been demonstrated in processing high-dimensional visual data and learning complex nonlinear features. Currently, although numerous studies have applied deep learning methods to robotic grasping and detection, how to utilize CNNs to build a detection system that simultaneously meets high accuracy and high real-time performance requirements, adaptable to computationally limited robotic platforms, remains a significant challenge in this field.
[0003] Existing deep learning-based crawling detection methods have achieved significant performance improvements, but they still have the following drawbacks:
[0004] 1. Insufficient robustness in complex scenarios: In unstructured environments, there are often complex situations such as stacked objects, mutual occlusion, background noise, and changes in lighting. When dealing with these interference factors, the existing models' ability to identify potential graspable areas decreases, resulting in insufficient grasping and localization accuracy and robustness.
[0005] 2. Poor adaptability to multi-scale objects: Traditional convolutional networks typically use continuous downsampling to acquire the global receptive field, but this can blur or lose fine geometric details such as edge contours that are crucial for grasping decisions. This makes the model poorly adaptable when simultaneously handling multiple objects of varying sizes in a scene, making it difficult to accurately generate grasping poses.
[0006] 3. Difficulty in balancing detection accuracy and real-time performance: Existing grasping and detection algorithms generally suffer from a trade-off between accuracy and speed. Some methods sacrifice some accuracy to improve detection speed; while others, although highly accurate, suffer from long inference times due to their complex network structures, making it difficult to meet the real-time grasping response requirements of robotic systems.
[0007] Therefore, this invention proposes a real-time robot grasping and detection method based on an improved CNN. Summary of the Invention
[0008] This invention aims to overcome the shortcomings of existing technologies and proposes a real-time robot grasping and detection method based on an improved CNN to achieve the following objective: improve the accuracy of robot grasping and detection.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a real-time robot grasping and detection method based on improved CNN, the method comprising the following steps:
[0010] Step S1: Image data acquisition and preprocessing;
[0011] Step S2: Based on the preprocessed image data, construct training dataset samples and test dataset samples;
[0012] Step S3: Construct an improved CNN model, randomly initialize the model parameters, input the training dataset, train the improved CNN model using the gradient descent algorithm, and fix the optimal model parameters; the improved CNN model includes a multi-scale feature extraction network based on depthwise separable convolution, an iterative refined pyramid module, and a prediction head;
[0013] Step S4: Import the test dataset into the trained optimal model to verify the effectiveness and evaluate the performance of the robot's grasping.
[0014] Preferably, in step S1, an RGB-D camera is used to acquire visual information of the scene; for each scene to be detected, the RGB-D camera simultaneously acquires an RGB color image and a corresponding depth image.
[0015] Preferably, the preprocessing in step S1 includes: adjusting the resolution of the acquired original RGB image and depth image to the same pixel size; and fusing the two modal images after resolution unification, that is, stitching the 3-channel RGB image and the 1-channel depth image together in the channel dimension to form a four-channel input tensor as the preprocessed image data.
[0016] Preferably, step S2 includes: dividing the preprocessed image data into training dataset samples and test dataset samples according to a preset ratio; wherein, each sample is assigned a corresponding grasping label; and finally, each sample and its corresponding grasping label are converted into a format that the neural network can directly learn.
[0017] Preferably, converting each sample and its corresponding grabbing label into a format that the neural network can directly learn includes: encoding each grabbing label into four pixel-level ground truth maps of the same size as the input image: a grabbing quality map, a cosine component map of the grabbing angle, a sine component map, and a grabbing width map. These ground truth maps will serve as supervision signals during model training, guiding the model to learn the mapping relationship from the input image to the desired output.
[0018] Preferably, step S2 further includes performing data augmentation operations on the data in the training dataset samples.
[0019] Preferably, in step S3, the multi-scale feature extraction network based on depthwise separable convolution uses CNN as the feature extraction network, and on this basis, the standard convolution of CNN is replaced with depthwise separable convolution.
[0020] Preferably, in step S3, the iterative refined pyramid module fuses the multi-scale features extracted by the feature extraction network through a cyclic iterative process. In each iteration, the iterative refined pyramid module performs a complete top-down and bottom-up bidirectional information transfer:
[0021] Top-down approach: The highest-level semantic features are upsampled and fused with the features of the next level, and strong semantic information is passed to shallower features that pay more attention to details;
[0022] Bottom-up approach: After top-down fusion, shallow features rich in precise spatial information are downsampled and fused with features from the previous level, thereby using precise positioning information to correct the semantic representation of the higher level.
[0023] This process is repeated iteratively until the preset number of iterations is reached, at which point the iteration terminates and a multi-scale optimized feature map is output. The input for the first iteration is the multi-scale features extracted by the feature extraction network, and the input for each subsequent iteration is the multi-scale feature map output from the previous iteration.
[0024] Preferably, in step S3, the prediction head includes two 3×3 convolutional layers for decoding the multi-scale optimized feature maps output by the iteratively refined pyramid module into a final pixel-level grasping prediction map, including a grasping quality map, a grasping angle map, and a grasping width map.
[0025] Preferably, in step S3, the model training uses the SmoothL1 loss function, defined as:
[0026] ;
[0027] Among them, threshold 1, any pixel The predicted value is The true value is .
[0028] The technical effects of this invention are as follows:
[0029] This invention, by fusing multimodal data from RGB color images and depth images, can simultaneously utilize the color texture information and spatial geometric features of objects. This improves the robustness and positioning accuracy of grasping and detection in environments with stacked objects, mutual occlusion, and changing lighting.
[0030] By introducing rich data augmentation strategies such as random rotation, scaling, and cropping during the training phase, this invention improves the detection speed of the CNN model through a multi-scale feature extraction network based on depthwise separable convolutions. Furthermore, by iteratively refining the pyramid module and rationally arranging the number of iterations, it improves detection accuracy while maintaining detection speed, achieving a balance between the two. Attached Figure Description
[0031] Figure 1 This is a flowchart of a real-time robot grasping and detection method based on an improved CNN, provided as an embodiment of the present invention. Detailed Implementation
[0032] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. This is to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solutions of the present invention, and to facilitate its implementation. It should be noted that the terms "first," "second," etc., used in this application are only for the convenience of describing the technical solutions and to distinguish components; the corresponding component configurations may be the same or different, and are not intended to limit the scope of this application. To make the technical solutions of the present invention clearer, the present invention will be explained and illustrated through the following embodiments.
[0033] This invention provides a real-time robot grasping and detection method based on an improved CNN. The aim is to extract and learn multi-dimensional features from image data using an improved CNN, thereby efficiently and accurately predicting key performance indicators of the robot grasping and detection system. For example... Figure 1 As shown, the method includes the following steps:
[0034] Step S1: Image data acquisition and preprocessing;
[0035] Step S2: Based on the preprocessed image data, construct training dataset samples and test dataset samples;
[0036] Step S3: Construct an improved CNN model, randomly initialize the model parameters, input the training dataset, train the improved CNN model using the gradient descent algorithm, and fix the optimal model parameters; the improved CNN model includes a multi-scale feature extraction network based on depthwise separable convolution, an iterative refined pyramid module, and a prediction head;
[0037] Step S4: Import the test dataset into the trained optimal model to verify the effectiveness and evaluate the performance of the robot's grasping.
[0038] Specifically, referring to step S1, this embodiment uses an RGB-D camera fixed above the robot's work area or integrated into the end of the robot arm to acquire visual information about the scene. For each scene to be detected, the RGB-D camera simultaneously acquires an RGB color image and a corresponding depth image.
[0039] Then, the acquired RGB color images and corresponding depth images are preprocessed to facilitate subsequent model learning. The preprocessing operations include: adjusting the resolution of the acquired original RGB images and depth images to the same pixel size (e.g., 300×300); and fusing the two modal images after resolution unification, that is, stitching the 3-channel RGB image and the 1-channel depth image in the channel dimension to form a four-channel input tensor (e.g., 300×300×4) as the preprocessed image data.
[0040] By fusing multimodal data from RGB color images and depth images, this embodiment can simultaneously utilize the color texture information and spatial geometric features of objects. This improves the robustness of grasping and detection and the accuracy of positioning in environments with stacked objects, mutual occlusion, and changes in lighting.
[0041] Referring to step S2, in this embodiment, the preprocessed image data is divided into training dataset samples and test dataset samples according to a preset ratio. For example, a 9:1 ratio is used, meaning 90% of the data is used as the training dataset and the remaining 10% is used as the test dataset to evaluate the model's performance and generalization ability. Each sample is assigned a corresponding grasping annotation (including center point, angle, width, etc.) to guide the model in learning the mapping relationship from the input image to the desired output. Finally, each sample and its corresponding grasping annotation are converted into a format that the neural network can directly learn. That is, each grasping annotation is encoded into four pixel-level ground truth maps of the same size as the input image: a grasping quality map, a cosine component map of the grasping angle, a sine component map, and a grasping width map. These ground truth maps will serve as supervision signals during model training, guiding the model to learn the mapping relationship from the input image to the desired output.
[0042] In addition, step S2 of this embodiment performs data augmentation operations on the data in the training dataset samples, while leaving the test dataset data unprocessed to ensure the objectivity of the evaluation. Data augmentation operations involve random rotation, scaling and cropping, and translation of the training dataset data, effectively expanding the diversity of the training samples. This enables the subsequent model to learn general grasping features that are insensitive to changes in object size, position, and pose, thereby improving its adaptability to objects of different sizes and its generalization ability to new, unseen objects.
[0043] Referring to step S3, this embodiment constructs an improved CNN model for predicting robot grasping detection values. The improved CNN model includes a multi-scale feature extraction network based on depthwise separable convolutions, an iterative refined pyramid module, and a prediction head. The multi-scale feature extraction network based on depthwise separable convolutions is used to extract features from the input RGB-D image to obtain a multi-scale feature map. The iterative refined pyramid module is used to fuse and optimize the multi-scale feature map to obtain an optimized multi-scale feature map. Finally, the optimized multi-scale feature map is converted into a final pixel-level grasping prediction map through the prediction head, including a grasping quality map, a grasping angle map, and a grasping width map.
[0044] Specifically, the multi-scale feature extraction network based on depthwise separable convolutions uses CNNs as the feature extraction network. CNNs extract features layer by layer by stacking convolutional layers. Furthermore, the standard CNN convolution (3×3 Conv) is replaced with depthwise separable convolutions. The standard CNN convolution structure consists of 3×3 Conv, batch normalization (BN), and ReLU activation function. The depthwise separable convolution, on the other hand, consists of 3×3 depthwise convolution, batch normalization (BN), ReLU activation function, 1×1 pointwise convolution, batch normalization (BN), and ReLU activation function. Compared to standard convolutions, depthwise separable convolutions can effectively reduce computation and parameter count, thereby improving the model's detection speed.
[0045] In this embodiment, after an RGB-D image is input, a multi-scale feature extraction network based on depthwise separable convolution outputs initial feature maps at different spatial scales through a series of depthwise separable convolution and downsampling operations. These feature maps capture information at different levels.
[0046] For the multi-scale features output by the multi-scale feature extraction network based on depthwise separable convolution, this embodiment constructs an iterative refined pyramid module. This module fuses the multi-scale features extracted by the feature extraction network through a cyclic iterative process. In each iteration, the iterative refined pyramid module performs a complete top-down and bottom-up bidirectional information transfer.
[0047] Top-down approach: The highest-level semantic features are upsampled and fused with the features of the next level, and strong semantic information is passed to shallower features that pay more attention to details;
[0048] Bottom-up approach: After top-down fusion, shallow features rich in precise spatial information are downsampled and fused with features from the previous level, thereby using precise positioning information to correct the semantic representation of the higher level.
[0049] This process is repeated iteratively until the preset number of iterations is reached, at which point the iteration terminates and a multi-scale optimized feature map is output. The input for the first iteration is the multi-scale features extracted by the feature extraction network, and the input for each subsequent iteration is the multi-scale feature map output from the previous iteration.
[0050] Compared to traditional high-precision pyramid modules, the iterative refined pyramid module in this embodiment has a simpler structure and shorter inference time per iteration. By reasonably arranging the number of iterations, detection accuracy can be improved through multiple iterations while balancing inference time. This embodiment overcomes the shortcomings of traditional pyramid modules' one-way, one-time fusion by achieving deep and progressive fusion of semantic and spatial information through multiple bidirectional information exchanges. Ultimately, high-level features contain accurate spatial information from the shallow layers, resulting in more accurate localization; while shallow features are infused with rich high-level semantic information. This improves model performance, enabling the recognition of fine geometric details such as edge contours, which are crucial for grasping decisions, and ultimately accurately generating the grasping pose.
[0051] The prediction head in this embodiment includes two 3×3 convolutional layers, which decode the multi-scale optimized feature maps output by the iteratively refined pyramid module into the final pixel-level grasping prediction map, including a grasping quality map, a grasping angle map, and a grasping width map.
[0052] In this embodiment, the improved CNN model enhances detection speed through a multi-scale feature extraction network based on depthwise separable convolutions, and further improves detection accuracy through an iteratively refined pyramid module. Ultimately, detection speed is maintained while detection accuracy is improved.
[0053] In addition, this embodiment uses the SmoothL1 loss function during model training, defined as:
[0054] ;
[0055] Among them, threshold 1, any pixel The predicted value is The true value is .
[0056] Simultaneously, the Adam optimizer was used to train the randomly initialized model. During training, the constructed training data samples were input into the model in batches of 32. Over a total of 60 training epochs, iterative optimization was performed using backpropagation and the SmoothL1 loss function. The initial learning rate was set to 0.001 and decayed at the 40th epoch, ultimately retaining the optimal model parameters.
[0057] Finally, referring to step S4, this embodiment imports the test dataset into the trained optimal model to verify the effectiveness and evaluate the performance of robot grasping.
[0058] In summary, this embodiment can perform high-precision, real-time detection of key parameters such as the quality, angle, and width of the robot's grasp, significantly improving the robot's grasping success rate and operational efficiency. Furthermore, this embodiment does not rely on a specific robot hardware platform or gripper type, possessing good versatility and can be widely applied to various automation scenarios such as intelligent manufacturing and logistics sorting.
[0059] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution; or the direct application of the inventive concept and technical solution to other situations without modification, are all within the protection scope of the present invention.
Claims
1. A real-time robot grasping and detection method based on an improved CNN, characterized in that: The method includes the following steps: Step S1: Image data acquisition and preprocessing; Step S2: Based on the preprocessed image data, construct training dataset samples and test dataset samples; Step S3: Construct an improved CNN model, randomly initialize the model parameters, input the training dataset, train the improved CNN model using the gradient descent algorithm, and fix the optimal model parameters; the improved CNN model includes a multi-scale feature extraction network based on depthwise separable convolution, an iterative refined pyramid module, and a prediction head; Step S4: Import the test dataset into the trained optimal model to verify the effectiveness and evaluate the performance of the robot's grasping.
2. The robot real-time grasping and detection method based on improved CNN according to claim 1, characterized in that: In step S1, an RGB-D camera is used to acquire visual information of the scene; for each scene to be detected, the RGB-D camera simultaneously acquires an RGB color image and a corresponding depth image.
3. The robot real-time grasping and detection method based on improved CNN according to claim 2, characterized in that: The preprocessing in step S1 includes: adjusting the resolution of the acquired original RGB image and depth image to the same pixel size; and fusing the two modal images after resolution unification, that is, stitching the 3-channel RGB image and the 1-channel depth image together in the channel dimension to form a four-channel input tensor as the preprocessed image data.
4. The robot real-time grasping and detection method based on improved CNN according to claim 1, characterized in that: Step S2 includes: dividing the preprocessed image data into training dataset samples and test dataset samples according to a preset ratio; wherein, each sample is assigned a corresponding grasping label; and finally, each sample and its corresponding grasping label are converted into a format that the neural network can directly learn.
5. A real-time robot grasping and detection method based on an improved CNN according to claim 4, characterized in that: Converting each sample and its corresponding grabbing label into a format that the neural network can directly learn involves encoding each grabbing label into four pixel-level ground truth maps of the same size as the input image: a grabbing quality map, a cosine component map of the grabbing angle, a sine component map, and a grabbing width map. These ground truth maps will serve as supervision signals during model training, guiding the model to learn the mapping relationship from the input image to the desired output.
6. A real-time robot grasping and detection method based on an improved CNN according to claim 4 or 5, characterized in that: Step S2 further includes performing data augmentation operations on the data in the training dataset samples.
7. The robot real-time grasping and detection method based on improved CNN according to claim 1, characterized in that: In step S3, the multi-scale feature extraction network based on depthwise separable convolution uses CNN as the feature extraction network, and on this basis, the standard convolution of CNN is replaced with depthwise separable convolution.
8. The robot real-time grasping and detection method based on improved CNN according to claim 1, characterized in that: In step S3, the iterative refined pyramid module fuses the multi-scale features extracted by the feature extraction network through a cyclic iterative process. In each iteration, the iterative refined pyramid module performs a complete top-down and bottom-up bidirectional information transfer: Top-down approach: The highest-level semantic features are upsampled and fused with the features of the next level, and strong semantic information is passed to shallower features that pay more attention to details; Bottom-up approach: After top-down fusion, shallow features rich in precise spatial information are downsampled and fused with features from the previous level, thereby using precise positioning information to correct the semantic representation of the higher level. This process is repeated iteratively until the preset number of iterations is reached, at which point the iteration terminates and a multi-scale optimized feature map is output. The input for the first iteration is the multi-scale features extracted by the feature extraction network, and the input for each subsequent iteration is the multi-scale feature map output from the previous iteration.
9. A real-time robot grasping and detection method based on an improved CNN according to claim 1, characterized in that: In step S3, the prediction head includes two 3×3 convolutional layers for decoding the multi-scale optimized feature maps output by the iteratively refined pyramid module into a final pixel-level grasping prediction map, including a grasping quality map, a grasping angle map, and a grasping width map.
10. A real-time robot grasping and detection method based on an improved CNN according to claim 1, characterized in that: In step S3, the model training uses the SmoothL1 loss function, defined as: ； Among them, threshold 1, any pixel The predicted value is The true value is .