A visual grasping detection method based on a convolutional neural network

Abstract

The application discloses a visual grasping detection method based on a convolutional neural network, and comprises the following steps: acquiring an RGB picture and a depth picture of an object to be grasped; performing preprocessing on the input RGB picture and the depth picture to obtain three-dimensional space coordinates of each pixel point of the RGB picture; removing the background of the RGB picture; performing feature extraction and feature fusion on the RGB picture with the background removed; performing target detection and grasping detection operations on the fused features based on a target detection network and a grasping detection network respectively to obtain target detection features and grasping detection features; performing alignment on the target detection features and the grasping detection features to obtain joint correlation of target detection results and grasping detection results; and screening final grasping detection results according to the joint correlation, a set final prediction quantity and a threshold of grasping quality in grasping parameters. The method can generate more grasping candidate objects, and improve the interpretability of model prediction results.

Description

Technical Field

[0001] This invention belongs to the field of grasping detection technology, and relates to a visual grasping detection method based on convolutional neural networks. Background Technology

[0002] Existing grasping detection methods can be broadly categorized into three types: methods that directly output grasping candidate objects without identifying the target object; methods that first identify the target object and then detect the grasping object; and methods that simultaneously identify the target object and detect the grasping object. The method of directly outputting grasping candidate objects without identifying the target object makes obtaining the grasping angle more convenient, but it reduces the accuracy of the grasping angle. Furthermore, this method is unsuitable for grasping multiple objects in cluttered scenes. The methods of first identifying the target object and then detecting the grasping object, and simultaneously identifying the target object and detecting the grasping object, decompose the problem into a grasping detection problem involving object detection. The difference is that the former method performs object detection first and then uses the result as input for grasping detection, while the latter method performs both object detection and grasping detection simultaneously and integrates the results. It establishes the relationship between the bounding box and the grasping rectangle by calculating the intersection-union ratio (IURR) of the two regions. If the IURR is greater than a set threshold, the detected object is grasped.

[0003] In existing methods, if the grabbable area is much smaller than the entire object, the intersection-over-union (IoU) between the object's grab rectangle and bounding box becomes too small. Therefore, this method filters out some potential grabbable objects. Summary of the Invention

[0004] To address the issue that the intersection-over-union ratio (IoU) between the grab rectangle and the bounding box is too small when the grabbable area is much smaller than the entire object, this invention provides the following technical solution:

[0005] A visual grasping detection method based on convolutional neural networks includes the following steps:

[0006] Obtain the RGB image and depth image of the object to be crawled;

[0007] The input RGB and depth images are preprocessed to obtain the three-dimensional spatial coordinates of each pixel in the RGB image;

[0008] Remove the background from an RGB image;

[0009] The background-removed RGB image is subjected to feature extraction and feature fusion.

[0010] The fused features are then used to perform target detection and crawling detection operations on the target detection network and crawling detection network respectively to obtain target detection features and crawling detection features.

[0011] Align the target detection features with the crawling detection features to obtain the joint correlation between the target detection results and the crawling detection results;

[0012] The final crawl detection results are selected based on the joint correlation, the set number of final predictions, and the crawl quality threshold in the crawl parameters.

[0013] Furthermore, the preprocessing of the input RGB image and depth image yields the three-dimensional spatial coordinates of each pixel in the RGB image. Using a pre-defined fixed camera and depth information from each point in the depth image, the three-dimensional coordinates of each pixel in the RGB image are calculated. The formula for calculating the three-dimensional coordinates of each pixel using camera parameters and depth information is as follows:

[0014]

[0015]

[0016]

[0017] Where (u, v) represent pixel coordinates. Represents a three-dimensional coordinate point. Represents the camera's focal length. Represents pixel coordinates in the pixel coordinate system coordinate axes and Scaling factor on the coordinate axis This represents the offset of the camera's optical axis from the origin in the image coordinate system to the origin in the pixel coordinate system. Represents depth value.

[0018] Furthermore, the RANSAC-based planar fitting method for removing the background of the RGB image's environment detects the largest three-dimensional plane in the scene where the target is located, and then assigns the three channel values ​​of the RGB image pixels corresponding to the three-dimensional coordinate points within and below the three-dimensional plane to 255, thereby removing the background of the detected target's RGB image.

[0019] Furthermore, the RANSAC-based planar fitting method for removing the background of the RGB image's environment detects the largest 3D plane in the scene where the target is located.

[0020] S31: Randomly select three points in the initial three-dimensional coordinate system and calculate their corresponding plane equations:

[0021]

[0022] S32: Calculate the algebraic distance from all points to the plane. And select a threshold. If the point If a point is not an interior point of the plane fitting model, it is considered an exterior point; otherwise, it is considered an exterior point. The formula for calculating the algebraic distance is:

[0023]

[0024] S33: Repeat step S32 to select the best fitting parameters, which are the plane fitting model parameters corresponding to the plane with the most interior points. At the end of each iteration, an iteration end evaluation factor is calculated based on the expected error rate, the best number of interior points, the total number of samples, and the current iteration number. The iteration end evaluation factor determines whether to stop the iteration. After the iteration ends, the best model parameters are the final parameter estimates.

[0025] Furthermore, the process of extracting and fusing features from the background-removed RGB image is as follows:

[0026] The background-removed RGB image is input into ResNet for feature extraction and fusion. The final feature map learned through different stages is used as multi-scale features, which are represented as follows: These feature maps have strides of {2, 4, 8, 16, 32} relative to the original image, and are used... Perform feature fusion, and denote the fused feature map as follows: .

[0027] Furthermore, the process of simultaneously performing target detection and grasping detection operations on the fused features to obtain target detection features and grasping detection features is as follows:

[0028] First, the fused features are input into stacked convolutional blocks to obtain task-relevant feature representations from texture and semantic information.

[0029] Then, a 3×3 convolutional block is used to fuse the task-related feature representations from the previous layer and the current layer to obtain the object detection task-related features. Features related to the capture and detection task ;

[0030] Finally, a 1×1 convolutional layer is used to match the number of channels to the final prediction, and the object detection operation outputs a... The output of the channel, where This represents the number of bounding boxes. It represents the number of target categories; the capture and detection operation will ultimately output a... The output of the channel, where This is the predicted number of grab rectangles. Representing the number of rotation angle boxes, the target detection head output is expressed as... The output of the capture detection head is represented as .

[0031] Furthermore, the process of aligning the target detection features and the grasping detection features to obtain the joint correlation between the target detection results and the grasping detection results is as follows:

[0032] Task-related features output by the object detection operation Task-related features of the capture and detection operation output ,

[0033] First, the two features will be passed through a 1×1 convolutional layer to obtain the feature... and characteristics The output of .

[0034] The two feature maps are then reshaped into two-dimensional matrices, and the output correlation matrix is ​​generated using transpose matrix multiplication. ,

[0035] Finally, a sigmoid activation function is used to obtain the joint probability of the capture detection results and the object detection results.

[0036] Furthermore, the process of obtaining the final crawl detection results based on the joint correlation, the set number of final predictions, and the crawl quality threshold in the crawling parameters is as follows:

[0037] Based on the joint probability of the capture detection results and the target detection results, and two set parameters , And the final prediction results are filtered out by capturing a quality threshold, where Indicates selection The highest predicted value of the target object indivual, This indicates that each target object has at most [number of objects]. The crawling prediction process then filters out crawling detection results that are below a set crawling quality threshold. The final output is a tuple containing parameters (...). ),in This represents the center pixel coordinates of the target detection bounding box. This represents the width and height of the target detection bounding box. Indicates the category of the detected object. This indicates the center coordinates of the rectangle being grabbed. This represents the width and height of the rectangle being grabbed. Expressed as rotation angle, This represents the success rate of the capture.

[0038] Furthermore, the target detection network and the grasping detection network have the same structure;

[0039] The target detection network consists of a stacked convolutional block, a 3×3 convolutional block, and a 1×1 convolutional layer.

[0040] A convolutional block consists of a convolutional layer, a normalization layer, and a Leaky ReLU activation function. A 3×3 convolutional block means that the convolutional layers within the block are 3×3 convolutional layers.

[0041] The stacked convolutional block consists of five convolutional blocks arranged in sequence: a 1×1 convolutional block, a 3×3 convolutional block, a 1×1 convolutional block, a 3×3 convolutional block, and a 1×1 convolutional block. The rotation angle box transforms the rotation angle range of the grasping rectangle from [0, π] into... Anchor points with rotation angles.

[0042] A visual grasping detection device based on convolutional neural networks, including

[0043] Acquisition module: Used to acquire the RGB image and depth image of the object to be captured;

[0044] The preprocessing module is used to preprocess the input RGB image and depth image to obtain the three-dimensional spatial coordinates of each pixel in the RGB image;

[0045] Removal module: Used to remove the background from RGB images;

[0046] Feature extraction and feature fusion: Used to extract and fuse features from RGB images with the background removed;

[0047] The target detection feature module is used to simultaneously perform target detection based on the target detection network using the fused features to obtain target detection features.

[0048] The feature extraction and detection module is used to perform simultaneous crawling and detection operations on the fused features based on the crawling and detection network to obtain crawling and detection features.

[0049] Alignment module: This module aligns the features detected by the target with the features detected by the grasping process to obtain the joint correlation between the target detection results and the grasping detection results.

[0050] The crawling module is used to filter out the final crawling detection results based on joint correlation, the set number of final predictions, and the crawling quality threshold in the crawling parameters.

[0051] This invention provides a novel visual grasping detection method based on convolutional neural networks, a method for robot grasping detection that simultaneously performs object detection and grasping detection from RGB and depth images, generating more grasping candidates. It designs two independent detection branches for object and grasping detection and models the correspondence between identified target objects and grasping candidates through an alignment module. Furthermore, it proposes a depth-based method to filter out background in cluttered scenes, allowing our object detection head and grasping detection head to focus on the features of the target object rather than textures from the environment, thereby improving prediction accuracy. Extensive experimental studies on real-world robot grasping detection demonstrate that our method outperforms existing grasping detection methods in terms of accuracy and interpretability. This application designs a module that uses depth images to filter out background, removing useless information and improving grasping accuracy in cluttered scenes; simultaneously, it designs an alignment module to model the correspondence between identified objects and grasping candidates, enabling our grasping detection method to generate more grasping candidates and improving the interpretability of the model's prediction results. Finally, we conducted real-world deployments and experiments, which showed that our method significantly outperformed other methods in terms of accuracy and interpretability of prediction results. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the method of the present invention;

[0054] Figure 2 A flowchart illustrating the background processing of this invention;

[0055] Figure 3 To capture the structural diagram of the detection head;

[0056] Figure 4 This is a flowchart of the experiment of the present invention;

[0057] Figure 5 (a) The image to be removed from the background, (b) The effect of removing the background according to the present invention;

[0058] Figure 6(a) is a comparison of image-level accuracy on the Cornell dataset; (b) is a comparison of object-level accuracy on the Cornell dataset; (c) is a comparison of frames processed per second on the Cornell dataset; and (d) is a comparison of accuracy on the Jacquard dataset.

[0059] Figure 7 (a) is the experimental results of the Cornell dataset and Jacquard dataset for image I; (b) is the experimental results of the Cornell dataset and Jacquard dataset for image II; (c) is the experimental results of the Cornell dataset and Jacquard dataset for image III; and (d) is the experimental results of the Cornell dataset and Jacquard dataset for image IV. Detailed Implementation

[0060] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0063] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0064] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0065] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0066] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0067] Figure 1This is a flowchart of the method of the present invention;

[0068] A deep learning-based robot grasping and detection method can be broken down into the following steps:

[0069] S1: Retrieve the RGB image and depth image of the object to be captured;

[0070] S2: Preprocess the input RGB image and depth image to obtain the three-dimensional spatial coordinates of each pixel in the RGB image;

[0071] S3: Remove the background from an RGB image;

[0072] S4: Extract and fuse features from the RGB image after removing the background;

[0073] S5: Simultaneously perform target detection and crawling detection operations on the target detection network and crawling detection network respectively based on the fused features to obtain target detection features and crawling detection features;

[0074] S6: Align the target detection features with the grasping detection features to obtain the joint correlation between the target detection results and the grasping detection results;

[0075] S7: Based on the joint correlation, the set number of final predictions, and the threshold for crawling quality in the crawling parameters, the final crawling detection results are selected.

[0076] Steps S1 / S2 / S3 / S4 / S5 / S6 / S7 are executed sequentially;

[0077] The preprocessing of the input RGB and depth images yields the 3D spatial coordinates of each pixel in the RGB image. Using a pre-defined fixed camera and depth information from each point in the depth image, the 3D coordinates of each pixel in the RGB image are calculated. The formula for calculating the 3D coordinates of each pixel using camera parameters and depth information is as follows:

[0078]

[0079]

[0080]

[0081] Where (u, v) represent pixel coordinates. Represents a three-dimensional coordinate point. Represents the camera's focal length. Represents pixel coordinates in the pixel coordinate system coordinate axes and Scaling factor on the coordinate axis This represents the offset of the camera's optical axis from the origin in the image coordinate system to the origin in the pixel coordinate system. Represents depth value.

[0082] Figure 2 A flowchart illustrating the background processing of this invention;

[0083] Furthermore: The RANSAC-based planar fitting method for removing the background of the RGB image's environment detects the largest three-dimensional plane in the scene where the target is located, and then assigns the three channel values ​​of the RGB image pixels corresponding to the three-dimensional coordinate points within and below the three-dimensional plane to 255, thereby removing the background of the detected target's RGB image.

[0084] Furthermore: the RANSAC-based planar fitting method for removing the background of the RGB image's environment, the process of detecting the largest 3D plane in the scene where the target is located is as follows:

[0085] S31: Randomly select three points in the initial three-dimensional coordinate system and calculate their corresponding plane equations:

[0086]

[0087] S32: Calculate the algebraic distance from all points to the plane. And select a threshold. If the point If a point is not an interior point of the plane fitting model, it is considered an exterior point; otherwise, it is considered an exterior point. The formula for calculating the algebraic distance is:

[0088]

[0089] S33: Repeat step S32 to select the best fitting parameters (A, B, C, D), which are the plane fitting model parameters corresponding to the plane with the most interior points. At the end of each iteration, an iteration end evaluation factor is calculated based on the expected error rate, the best number of interior points, the total number of samples, and the current iteration number. The iteration end evaluation factor determines whether to stop the iteration. After the iteration ends, the best model parameters are the final parameter estimates.

[0090] The process of extracting and fusing features from the background-removed RGB image is as follows:

[0091] The background-removed RGB image is input into ResNet for feature extraction and feature fusion. The last feature map learned through different stages will be used as multi-scale features. These scale feature maps are denoted as and their strides relative to the original image are {2,4,8,16,32}. Feature fusion is performed using , and the fused feature map is denoted as .

[0092] Figure 3 To capture the structural diagram of the detection head;

[0093] The process of simultaneously performing target detection and crawling detection operations on the fused features to obtain target detection features and crawling detection features is as follows:

[0094] First, the fused features are input into stacked convolutional blocks to obtain task-relevant feature representations from texture and semantic information.

[0095] Then, a 3×3 convolutional block is used to fuse the task-related feature representations from the previous layer and the current layer to obtain the object detection task-related features. Features related to the capture and detection task ;

[0096] Finally, a 1×1 convolutional layer is used to match the number of channels to the final prediction, and the object detection operation outputs a... The output of the channel, where This represents the number of bounding boxes. It represents the number of target categories; the capture and detection operation will ultimately output a... The output of the channel, where This is the predicted number of grab rectangles. Representing the number of rotation angle boxes, the target detection head output is expressed as... The output of the capture detection head is represented as .

[0097] The process of aligning the target detection features and the grasping detection features to obtain the joint correlation between the target detection results and the grasping detection results is as follows:

[0098] The task-related features output by the object detection operation and the task-related features output by the grasping detection operation are combined.

[0099] First, the two features are passed through a 1×1 convolutional layer to obtain the outputs of feature and feature .

[0100] The two feature maps are then reshaped into two-dimensional matrices, and the output correlation matrix is ​​generated using transpose matrix multiplication.

[0101] Finally, a sigmoid activation function is used to obtain the joint probability of the capture detection results and the object detection results.

[0102] The process of obtaining the target to be crawled based on joint correlation, the set number of final predictions, and the crawling quality threshold in the crawling parameters is as follows:

[0103] Based on the joint probability of the capture detection results and the target detection results, and two set parameters , And the final prediction results are filtered out by capturing a quality threshold, where Indicates selection The highest predicted value of the target object indivual, This indicates that each target object has at most [number of objects]. The crawling prediction process then filters out crawling detection results with crawling quality below a set threshold (crawling quality is equivalent to probability, and the threshold is used to filter out results with low probability). The final output is a tuple containing parameters (...). ),in This represents the center pixel coordinates of the target detection bounding box. This represents the width and height of the target detection bounding box. Indicates the category of the detected object. This indicates the center coordinates of the rectangle being grabbed. This represents the width and height of the rectangle being grabbed. Expressed as rotation angle, This represents the success rate of the capture.

[0104] The target detection network and the grasping detection network have the same structure; although they have the same structure, they perform different functions, so the network parameters updated during training are different, and therefore the output features are different.

[0105] The target detection network consists of a stacked convolutional block, a 3×3 convolutional block, and a 1×1 convolutional layer.

[0106] A convolutional block consists of a convolutional layer, a normalization layer, and a Leaky ReLU activation function. A 3×3 convolutional block means that the convolutional layers within the block are 3×3 convolutional layers.

[0107] The stacked convolutional block consists of five convolutional blocks arranged in sequence: a 1×1 convolutional block, a 3×3 convolutional block, a 1×1 convolutional block, a 3×3 convolutional block, and a 1×1 convolutional block. The rotation angle box transforms the rotation angle range of the grasping rectangle from [0, π] into... Anchor points with rotation angles.

[0108] The process of simultaneously performing target detection and crawling detection operations on the fused features based on the target detection network and the crawling detection network respectively to obtain target detection features and crawling detection features can be abstracted into the following formula:

[0109]

[0110]

[0111]

[0112]

[0113] The two features output from the convolutional blocks of the object detection network and the grasping detection network are passed through a 1×1 convolutional layer to obtain the predicted features we need. Then, the predicted features are reshaped into a two-dimensional matrix and multiplied by the transpose matrix to obtain the association matrix between the target object and the grasping configuration. Finally, the sigmoid function is used to transform the association matrix into a joint probability in the range [0,1].

[0114] A visual grasping detection device based on convolutional neural networks, including

[0115] Acquisition module: Used to acquire the RGB image and depth image of the object to be captured;

[0116] The preprocessing module is used to preprocess the input RGB image and depth image to obtain the three-dimensional spatial coordinates of each pixel in the RGB image;

[0117] Removal module: Used to remove the background from RGB images;

[0118] Feature extraction and feature fusion: Used to extract and fuse features from RGB images with the background removed;

[0119] The target detection feature module is used to simultaneously perform target detection based on the target detection network using the fused features to obtain target detection features.

[0120] The feature extraction and detection module is used to perform simultaneous crawling and detection operations on the fused features based on the crawling and detection network to obtain crawling and detection features.

[0121] Alignment module: This module aligns the features detected by the target with the features detected by the grasping process to obtain the joint correlation between the target detection results and the grasping detection results.

[0122] The crawling module is used to filter out the final crawling detection results based on joint correlation, the set number of final predictions, and the crawling quality threshold in the crawling parameters.

[0123] This embodiment uses a server with a GTX 3090 graphics card as the test environment for the method of this invention. All experiments and model training are performed on the server. Specific hardware configuration information is shown in Table 1.

[0124] Table 1 Hardware Configuration

[0125]

[0126] This embodiment uses PyCharm as the development environment and Python as the programming language to complete the method design and development. The software environment for running the method in this embodiment includes: operating system Ubuntu 18.04 and Python 3.8. The specific software environment is shown in Table 2.

[0127] Table 2 Software Environment

[0128]

[0129] Table 3 Statistical information of the experimental dataset

[0130]

[0131] In this embodiment, the crawling detection experiment is implemented on the server. The verification process of this method is as follows: Figure 4 The main steps shown are as follows:

[0132] Step 1: Data preprocessing. Read the RGB image and depth image. Using a pre-set fixed camera and the depth information of each point in the depth image, calculate the three-dimensional coordinates of each pixel in the RGB image.

[0133] Step 2: Remove the image background, use the RANSAC-based planar fitting method to detect the largest 3D plane in the scene, and then assign the R, G, and B channel values ​​of the RGB image pixels corresponding to the 3D coordinate points that are in or below the 3D plane to 255.

[0134] Step 3: Training and selection of the model in the method of this invention. The dataset is divided into training dataset and test dataset. Training includes multiple rounds of iteration. In each round of iteration, the training dataset is traversed once, and a small batch of samples is taken from it each time. The samples are fed into the model to perform forward computation to obtain the detection prediction value. The loss function value between the detection result and the true value is calculated. Gradient backpropagation is performed, and the model parameters are updated. The model is selected based on the loss function value curve and the detection prediction accuracy curve.

[0135] Step 4: Result evaluation. Input the test dataset into the trained model to obtain the capture prediction results. Calculate the Jaccard Index value between the prediction results and the true values. If the Jaccard Index value is greater than 0.25, the prediction is considered correct, and the accuracy rate is calculated.

[0136] This embodiment uses two datasets to verify the effectiveness of the invention: 1) Cornell, which contains 878 images along with corresponding depth images and 3D point clouds. The image resolution is 640×480, and each image contains a graspable object at a different location and orientation. 2) Jacquard, which was collected from a simulator with a ShapeNet dataset CAD model. The dataset contains 54,485 images and 11,619 different objects. The large number of samples is beneficial for our model training, but we still use some data augmentation strategies to increase the robustness of the learned model. The image resolution in this dataset is 1024×1024. We reduce the original images to 512×512 for training and testing.

[0137] Evaluation Metrics: This paper uses accuracy (Acc) to evaluate the precision of the capture and detection results. Accuracy refers to the proportion of all correctly judged results out of the total observed samples. A Jaccard Index greater than 0.25 is defined as a correct result. The calculation formulas are as follows:

[0138]

[0139]

[0140] in, To predict the correct true cases, To predict false positives, FN To predict false counterexamples of errors, TN To predict correct true counterexamples, For the predicted grab rectangle, To capture the true value of the rectangle.

[0141] After training and testing, the model was tested on the Cornell and Jacquard datasets, and the actual results are as follows: Figure 7 As shown.

[0142] Comparison Method: This embodiment compares our solution with other solutions utilizing two datasets. Specific experimental results are as follows: Figure 6 As shown, Figure 6 (a) is a comparison of image-level accuracy on the Cornell dataset; (b) is a comparison of object-level accuracy on the Cornell dataset; (c) is a comparison of frames processed per second on the Cornell dataset; and (d) is a comparison of accuracy on the Jacquard dataset.

[0143] Experimental Results: Sample results were obtained from experiments on the Cornell and Jacquard datasets. Figure 7 As shown, Figure 7 (a) is the experimental results of the Cornell dataset and Jacquard dataset for image I; (b) is the experimental results of the Cornell dataset and Jacquard dataset for image II; (c) is the experimental results of the Cornell dataset and Jacquard dataset for image III; and (d) is the experimental results of the Cornell dataset and Jacquard dataset for image IV.

[0144] Figure 5 (a) The image to be removed from the background, and (b) The effect of removing the background according to the present invention, which shows the effectiveness of the background removal module of the present invention.

[0145] Finally, it should be noted that the above descriptions are merely embodiments of the present invention and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A visual grasping and detection method based on convolutional neural networks, characterized in that: Includes the following steps: Obtain the RGB image and depth image of the object to be crawled; The input RGB and depth images are preprocessed to obtain the three-dimensional spatial coordinates of each pixel in the RGB image; Remove the background from an RGB image; The background-removed RGB image is subjected to feature extraction and feature fusion. The fused features are then used to perform target detection and crawling detection operations on the target detection network and crawling detection network respectively to obtain target detection features and crawling detection features. The process of simultaneously performing target detection and crawling detection operations on the fused features to obtain target detection features and crawling detection features is as follows: First, the fused features are input into stacked convolutional blocks to obtain task-relevant feature representations from texture and semantic information. Then, a 3×3 convolutional block is used to fuse the task-related feature representations from the previous layer and the current layer to obtain the object detection task-related features. Features related to the capture and detection task ; Finally, a 1×1 convolutional layer is used to match the number of channels to the final prediction, and the object detection operation outputs a... The output of the channel, where This represents the number of bounding boxes. It represents the number of target categories; the capture and detection operation will ultimately output a... The output of the channel, where This is the predicted number of grab rectangles. Representing the number of rotation angle boxes, the target detection head output is expressed as... The output of the capture detection head is represented as ; Align the target detection features with the crawling detection features to obtain the joint correlation between the target detection results and the crawling detection results; The process of aligning the target detection features and the grasping detection features to obtain the joint correlation between the target detection results and the grasping detection results is as follows: Task-related features output by the object detection operation Task-related features of the capture and detection operation output , First, the two features will be passed through a 1×1 convolutional layer to obtain the feature... and characteristics The output of . The two feature maps are then reshaped into two-dimensional matrices, and the output correlation matrix is ​​generated using transpose matrix multiplication. , Finally, a sigmoid activation function is used to obtain the joint probability of the grab detection results and the target detection results; Based on the joint correlation, the set number of final predictions, and the threshold of crawling quality in the crawling parameters, the final crawling detection results are selected. The process of obtaining the final crawl detection results based on joint correlation, the set number of final predictions, and the crawl quality threshold in the crawling parameters is as follows: Based on the joint probability of the capture detection results and the target detection results, and two set parameters , And the final crawl detection results are filtered out using a crawl quality threshold, among which... Indicates selection The highest predicted value of the target object indivual, This indicates that each target object has at most [number of objects]. Capture and predict. Based on the set grasping quality threshold, grasping detection results with grasping quality below the threshold are filtered out. The final output format is a set of tuples, and the parameters in the tuples are ( ),in This represents the center pixel coordinates of the target detection bounding box. This represents the width and height of the target detection bounding box. Indicates the category of the detected object. This indicates the center coordinates of the rectangle being grabbed. This represents the width and height of the rectangle being grabbed. Expressed as rotation angle, This represents the success rate of the capture. The preprocessing of the input RGB and depth images yields the 3D spatial coordinates of each pixel in the RGB image. Using a pre-defined fixed camera and depth information from each point in the depth image, the 3D coordinates of each pixel in the RGB image are calculated. The formula for calculating the 3D coordinates of each pixel using camera parameters and depth information is as follows: Where (u, v) represent pixel coordinates. Represents a three-dimensional coordinate point. Represents the camera's focal length. Represents pixel coordinates in the pixel coordinate system coordinate axes and Scaling factor on the coordinate axis This represents the offset of the camera's optical axis from the origin in the image coordinate system to the origin in the pixel coordinate system. Represents depth value; The RANSAC-based planar fitting method for removing the background of an RGB image detects the largest 3D plane in the scene containing the target as follows: S31: Randomly select three points in the initial three-dimensional coordinate system and calculate their corresponding plane equations: S32: Calculate the algebraic distance from all points to the plane. And select a threshold. If the point If a point is not an interior point of the plane fitting model, it is considered an exterior point; otherwise, it is considered an exterior point. The formula for calculating the algebraic distance is: S33: Repeat step S32 to select the best fitting parameters, i.e., the plane fitting model parameters corresponding to the plane with the most interior points; at the end of each iteration, an iteration end evaluation factor is calculated based on the expected error rate, the best number of interior points, the total number of samples, and the current iteration number. The iteration end evaluation factor determines whether to stop the iteration. After the iteration ends, the best model parameters are the final parameter estimates; the target detection network and the grasping detection network have the same structure. The target detection network consists of a stacked convolutional block, a 3×3 convolutional block, and a 1×1 convolutional layer. A convolutional block consists of a convolutional layer, a normalization layer, and a Leaky ReLU activation function. A 3×3 convolutional block means that the convolutional layers within the block are 3×3 convolutional layers. The stacked convolutional block consists of five convolutional blocks arranged in sequence: a 1×1 convolutional block, a 3×3 convolutional block, a 1×1 convolutional block, a 3×3 convolutional block, and a 1×1 convolutional block. The rotation angle box transforms the rotation angle range of the grasping rectangle from [0, π] into... Anchor points with rotation angles.

2. The visual grasping and detection method based on convolutional neural networks according to claim 1, characterized in that: The RANSAC-based planar fitting method for removing the background of the RGB image detects the largest three-dimensional plane in the scene where the target is located, and then assigns the three channel values ​​of the RGB image pixels corresponding to the three-dimensional coordinate points that are in or below the three-dimensional plane to 255, thereby removing the background of the RGB image of the detected target.

3. The visual grasping and detection method based on convolutional neural networks according to claim 1, characterized in that: The process of extracting and fusing features from the background-removed RGB image is as follows: The background-removed RGB image is input into ResNet for feature extraction and fusion. The final feature map learned through different stages is used as multi-scale features, which are represented as follows: These feature maps have strides of {2, 4, 8, 16, 32} relative to the original image, and are used... Perform feature fusion, and denote the fused feature map as follows: .

4. The apparatus for a visual grasping and detection method based on a convolutional neural network according to any one of claims 1-3, characterized in that: include: Acquisition module: Used to acquire the RGB image and depth image of the object to be captured; The preprocessing module is used to preprocess the input RGB image and depth image to obtain the three-dimensional spatial coordinates of each pixel in the RGB image; Removal module: Used to remove the background from RGB images; Feature extraction and feature fusion: Used to extract and fuse features from RGB images with the background removed; The target detection feature module is used to simultaneously perform target detection based on the target detection network using the fused features to obtain target detection features. The feature extraction and detection module is used to perform simultaneous crawling and detection operations on the fused features based on the crawling and detection network to obtain crawling and detection features. Alignment module: This module aligns the features detected by the target with the features detected by the grasping process to obtain the joint correlation between the target detection results and the grasping detection results. The crawling module is used to filter out the final crawling detection results based on joint correlation, the set number of final predictions, and the crawling quality threshold in the crawling parameters.

Images