A grasping detection method and robotic arm using grasping confidence characterization

Abstract

This invention discloses a grasping detection method and a robotic arm using grasping confidence representation, comprising the following steps: S1, extracting graspable locations from a label image, and generating images representing grasping quality, angle, and width at each graspable location; S2, performing multi-angle morphological dilation on the grasping quality, angle, and width images; S3, extracting a skeleton from the grasping quality image to obtain a continuous skeleton, and assigning different weights to the pixels in the skeleton; S4, performing weight propagation on the continuous skeleton, setting nearby background pixels as foreground pixels and assigning them attenuated weights; S5, comparing the quality, angle, and width images generated from the label image with the quality, angle, and width images predicted from the GRBD image data of the object corresponding to the label image using a fully convolutional network, applying a loss function, and continuously training the grasping detection network to obtain hyperparameters. This addresses the deficiency of not being able to represent the confidence of graspable regions.

Description

Technical Field

[0001] This invention relates to the field of robot grasping control, and more specifically to a grasping detection method and a robotic arm that uses grasping confidence characterization. Background Technology

[0002] Robots can replace humans in performing simple, repetitive grasping tasks, but they cannot achieve the same freedom, flexibility, and reliability as humans when grasping complex and varied scenarios. Humans can use past experience to judge the weight, center of mass, material, and surface friction of the object to obtain the most comfortable grasping position. However, for robots, acquiring this information about the object and predicting a suitable grasping position presents a huge challenge in terms of computation and real-time performance. Furthermore, analyzing the grasping of unknown objects is not a strength of robots.

[0003] Regarding the representation of grasping regions, existing methods typically provide several discrete directed rectangular regions as the ground truth. However, this representation method has certain limitations: it only provides a partially effective grasping representation, and any region that deviates from the ground truth region is considered an invalid grasping region, which is inconsistent with reality. Even if the predicted grasping region deviates from the ground truth region, it may still be considered a valid grasping representation. Therefore, existing methods that compare the angle deviation and coverage indicators between the predicted and ground truth regions cannot effectively reflect the actual graspable probability of the current region. A grasping representation method with a continuous feature distribution that can reflect all feasible grasping regions of the current object and their grasping probabilities is more beneficial for network prediction training and improves the model's grasping detection accuracy.

[0004] Therefore, there is an urgent need for a method to address the deficiency in existing technologies that do not perform confidence characterization of the grabbable regions generated by directed rectangles in the dataset. Summary of the Invention

[0005] The purpose of this invention is to address the current limitation that it is not possible to perform confidence representation on crawlable regions generated by directed bounding boxes in a dataset. This invention provides a crawl detection method that uses crawl confidence representation to perform crawl confidence representation processing on the labeled data in the dataset and assists the network in crawl detection.

[0006] The technical solution of the present invention is as follows:

[0007] A crawling detection method using crawling confidence representation includes the following steps:

[0008] S1. Extract the grabbable locations from the label image, and generate images representing the grabbable quality, angle, and width at the grabbable locations respectively;

[0009] S2. Perform multi-angle morphological dilation on the captured quality, angle, and width images to eliminate discontinuous regions;

[0010] S3. Extract the skeleton from the captured quality image to obtain a continuous skeleton, and assign different weights to the pixels in the skeleton.

[0011] S4. Perform weight propagation on the continuous skeleton, set the nearest background pixels as foreground pixels and assign them a decaying weight.

[0012] S5. The quality, angle, and width images generated from the label image are compared with the quality, angle, and width images predicted from the GRBD image data of the object corresponding to the label image using a fully convolutional network. The loss function is applied to continuously train the grasping and detection network to obtain hyperparameters.

[0013] Furthermore, S1 specifically includes the following steps:

[0014] S11. Extract the center region of one-third of the matrix box in the label image as the graspable position, and set the graspable position as the foreground pixel with a value of 1 in the quality image;

[0015] S12. Generate angle and width images using the same method, with the foreground pixel values ​​being the angle and width values ​​of the rectangle, respectively.

[0016] Furthermore, S2 specifically includes the following steps:

[0017] S21. Determine the size of the dilation kernel based on the width value in the image at the current foreground pixel.

[0018] S22. Determine the direction of the dilation kernel based on the angle value in the angle image at the current foreground pixel.

[0019] S23. Set the background pixels that intersect with the dilation kernel as foreground pixels and assign them the corresponding values.

[0020] Furthermore, S3 specifically includes the following steps:

[0021] S31. Obtain the skeleton series of the quality image generated by S2;

[0022] S32. Remove burrs from the skeleton series to make the skeleton series smooth;

[0023] S33. When determining the continuity of the generated skeleton series, if two adjacent pixels are 8-connected and the absolute value of the angle difference in the angle image is less than 30°, then the skeleton is continuous; otherwise, the skeleton is divided into two new skeletons, and S3.3 is looped to traverse all skeleton points.

[0024] S34. Assign different capture scores to all skeleton sequences according to a Gaussian distribution, with the score at the center point of the skeleton being 1 and gradually decreasing to 0.5 towards both ends.

[0025] Furthermore, S4 specifically includes the following steps:

[0026] S41. Propagate continuous skeleton sequences using a cross-shaped kernel;

[0027] S42. Set all background pixels that coincide with the endpoints of the cross kernel as foreground pixels, and set their weight values ​​to the corresponding weight values ​​on the cross kernel.

[0028] S43. Iterate multiple times using the newly generated background pixels to generate a weighted image;

[0029] S44. Apply Gaussian smoothing to the generated quality image.

[0030] Furthermore, the specific operation of S41 is as follows: the current pixel point is taken as the center of the cross kernel, and the weight value of the center of the cross kernel is set to the weight value of the current pixel, and the weight values ​​of the endpoints of the cross kernel are assigned according to the attenuation strategy.

[0031] Furthermore, the label image is an image annotated with a directed matrix bounding box.

[0032] The present invention also includes a grasping robot that uses a grasping detection method with grasping confidence characterization to grasp objects.

[0033] Compared with existing technologies, the advantages of this invention are:

[0034] This invention discloses a grasping detection method and robotic arm that uses grasping confidence as a representation. Existing grasping detection methods simply extract grasping position information from rectangular labels based on whether the label can be grasped at its current position, without defining grasping confidence for these positions. Such data is not conducive to the training and prediction of grasping detection networks. Therefore, this invention processes the label data and assigns different confidence levels, which can effectively improve the robustness of the trained model and the grasping detection success rate. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a grasping detection method using grasping confidence characterization and a grasping detection method for a robotic arm. Detailed Implementation

[0036] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0037] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0038] Please see Figure 1 A crawling detection method using crawling confidence representation includes the following steps:

[0039] S1. Extract the grabbable locations from the label image, and generate images representing the grabbable quality, angle, and width at the grabbable locations respectively;

[0040] S1 specifically includes the following steps:

[0041] S11. Extract the center region of one-third of the matrix box in the label image as the graspable position, and set the graspable position as the foreground pixel with a value of 1 in the quality image;

[0042] S12. Generate angle and width images using the same method, with the foreground pixel values ​​being the angle and width values ​​of the rectangle, respectively.

[0043] S2. Perform multi-angle morphological dilation on the captured quality, angle, and width images to eliminate discontinuous regions;

[0044] S2 specifically includes the following steps:

[0045] S21. Determine the size of the dilation kernel based on the width value in the image at the current foreground pixel; the relationship between the width value and the size of the dilation kernel is expressed by the following formula:

[0046]

[0047] Among them, w max and w min k represents the maximum width of the rectangle, 150, and the minimum width, 30, respectively. max and k minThese represent the maximum expansion kernel size of 15 and the minimum expansion kernel size of 5, respectively. k is the expansion kernel size, and w is the width value. The upper bracket on the right side of the equals sign indicates rounding up.

[0048] S22. Determine the direction of the dilation kernel based on the angle value in the angle image at the current foreground pixel; the relationship between the angle value and the direction of the dilation kernel is as follows: Figure 1 As shown at the multi-angle expansion point.

[0049] S23. Set the background pixels that intersect with the dilation kernel as foreground pixels and assign them the corresponding values.

[0050] S3. Extract the skeleton from the captured quality image to obtain a continuous skeleton, and assign different weights to the pixels in the skeleton.

[0051] S3 specifically includes the following steps:

[0052] S31. Obtain the skeleton series of the quality image generated by S2;

[0053] S32. Remove burrs from the skeleton series to make the skeleton series smooth;

[0054] S33. When determining the continuity of the generated skeleton series, if two adjacent pixels are 8-connected and the absolute value of the angle difference in the angle image is less than 30°, then the skeleton is continuous; otherwise, the skeleton is divided into two new skeletons, and S3.3 is looped to traverse all skeleton points.

[0055] S34. Assign different capture scores to all skeleton sequences according to a Gaussian distribution, with the score at the center point of the skeleton being 1 and gradually decreasing to 0.5 towards both ends.

[0056] S4. Perform weight propagation on the continuous skeleton, set the nearest background pixels as foreground pixels and assign them a decaying weight.

[0057] S4 specifically includes the following steps:

[0058] S41. Propagate continuous skeleton sequences using a cross-shaped kernel;

[0059] S42. Set all background pixels that coincide with the endpoints of the cross kernel as foreground pixels, and set their weight values ​​to the corresponding weight values ​​on the cross kernel.

[0060] S43. Iterate multiple times using the newly generated background pixels to generate a weighted image;

[0061] S44. Apply Gaussian smoothing to the generated quality image.

[0062] The specific operation of S41 is as follows: the current pixel point is taken as the center of the cross kernel, and the weight value of the center of the cross kernel is set to the weight value of the current pixel. The weight values ​​of the endpoints of the cross kernel are assigned according to the attenuation strategy.

[0063] S5. The quality, angle, and width images generated from the label image are compared with the quality, angle, and width images predicted from the GRBD image data of the object corresponding to the label image using a fully convolutional network. A loss function is applied to continuously train the grasping detection network to obtain hyperparameters. Hyperparameters are parameters obtained after network training, representing the trained network model. The trained model can predict grasping images and thus determine the grasping pose.

[0064] The label image is an image annotated with a directed matrix bounding box.

[0065] The present invention also includes a grasping robot that uses a grasping detection method with grasping confidence characterization to grasp objects.

[0066] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A grasp detection method using grasp confidence characterization, characterized by, The method comprises the following steps: S1, extracting a graspable position in a label image, and generating a quality image, an angle image and a width image respectively representing the graspable position; S2, performing multi-angle morphological dilation on the quality image, the angle image and the width image to eliminate discontinuous regions; S3, performing skeleton extraction on the quality image to obtain a continuous skeleton, and assigning different weights to pixel points in the skeleton; S4, performing weight propagation on the continuous skeleton, setting a neighboring background pixel point as a foreground pixel point and assigning a decaying weight to the foreground pixel point; S5, performing a loss function on the quality image, the angle image and the width image generated by the label image and a quality image, an angle image and a width image obtained by using a fully convolutional network to predict a corresponding object GRBD image data, constantly training a grasp detection network, and obtaining a hyperparameter; S1 specifically comprises the following steps: S11, extracting a center third region in a matrix frame in a label image as a graspable position, and setting the graspable position as a foreground pixel in a quality image with a value of 1; S12, generating an angle image and a width image in the same way, and setting foreground pixel values as an angle value and a width value of the rectangular frame respectively; The S2 specifically comprises the following steps: S21, determining a dilation kernel size according to a width value in a width image at a current foreground pixel point; S22, determining a dilation kernel direction according to an angle value in an angle image at the current foreground pixel point; S23, setting a background pixel having an intersection with the dilation kernel as a foreground pixel and assigning a corresponding value to the foreground pixel; S3 specifically comprises the following steps: S31, obtaining a skeleton series of the quality image generated in S2; S32, removing burrs of the skeleton series to make the skeleton series smooth; S33, when judging the generated skeleton series, if two adjacent pixel points are eight-connected and an absolute value of a difference between angle values in the angle image is less than 30°, the skeleton is continuous; otherwise, the skeleton is divided into two new skeletons, and S33 is recycled to traverse all skeleton points; S34, assigning different grasp scores to all skeleton series according to a Gaussian distribution, and gradually reducing a score from a center point of the skeleton to both ends to 0.

5.

2. The grasp detection method using grasp confidence representation of claim 1, wherein, The S4 specifically comprises the following steps: S41, performing propagation on the continuous skeleton series using a cross kernel; S42, setting all background pixels coinciding with end points of the cross kernel as foreground pixels, and setting a weight value as a corresponding weight value on the cross kernel; S43, iteratively generating a weight image using newly generated background pixels for multiple times; S44, performing Gaussian smoothing on the generated quality image.

3. The grasp detection method using grasp confidence characterization of claim 2, wherein, The S41 specifically operates as follows: a current pixel point is taken as a center of a cross kernel, a weight value of the center of the cross kernel is set as a weight value of the current pixel, and weight values of end points of the cross kernel are assigned according to a decay strategy.

4. The grasp detection method using grasp confidence characterization of claim 1, wherein, The label image is a directed matrix frame labeled image.

5. A gripping robot, characterized in that The robot arm adopts the grasp detection method using a grasp confidence value as claimed in any one of claims 1 to 4 to grasp an object.

Images