Robot 6d pose estimation method and system based on rgb image

Abstract

The application belongs to the field of robot grabbing, and particularly discloses a robot 6D grabbing pose estimation method and system based on an RGB image, which comprises the following steps: a camera collects an RGB image of a target object, a three-dimensional point cloud model of the object is obtained in advance, and a region heat map and an offset heat map are obtained through a key point heat map prediction model; the key point heat map prediction model uses a Keypoints-RCNN network, which is used to extract a feature map from the RGB image, and then obtain the region heat map and the offset heat map about the key point; a post-processing method is used to binarize the region heat map, and the offset heat map is filtered through the binarized region heat map to obtain the coordinates of the key point relative to the region heat map, and then the pose of the object in a world coordinate system is obtained, so as to determine the grabbing pose of the robot and realize the grabbing of the target object by the robot. The application can greatly improve the grabbing precision of the robot on the target object.

Description

Technical Field

[0001] This invention belongs to the field of robot grasping, and more specifically, relates to a robot 6D grasping posture estimation method and system based on RGB images. Background Technology

[0002] Industrial robots are crucial automation equipment in modern manufacturing, primarily responsible for material handling on production lines. Currently, most material handling robots on production lines are operated through teaching or pre-programming, which leads to poor production line flexibility. Therefore, extensive research has been conducted on robot vision-based grasping. In robot grasping problems, the object's pose relative to the robot is critical, directly determining whether the robot can accurately grasp the target object. Pose estimation can be categorized based on the network input into RGB images and RGBD images. The difference lies in whether depth information is included. Including depth information leads to more accurate estimation, but RGB cameras are cheaper and more widely used; therefore, researching pose estimation methods for RGB images is very important.

[0003] Currently, there are many approaches to object pose estimation. For example, patent CN114119753A describes a 6D pose estimation method for transparent objects grasped by a robotic arm, using RGB, edge, and depth images as inputs, embedding a self-attention mechanism to fuse features and extract key points. Patent CN109215080A proposes a deep learning-based iterative matching method for training a 6D pose estimation network, using RGB images as input and iteratively calculating key points multiple times. Many existing designs utilize key point heatmaps, such as patent CN114842085A, a full-scene vehicle pose estimation method that introduces a Swing Transformer as the backbone network, converting image segmentation results into a position encoding problem. However, the pose estimation results of these methods cannot achieve high accuracy. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a robot 6D grasping posture estimation method and system based on RGB images. Its purpose is to improve the robot's grasping accuracy of target objects by accurately determining the object posture.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a robot 6D grasping pose estimation method based on RGB images is proposed, comprising the following steps:

[0006] The RGB image and 3D point cloud model of the target object are acquired, and a region heatmap and an offset heatmap are obtained through a keypoint heatmap prediction model. The keypoint heatmap prediction model includes a region extraction backbone, a classification prediction branch, a region prediction branch, and an offset prediction branch, wherein:

[0007] The region extraction backbone is used to extract feature maps from RGB images; the classification prediction branch is used to determine the target object category and predict bounding boxes based on the feature maps, and then determine local feature maps based on the bounding boxes; the region prediction branch is used to determine the probability that each pixel belongs to a key point based on long-distance features in the local feature maps, and obtain a region heatmap, wherein the key points are pre-selected in the 3D point cloud model of the object; the offset prediction branch is used to determine the offset coordinates of each pixel relative to the key points based on long-distance features in the local feature maps, and obtain an offset heatmap.

[0008] The region heatmap is binarized, and the offset heatmap is filtered using the binarized region heatmap to obtain the coordinates of the key points relative to the region heatmap, and then the coordinates of the key points relative to the RGB image are determined; based on the coordinates of the key points relative to the RGB image, the pose of the object in the camera coordinate system is obtained; then based on the transformation matrix from the camera coordinate system to the world coordinate system, the pose of the object in the world coordinate system is obtained.

[0009] Based on the object's posture in the world coordinate system, the robot's grasping posture is determined, enabling the robot to grasp the target object.

[0010] As a further preferred method, the region heatmap is binarized using the top-K binarization method:

[0011] Sort the output values ​​of pixels in the region heatmap from largest to smallest, select the top K pixels, set these K pixels to 1, and set the other pixels to 0 to complete the binarization of the region heatmap; the method for determining the value of K is as follows: draw a circle with the pixel with the largest output value in the region heatmap as the center and a preset radius R, and the area of ​​this circle within the region heatmap is K; the output value refers to the probability that the pixel belongs to a key point.

[0012] As a further preferred method, the offset heatmap is filtered using the binarized region heatmap to obtain the coordinates of the key points relative to the region heatmap, including the following steps:

[0013] (1) The region is divided into 3×3 cross-shaped structural units. The corrosion process is carried out, and the remaining area after corrosion is The complement region is denoted as Initially, the iteration number i = 1. The method for determining the initial value of the iteration is as follows: for the binarized region heatmap, the selected pixel region is the initial region.

[0014] (2) According to the region Calculate key point locations using offset heatmaps And calculate the area The area Refer to Draw a circle with a preset radius and center; the area of ​​this circle within the region heatmap.

[0015] (3) Expansion with respect to distance, in Internal selection Set the nearest N points to 1 to obtain the region. in

[0016] (4)Judgment and The degree of overlap is determined, and iteration stops when the degree of overlap meets the criterion, outputting the result at this point. That is, the coordinates of the key point relative to the heatmap of the region; otherwise, let i = i + 1 and return to step (1).

[0017] As a further preferred option, in step (2), the key point location The calculation method is as follows:

[0018]

[0019] Where p is Pixel coordinates within, It is the predicted offset value at pixel p, which is determined based on the offset heatmap; The area remaining after corrosion Number of pixels in the middle.

[0020] As a further preferred option, the Keypoints-RCNN network is trained using a pre-acquired training set, and the trained Keypoints-RCNN network is used as a keypoint heatmap prediction model. During training, classification training is performed based on the target object category determined by the classification prediction branch. That is, different region prediction branches and offset prediction branches are trained for different target object categories.

[0021] As a further preferred option, when training the Keypoints-RCNN network, the loss function L is:

[0022] L = L RPN +L cr +L hm +L ofst

[0023] Among them, L RPN ,L cr ,L hm ,L ofst These are the loss functions for the region extraction backbone, classification prediction branch, region prediction branch, and offset prediction branch, respectively.

[0024] Loss function L of the regional prediction branchhm For D(F) 1 The mean cross-entropy of the region and F C The mean cross-entropy of the regions is summed; D(F) 1 ) represents the positive nearest neighbor region, F C For other areas, F C =F 1 -D(F 1 ), F 1 The region where the true value is 1;

[0025] Loss function L of the offset prediction branch ofst =L ofst_comp +L ofst_syn L ofst_comp This is the component loss, used to measure region F. 1 The error between the predicted offset and the true offset at each point within the range is calculated using the mean square error function; L ofst_syn The Huber loss function is used to measure the overall systematic error as a composite loss.

[0026] As a further preferred method, the robot's grasping posture is determined based on the object's posture in the world coordinate system, including the following steps:

[0027] A set of candidate grasping postures is pre-acquired. Based on the robot's range of motion and interference from non-target objects, grasping postures in the candidate grasping posture set are excluded. Then, based on the object's posture in the world coordinate system, the remaining grasping postures are evaluated to determine the final robot grasping posture.

[0028] As a further preferred option, the remaining grasping postures are evaluated based on the object's pose in the world coordinate system, including the following steps:

[0029] For each remaining grasping posture: determine the coordinates of the center point of the gripping contact point based on the object's posture in the world coordinate system and the grasping posture; then calculate the torque of the object's gravity about the center point;

[0030] By comparing the torques corresponding to each grasping posture, the grasping posture with the smallest torque is selected as the final robot grasping posture.

[0031] According to a second aspect of the present invention, a robot 6D grasping posture estimation system based on RGB images is provided, comprising a processor for executing the above-described robot 6D grasping posture estimation method based on RGB images.

[0032] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described robot 6D grasping posture estimation method based on RGB images.

[0033] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:

[0034] 1. This invention designs a key point heatmap prediction model, which applies the region + offset method to the object's pose estimation, and then obtains the object's pose through post-processing of the region heatmap and offset heatmap, thereby improving the robot's grasping accuracy of the target object.

[0035] 2. This invention proposes a post-processing method for analyzing key point heatmaps, which includes a Top-K method based on expected area and an opening operation based on key point location. It integrates information from key point region heatmaps and offset heatmaps, thereby improving the accuracy of key point location estimation.

[0036] 3. When training the Keypoints-RCNN network, for the loss function of region prediction, the weight of the ground truth neighborhood is increased, so that the network focuses on the accuracy of predicting region boundaries; for the loss function of offset prediction, a synthetic loss is added to reduce the constant error of the predicted offset.

[0037] 4. When generating the grasping posture, based on the object point cloud model, stable grasping postures of the object in various directions are pre-searched offline as candidate postures. Then, the scene environment is filtered during the online calculation process, which has a fast calculation speed and finally outputs the grasping posture, which can balance the accuracy and real-time performance of the grasping posture. Attached Figure Description

[0038] Figure 1 This is a flowchart of a robot 6D grasping posture estimation method based on RGB images according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the Keypoints-RCNN network structure according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the prediction result area according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the top-K binarization method according to an embodiment of the present invention, wherein (a) is before binarization and (b) is after binarization;

[0042] Figure 5 This is a schematic diagram of the opening operation based on key point location in an embodiment of the present invention, wherein (a) is the erosion process and (b) is the expansion process;

[0043] Figure 6 Figures (a) and (b) are schematic diagrams illustrating the frictional stability conditions in an embodiment of the present invention.

[0044] Figure 7 This is a schematic diagram of the candidate grasping posture calculation process in an embodiment of the present invention, wherein (a) is the object model, (b) is rotation and voxelization, (c) is searching for feasible gripping point groups layer by layer, and (d) is calculating the gripping coordinates under O0;

[0045] Figure 8 This is a schematic diagram of the maximum clamping depth in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0047] This invention provides a robot 6D grasping pose estimation method based on RGB images. Based on the Faster-R-CNN framework, it designs a keypoint heatmap prediction model of "region + offset" and a post-processing method to decode the heatmap into keypoint coordinates, thereby determining the object's pose and enabling the robot to accurately grasp the target object. Figure 1 As shown, the specific steps include the following:

[0048] S1. Obtain the RGB image and 3D point cloud model of the target object, and determine the regional heatmap and offset heatmap based on the key point heatmap prediction model.

[0049] (1) The keypoint heatmap prediction model uses the Keypoints-RCNN network, the network structure of which is as follows: Figure 2 As shown, it includes a region extraction backbone, a classification prediction branch, a region prediction branch, and a shift prediction branch.

[0050] The region extraction backbone takes RGB images as input and includes a ResNet-101 network and an RPN network. The ResNet-101 network extracts primary features from the RGB images to obtain a global feature map. The RPN network selects image regions that may contain the target object based on these primary features and uses the ROIAlign method to reshape the image regions into tensors of uniform size, obtaining preliminary local feature maps. These feature maps are then input into the three branch networks mentioned above.

[0051] The classification prediction branch, based on preliminary local feature maps, predicts the target object category and accurate bounding boxes. Specifically, the classification prediction branch follows the structure of the Faster-R-CNN network, flattening the feature map into a one-dimensional vector through a 32×32 convolutional layer, then changing its dimension through a 1×1 convolutional layer, and finally obtaining the predicted category and bounding box correction parameters through a fully connected layer. The former will be used to select the 3D point cloud of the corresponding object category in the post-processing PnP method; the latter will correct the region position to obtain a more accurate bounding box. Using the more accurate bounding box, the ROIAlign method is applied again to the global feature map obtained from the backbone network to extract the region, cropping a 32×32 accurate local feature map from the feature map. These local feature maps will be used in the region prediction branch and the offset prediction branch; the more accurate bounding box will also be used to recover keypoint coordinates in post-processing.

[0052] The region prediction branch predicts the range of keypoints based on long-range features from precise local feature maps, i.e., determining the probability that each pixel belongs to a keypoint, thus obtaining a region heatmap. Specifically, the region prediction branch concatenates five groups of 32×32 convolutional layers and batch normalization layers. The first four groups use the LeakyReLU activation function to prevent neurons from becoming inactive during training. Since the final output is the probability that a pixel belongs to a keypoint, the Sigmoid activation function is used to keep the output value between 0 and 1. In some embodiments, 24 keypoints are pre-selected, which can output 24 region heatmaps for those 24 keypoints.

[0053] The offset prediction branch predicts the offset coordinates of pixels relative to keypoints within a given range based on long-range features from the precise local feature map, resulting in an offset heatmap. Specifically, the structure of the offset prediction network is consistent with the region prediction network, but it adds an extra dimension to output the offset coordinates of each pixel to the keypoint; that is, for each keypoint, it outputs a 32×32×2 feature map. By combining it with the region prediction branch, more accurate keypoint location predictions can be obtained. This method of using offset feature maps is essentially a voting mechanism; the specific voting method is described in step S22. In some embodiments, 24 keypoints are pre-selected, resulting in 48 offset heatmaps for these 24 keypoints. This is because the offset coordinates need to distinguish between the x and y axes, hence the need for 48 images.

[0054] Specifically, before keypoint location prediction, long-range features from the precise local feature map are extracted using the nonlocal module. Since the object is treated as a rigid body, the keypoints on its surface have defined spatial relationships and should therefore be correlated. Some keypoints project far apart; the nonlocal module can fuse these long-range features and implicitly capture the relationships between them, thereby improving prediction accuracy. This output is then fed to the region prediction branch and the offset prediction branch, respectively.

[0055] Specifically, key points need to be pre-selected from the 3D point cloud of the target object, and their locations are known. Key points are a series of points distributed on the surface of the target object to represent the object's pose. In order to minimize the impact of key point position prediction error on object pose prediction error, these points should be selected as widely as possible, preferably using the farthest point sampling algorithm.

[0056] (2) Before use, the Keypoints-RCNN network is trained in advance using the training set. The trained Keypoints-RCNN network is the key point heatmap prediction model.

[0057] In some embodiments, classification training can be performed based on the target object category determined by the classification prediction branch. That is, different region prediction branches and offset prediction branches are trained for different target object categories, and the number of categories can be set by the user. The training set includes multiple sets of training data, and each set of training data includes the corresponding RGB image, bounding box, category, and coordinates of key points in the RGB image.

[0058] Specifically, during training, the network's loss function has four parts: L RPN ,L cr ,L hm ,L ofst These correspond to four modules: region extraction backbone, classification prediction branch, region prediction branch, and offset prediction branch, respectively. RPN ,L cr The definition is the same as that of Faster R-CNN, so it will not be repeated here.

[0059] The total loss L is expressed as follows:

[0060] L = L RPN +L cr +L hm +L ofst

[0061] Loss function for the region prediction branch:

[0062] For binary classification problems, the commonly used loss function is the cross-entropy loss function. For example... Figure 3 As shown, let the true value of a region be F. In the region prediction of key points, the region with a true value of 0 is F.0 The area of ​​region F is significantly larger than the true value of 1. 1 Area, which leads to an imbalance between positive and negative samples. Furthermore, in F... 1 Nearby pixels are of greater interest and are referred to as the positive nearest neighbor region D(F). 1 They directly determine the accuracy of regional predictions. The positive example regions, excluding the nearest neighbor regions, are called the central region, denoted as F. C =F 1 -D(F 1 Therefore, calculate D(F) separately. 1 ) and F C Sum the mean cross-entropy values ​​of the regions:

[0063]

[0064] in, It is the nearest neighbor region of the true value; It is the central region of truth values. This is the cross-entropy formula, where f(p) is the true probability that pixel p is a positive example. Pixel p is the predicted probability value of a positive example.

[0065] Loss function for the offset prediction branch:

[0066] Because the offset values ​​from each point on the heatmap to the key point vary significantly, accurately regressing the offset of each point is difficult and unnecessary. We can focus only on F... 1 Point offset prediction within a region. The offset prediction loss function consists of two parts: one is the component loss, used to measure F... 1 The error between each point within the region and the true offset is calculated using the mean square error function. Let o(p) be the true offset at pixel p. Let p be the predicted offset value at pixel position p. Then the component loss is defined as:

[0067]

[0068] The second is the composite loss, which measures the overall systematic error. Let... Let F be the coordinates of the predicted key points within the heatmap, and kp be the true coordinates of the key points within the heatmap. Calculate F. 1 Inside The difference between kp and the result. Using the Huber loss function:

[0069]

[0070] L ofst_comp Using the mean squared error function and L ofst_synThe Huber function is used because it is more sensitive to larger errors than the latter, thus suppressing the local convergence problem where the synthesized value error decreases due to large offsets in some pixels. The offset loss is the sum of the two, L. ofst =L ofst_comp +L ofst_syn .

[0071] S2. Based on the regional heatmap and the offset heatmap, the coordinates of the key points relative to the RGB image are determined by the post-processing model, thereby obtaining the pose of the object in the world coordinate system.

[0072] The network outputs region heatmaps and offset heatmaps, which are feature maps of keypoint locations. The goal is to convert these feature maps into keypoint coordinates. Post-processing is also required. Post-processing not only parses the feature map into coordinates, but also corrects the network's prediction results to a certain extent, making the object's pose estimation more accurate. Post-processing mainly includes three steps: Top-K binarization based on the expected area, opening operation based on keypoint location, and solving for the object's pose.

[0073] S21. Top-K binarization based on expected area;

[0074] The region heatmap output by the network contains values ​​between 0 and 1. A binarization method needs to be chosen to select a subset of pixels for keypoint calculation. A preferred approach is the top-K method: selecting the K points with the largest output values ​​and setting them to 1, while setting the rest to 0, to binarize the region heatmap. Selecting too many pixels will result in the calculation of offsets including too many F values. 0 The pixels within the region are not included in the offset loss function during training, resulting in larger errors; if too few pixels are selected, the influence of random errors becomes even greater. Therefore, an appropriate value of K needs to be chosen based on the location of the keypoint peaks.

[0075] Specifically, the selection of the K value follows this strategy:

[0076] Select the point with the largest pixel output value in the regional heatmap, and denote it as p. top The output value indicates the probability that a pixel belongs to a key point.

[0077] Calculate K using the function leftA(·): K = leftA(R, p top ); its representation is p top Draw a circle with center R and radius R (R is preset). `leftA(·)` is a function to calculate the area of ​​the remaining circles within the heatmap's range. Figure 4 As shown.

[0078] S22, Opening operation based on key point location;

[0079] The method for converting feature maps into keypoint coordinates involves filtering the offset heatmap using a binarized region heatmap, selecting only locations where the value is 1. The calculation, which is essentially a voting mechanism, uses the following formula to calculate the key point location:

[0080]

[0081] Where p is the pixel coordinate. It is the predicted offset value at pixel p, which is determined based on the offset heatmap; Refers to the area The number of pixels in the middle. Since the offset heatmap has two independent outputs, they need to be calculated separately and then averaged. In fact, the offset map also contains information about which points are near the ground truth of the keypoints, which can be used to... Make corrections.

[0082] Opening is a morphological method that combines erosion and dilation operations in sequence. In this method, the erosion operation remains constant, but the dilation operation is performed based on the predicted keypoint locations. Through erosion, the efficiency can be reduced... The middle belongs to F 0 The proportion of pixels is determined to eliminate their interference. The voting results of the remaining pixels are used as the new center point, and the nearest pixel to this center point is selected. Each point is expanded, and combined with the remaining points to form a new...

[0083] like Figure 5 As shown, the specific steps include the following:

[0084] S221, Using 3×3 cross-shaped structural units to define the region The corrosion process is carried out, and the remaining area after corrosion is The complement region is denoted as Initially, the iteration number i = 1. The method for determining the initial value of the iteration is as follows: for the binarized region heatmap, the selected pixel region is the initial region. The value K is the area.

[0085] Specifically, remember After excluding the outermost pixels, the general situation is as follows: Therefore, the impact of some uncertain pixels can be reduced.

[0086] S222, According to the region Calculate key point locations using offset heatmaps (Calculated using the formula for the location of the key points mentioned above), and the area is calculated.

[0087] S223, Expansion with distance, in Internal selection Set the nearest N points to 1 to obtain the region. in That is, to expand to the point that

[0088] S224. Judgment and The degree of overlap is determined, and iteration stops when the degree of overlap meets the criterion, outputting the result at this point. This refers to the coordinates of the key point relative to the region heatmap; otherwise, let i = i + 1 and return to step S221. In some embodiments, the condition is satisfied. Stop iteration and output. Threshold ∈ = 0.1.

[0089] S23. Solve for the object's orientation;

[0090] S231. Restore the coordinates of the key points relative to the region heatmap to the image coordinate system, that is, obtain the coordinates of the key points relative to the RGB image;

[0091] The keypoint coordinates calculated above are relative to a 32×32 heatmap size, while the actual keypoint coordinates are within the original image. Therefore, they need to be reconstructed based on the position and size of the bounding box. Let the keypoint heatmap coordinates calculated by S22 be... Then we have:

[0092]

[0093] Where h, w, t, l are the height, width, x-coordinate of the top-left corner, and y-coordinate of the top-left corner of the bounding box, respectively; This refers to the coordinates of the key points relative to the RGB image.

[0094] S232. Solve the PnP problem to obtain the object's pose in the camera coordinate system;

[0095] The above steps yielded the 2D projection coordinates of the keypoints in the image coordinate system. Since the 3D coordinates of the keypoints in the object coordinate system are known, the pose of the object in the camera coordinate system can be determined based on this. This is a typical PnP problem. The `solvePnP` function from the OpenCV library can easily solve this problem, so it will not be elaborated upon here.

[0096] S233. Transform to the world coordinate system to obtain the object's attitude in the world coordinate system;

[0097] The above steps yielded the object's pose in the camera coordinate system. Then, with the camera position fixed, hand-eye calibration can be used to calculate the transformation matrix from the camera coordinate system to the world coordinate system, thus obtaining the object's pose in the world coordinate system, which is used to generate the subsequent grasping posture.

[0098] S3. Based on the object's posture, determine the robot's grasping posture to enable the robot to grasp the target object.

[0099] The robot's grasping posture can be determined based on the object's pose in the world coordinate system. This embodiment further presents a method that combines an object point cloud model to pre-determine candidate grasping postures, and then combines the object's pose to determine the robot's grasping posture.

[0100] Specifically, the grasping action can be modeled as an interference problem between the gripper and the object's point cloud. By modeling the gripper, the feasibility of grasping objects in different postures can be tested, and the postures with higher feasibility are called candidate grasping postures; the candidate grasping postures for each object are calculated independently in advance. In multi-object scenes, since objects may interfere with each other, it is necessary to select candidate grasping postures and fine-tune them to determine the final grasping posture in the scene.

[0101] S31, Calculation of candidate grasping posture;

[0102] Generating candidate grasping poses based on a 3D point cloud model of the object's surface requires understanding that this step is independent of the object's pose and is calculated offline in advance for the target object.

[0103] In some embodiments, such as Figure 7 As shown, S31 includes the following steps:

[0104] Obtain the 3D point cloud of the object and generate N rotation matrices R. i The purpose of rotating the coordinate system is to examine the possibilities of gripping an object from different directions; the rotation matrix R is controlled by setting the subdivision number N. i The more numerous the items, the more comprehensively the possibilities of clamping are considered.

[0105] For each rotation matrix R i Perform the following steps:

[0106] (1) The three-dimensional point cloud of the object has an initial object coordinate system O0. The object coordinate system O0 is rotated according to the rotation matrix R. i Rotate to obtain a new coordinate system O. i The point cloud is obtained in the new coordinate system O. i The coordinates below;

[0107] (2) Perform voxelization on the point cloud in the new coordinate system to generate voxel tensors;

[0108] (3) In the voxel tensor, slice layer by layer along the preset coordinate axis (the preset coordinate axis is the axis parallel to the Z-axis of the gripper); and traverse along the preset coordinate axis to find the set of gripping points that meet the constraint conditions on each contour slice. All the obtained gripping point sets form a set G. i ;

[0109] (4) Set G i The coordinates of the clamping point group are transformed into coordinates in the initial object coordinate system O0 and added to the set G0;

[0110] Traverse all rotation matrices R i Then, all elements in set G0 are the candidate grasp poses.

[0111] Furthermore, the process of finding a set of clamping points that meet the constraints on each contour slice includes the following steps:

[0112] Multiple clamping point groups are initially determined on the contour slice using Hough transform: Based on the shape of the clamp, the Hough pattern of each pixel on the contour slice in Hough space is determined, and these patterns are discretized into a grid in Hough space at a certain resolution and then superimposed; then, Hough space grid points that satisfy the superposition value (determined by the number of contact points between the clamp and the object) are selected, that is, the intersection points of the Hough patterns that satisfy the conditions; each set of Hough patterns corresponding to the intersection point corresponds to a set of pixels on the contour slice, which is a clamping point group; thus, multiple clamping point groups are obtained.

[0113] Considering the stability of the contact between each clamping point group and the clamper, remove clamping point groups that do not meet the stability conditions;

[0114] Then, for each clamping point group, the maximum reachable depth along the preset coordinate axis is calculated, and the clamping point position is modified according to this maximum reachable depth, such as... Figure 8 As shown, the modified gripping point is then added to set G. i This increases the potential contact area between the gripper and the object, ensuring that the object will not fall due to unforeseen disturbances during the gripping process.

[0115] Furthermore, the stability condition is:

[0116] After initially obtaining the gripping point set through the Hough transform, the stability of the contact between the obtained gripping point set and the gripper needs to be considered. Many factors affect the stability of a gripping posture; two conditions are considered:

[0117] (i) Clamping force balance condition: The force applied to the surface of the object by the clamping point should be balanced, especially to prevent lateral torque from causing the object to roll. Since the clamping point is located in roughly the same plane, this condition is naturally satisfied.

[0118] (ii) Two key conditions for frictional stability are: to explore these conditions, it is necessary to know the normal vector of the object's surface and the normal vector of the gripper's fingers at the contact point. The angle between these two constitutes the pressure angle; the smaller the pressure angle, the less likely slippage will occur. This condition is decomposed into horizontal and vertical directions, and an appropriate pressure angle threshold α is selected. h ,α v (This is related to the material of the contact surface and is artificially specified) to determine whether it is stable, such as Figure 6 As shown, this is because vertical friction is primarily relied upon to resist gravity, and calculating it separately is more consistent with reality. If any clamping point in a clamping point group fails to meet condition (ii), the results for that group are discarded.

[0119] S32, Scene capture posture generation;

[0120] In S31, candidate grasping poses have already been calculated, and the robot's grasping pose can be determined by combining them with the object's pose. However, in actual multi-object grasping scenarios, two factors need to be considered: the accessibility of the grasping position and the influence of the gripper on non-target objects. First, due to the robot's limited range of motion, most grasping poses are unreachable and are excluded. The remaining positions need to be verified again to see if they interfere with the surrounding point cloud (determined based on other non-target objects in the scene), and any interfering poses are also excluded. Finally, the grasping pose with the highest grasping score among the remaining poses is selected as the scene's grasping pose. Since the candidate grasping poses are pre-calculated offline, calculating the scene's grasping pose does not take much time.

[0121] Specifically, the method for determining the crawling score is as follows:

[0122] Based on the pre-acquired 3D point cloud model of the object, the position of the object's center of gravity is pre-calculated; for each remaining grasping posture: the coordinates of the center point of the gripping contact point are determined and calculated according to the object's posture in the world coordinate system and the grasping posture; then, based on the position of the object's center of gravity, the torque of the object's gravity about the center point is calculated.

[0123] By comparing the torques corresponding to each grasping posture, the grasping posture with the smallest torque is selected as the final robot grasping posture.

[0124] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot 6D grasping pose estimation method based on RGB images, characterized in that, The steps include the following: The RGB image and 3D point cloud model of the target object are acquired, and a region heatmap and an offset heatmap are obtained through a keypoint heatmap prediction model. The keypoint heatmap prediction model includes a region extraction backbone, a classification prediction branch, a region prediction branch, and an offset prediction branch, wherein: The region extraction backbone is used to extract feature maps from RGB images; the classification prediction branch is used to determine the target object category and predict bounding boxes based on the feature maps, and then determine local feature maps based on the bounding boxes; the region prediction branch is used to determine the probability that each pixel belongs to a key point based on long-distance features in the local feature maps, and obtain a region heatmap, wherein the key points are pre-selected in the 3D point cloud model of the object; the offset prediction branch is used to determine the offset coordinates of each pixel relative to the key points based on long-distance features in the local feature maps, and obtain an offset heatmap. The region heatmap is binarized, and the offset heatmap is filtered using the binarized region heatmap to obtain the coordinates of the key points relative to the region heatmap, and then the coordinates of the key points relative to the RGB image are determined; based on the coordinates of the key points relative to the RGB image, the pose of the object in the camera coordinate system is obtained; then based on the transformation matrix from the camera coordinate system to the world coordinate system, the pose of the object in the world coordinate system is obtained. Based on the object's posture in the world coordinate system, the robot's grasping posture is determined, enabling the robot to grasp the target object.

2. The robot 6D grasping pose estimation method based on RGB images as described in claim 1, characterized in that, Binarization of the region heatmap is performed using the top-K binarization method: Sort the output values ​​of pixels in the region heatmap from largest to smallest, select the top K pixels, set these K pixels to 1, and set the other pixels to 0 to complete the binarization of the region heatmap; the method for determining the value of K is as follows: draw a circle with the pixel with the largest output value in the region heatmap as the center and a preset radius R, and the area of ​​this circle within the region heatmap is K; the output value refers to the probability that the pixel belongs to a key point.

3. The robot 6D grasping pose estimation method based on RGB images as described in claim 2, characterized in that, The offset heatmap is filtered using the binarized region heatmap to obtain the coordinates of key points relative to the region heatmap, including the following steps: (1) The region is divided into 3×3 cross-shaped structural units. The corrosion process is carried out, and the remaining area after corrosion is The complement region is denoted as Initially, the iteration number i = 1. The method for determining the initial value of the iteration is as follows: for the binarized region heatmap, the selected pixel region is the initial region. (2) According to the region Calculate key point locations using offset heatmaps And calculate the area The area Refer to Draw a circle with a preset radius and center; the area of ​​this circle within the region heatmap. (3) Expansion with respect to distance, in Internal selection Set the nearest N points to 1 to obtain the region. in (4)Judgment and The degree of overlap is determined, and iteration stops when the degree of overlap meets the criterion, outputting the result at this point. That is, the coordinates of the key point relative to the heatmap of the region; otherwise, let i = i + 1 and return to step (1).

4. The robot 6D grasping pose estimation method based on RGB images as described in claim 3, characterized in that, In step (2), the location of key points The calculation method is as follows: Where p is Pixel coordinates within, It is the predicted offset value at pixel p, which is determined based on the offset heatmap; The area remaining after corrosion Number of pixels in the middle.

5. The robot 6D grasping pose estimation method based on RGB images as described in claim 1, characterized in that, The Keypoints-RCNN network is trained using a pre-acquired training set, and the trained Keypoints-RCNN network is used as a keypoint heatmap prediction model. During training, classification training is performed based on the target object category determined by the classification prediction branch. That is, different region prediction branches and offset prediction branches are trained for different target object categories.

6. The robot 6D grasping pose estimation method based on RGB images as described in claim 5, characterized in that, When training the Keypoints-RCNN network, the loss function L is: L=L RPN +L cr +L hm +L ofst Among them, L RPN ,L cr ,L hm ,L ofst These are the loss functions for the region extraction backbone, classification prediction branch, region prediction branch, and offset prediction branch, respectively. Loss function L of the regional prediction branch hm For D(F) 1 The mean cross-entropy of the region and F C The mean cross-entropy of the regions is summed; D(F) 1 ) represents the positive nearest neighbor region, F C For other areas, F C =F 1 -D(F 1 ), F 1 The region where the true value is 1; Loss function L of the offset prediction branch ofst =L ofst_comp +L ofst_syn L ofst_comp This is the component loss, used to measure region F. 1 The error between the predicted offset and the true offset at each point within the range is calculated using the mean square error function; L ofst_syn The Huber loss function is used to measure the overall systematic error as a composite loss.

7. The robot 6D grasping pose estimation method based on RGB images as described in any one of claims 1-6, characterized in that, Based on the object's pose in the world coordinate system, the robot's grasping pose is determined, including the following steps: A set of candidate grasping postures is pre-acquired. Based on the robot's range of motion and interference from non-target objects, grasping postures in the candidate grasping posture set are excluded. Then, based on the object's posture in the world coordinate system, the remaining grasping postures are evaluated to determine the final robot grasping posture.

8. The robot 6D grasping pose estimation method based on RGB images as described in claim 7, characterized in that, Based on the object's pose in the world coordinate system, the remaining grasping poses are evaluated, including the following steps: For each remaining grasping posture: determine the coordinates of the center point of the gripping contact point based on the object's posture in the world coordinate system and the grasping posture; then calculate the torque of the object's gravity about the center point; By comparing the torques corresponding to each grasping posture, the grasping posture with the smallest torque is selected as the final robot grasping posture.

9. A robot 6D grasping pose estimation system based on RGB images, characterized in that, Includes a processor for executing the RGB image-based robot 6D grasping pose estimation method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot 6D grasping posture estimation method based on RGB images as described in any one of claims 1-8.

Images