A 3D pose detection algorithm for oblate spherical fruits based on key point detection

Abstract

The present invention relates to the field of artificial intelligence, and particularly to a 3D pose detection algorithm for oblate spherical fruits that combines key point detection based on deep learning with point cloud processing. A 3D pose detection algorithm for oblate spherical fruits based on key point detection is provided, and this algorithm should be characterized by fast detection speed and high accuracy. The technical solution is as follows: A 3D pose detection algorithm for oblate spherical fruits based on key point detection, comprising the following steps: 1) Collect RGB-D images of oblate spherical fruits; 2) Process with a key point detection model; 3) Generate a point cloud map; 4) Point cloud segmentation; 5) Voxel downsampling; 6) Outlier removal; 7) Color Euclidean clustering; 8) RANSAC sphere fitting; 9) Key point indexing; 10) Connect the 3D key points to the centroid to obtain the 3D pose of the oblate spherical fruit.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a 3D pose detection algorithm for oblate fruits that combines key point detection based on deep learning with point cloud processing. Background Art

[0002] A large number of fruits and vegetables sold in the current market are oblate fruits, such as tomatoes, apples, oranges, etc. Agricultural harvesting is a labor-intensive industry. With the continuous expansion of the planting area of oblate fruits, labor costs are also increasing year by year. Robot picking is the future development direction of agricultural harvesting. Mechanized large-area brute-force harvesting has been widely applied to oblate fruits grown in fields, but non-destructive harvesting robots for expensive fresh oblate fruits have not been applied commercially yet.

[0003] In a complex background picking environment, it is very important to confirm the 3D pose of oblate fruits, which can provide information guidance for the pose of the robotic arm and obstacle avoidance during picking. At the same time, existing research on key point detection of oblate fruits usually uses a cascaded object detection and key point detection model, which has problems such as slow detection speed and low accuracy, wasting a large amount of time and space and being unable to be applied to real picking working conditions. There is less research on 3D pose detection and it needs to be further improved. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the above background art and provide a 3D pose detection algorithm for oblate fruits based on key point detection, which should have the characteristics of fast detection speed and high accuracy.

[0005] The technical solution of the present invention is as follows:

[0006] A 3D pose detection algorithm for oblate fruits based on key point detection, comprising the following steps:

[0007] 1) Collect RGB-D images of oblate fruits;

[0008] 2) Input the RGB image into the key point detection model to obtain the bounding box and key points of the oblate fruit;

[0009] 3) Generate a point cloud map according to the RGB-D image;

[0010] 4) Map the bounding box and key points to the point cloud map and crop the point cloud map according to the bounding box;

[0011] 5) Perform voxel downsampling on the point cloud map;

[0012] 6) Remove outliers in the point cloud map according to Euclidean clustering;

[0013] 7) Perform clustering operation on the point cloud map according to color Euclidean clustering;

[0014] 8) Use the RANSAC sphere fitting algorithm to fit each bounding box respectively to obtain the centroid of the oblate spherical fruit;

[0015] 9) Expand the pixel points around the key points to find the effective point cloud as the 3D key points of the oblate spherical fruit;

[0016] 10) Connect the 3D key points with the centroid to obtain the 3D pose of the oblate spherical fruit.

[0017] The key point detection model includes a Backbone network, a Neck network, and a Head network; the Head network includes key point regression.

[0018] In the key point regression, the output sizes of the three detection layers are 25×25, 50×50, and 100×100 respectively; the formulas of the three detection layers are: 25×25×[(7 + ncls)×3], 50×50×[(7 + ncls)×3], 100×100×[(7 + ncls)×3].

[0019] In the Backbone network of the key point detection model, there is also a CA attention mechanism module; the CA attention mechanism module is located between the C3_1_1 module and the Conv2 module.

[0020] In the Backbone network of the key point detection model, the parameters of the SPP module are 5, 7, 9, and 13.

[0021] In the key point detection model, the bounding box loss function is the SIoU loss function, and the key point loss function is the Wing loss function.

[0022] The method for establishing the key point detection model is as follows:

[0023] 1) Make a tomato key point detection dataset;

[0024] 2) Input the tomato key point detection dataset into the key point detection model for training;

[0025] 3) Deploy the key point detection model to the ros framework.

[0026] The RANSAC sphere fitting algorithm includes:

[0027] 1) Randomly select 4 points in the bounding box and calculate the spherical surface where these 4 points are located;

[0028] 2) Calculate the distance d from all points in the bounding box to the spherical surface, set a threshold dr, if d < dr, mark it as an inner point of the spherical surface, otherwise mark it as an outer point of the spherical surface, and record the number of inner points;

[0029] 3) Iterate the previous two steps, select the spherical surface with the largest number of inner points, and calculate the center of the spherical surface, which is the centroid of the oblate fruit.

[0030] The beneficial effects of the present invention are as follows:

[0031] The present invention only needs to input an RGB-D image of an oblate fruit, and can complete the 3D pose detection of the oblate fruit by obtaining the centroid and key points on the image; in the detection layer of the key point detection model of the present invention, key point regression is added, so that the model can complete the tasks of bounding box and key point detection, and then based on point cloud processing (including point cloud segmentation, voxel downsampling, outlier removal, color Euclidean clustering, RANSAC sphere fitting and key point indexing), the 3D pose of the oblate fruit is obtained, which has high detection speed and accuracy, and can be applied to the actual harvesting operation of oblate fruits, providing information guidance for the obstacle avoidance and picking of the robotic arm. Description of the Drawings

[0032] Figure 1 is the schematic diagram of the principle of the present invention.

[0033] Figure 2 is the labeling schematic diagram of the data set of the present invention.

[0034] Figure 3 is the architecture diagram of the key point detection model of the present invention.

[0035] Figure 4 is the RGB-D image of the tomato.

[0036] Figure 5 is the output image of the key point detection model of the present invention.

[0037] Figure 6 is the image after point cloud segmentation of the present invention.

[0038] Figure 7 is the image after voxel downsampling of the present invention.

[0039] Figure 8 is the image after color Euclidean clustering of the present invention.

[0040] Figure 9 is the image after RANSAC sphere fitting of the present invention.

[0041] Figure 10 is the 3D pose diagram of the tomato of the present invention.

[0042] Figure 11Schematic diagram of the CA attention mechanism module of the key point detection model of the present invention.

[0043] Figure 12 Schematic diagram of the SPP module of the key point detection model of the present invention.

[0044] Figure 13 Schematic diagram of the Focus module of the key point detection model of the present invention.

[0045] Figure 14 Schematic diagram of the convolutional layer module of the key point detection model of the present invention.

[0046] Figure 15 Schematic diagram of the C3 module of the key point detection model of the present invention.

[0047] Figure 16 Schematic diagram of the Bottleneck module of the key point detection model of the present invention. Detailed implementation manners

[0048] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] The oblate fruits in this embodiment are taken as tomatoes for example.

[0050] A 3D pose detection algorithm for oblate fruits based on key point detection includes the following steps:

[0051] 1) Collect RGB-D images

[0052] Use a Rralense L515 camera to collect RGB-D images of tomatoes; the RGB-D images include RGB images and depth images;

[0053] The RGB image provides the x and y coordinates under pixel coordinates, and the depth image provides the z coordinate under camera coordinates, that is, the distance between the camera and the point; according to the RGB-D image information and the internal parameters of the camera, the coordinates of any pixel point in the camera coordinate system can be calculated, and a point cloud map in different coordinate systems can be synthesized accordingly;

[0054] 2) Process by the key point detection model

[0055] Input the RGB image into the key point detection model for processing, and the key point detection model outputs an RGB image with tomato bounding boxes and tomato key points; there is one key point in each bounding box, and the key point is the center of the calyx of the tomato; both the bounding box and the key point are two-dimensional position information, and the subsequent steps are indexed on the point cloud map to obtain the three-dimensional information of the bounding box and the key point;

[0056] 3) Generate a point cloud map

[0057] Generate a point cloud map based on the RGB-D image;

[0058] 4) Point cloud segmentation

[0059] Map the bounding box and key points to the point cloud map and crop the point cloud map according to the bounding box;

[0060] Using the bounding box information to crop the point cloud map and segment the tomatoes can avoid using the entire point cloud map, reduce the number of points in the point cloud to be processed, and speed up the algorithm operation;

[0061] 5) Voxel downsampling

[0062] Perform voxel downsampling on the cropped point cloud map;

[0063] With the improvement of the accuracy of point cloud acquisition instruments, the number of point clouds to be processed and calculated is increasing continuously. Traditional point cloud feature algorithms can no longer adapt to the processing of point clouds with a huge number of points; voxel downsampling will create a three-dimensional voxel grid based on the original point cloud data, calculate the centroid of all points in each cube to replace all points in the voxel grid, reduce the number of point clouds, maintain the morphological characteristics of the oblate fruit (tomato) point cloud, and speed up the algorithm;

[0064] 6) Outlier removal

[0065] Remove outliers from the downsampled point cloud according to Euclidean clustering;

[0066] There will be some inaccurate depths at the boundary points of the tomato contour. Therefore, remove the outliers from the downsampled point cloud according to Euclidean clustering;

[0067] 7) Color Euclidean clustering

[0068] Since the calyx position of the tomato is flat, first perform color Euclidean clustering on the point cloud for clustering operation to remove the point cloud of the calyx part, and then perform RANSAC sphere fitting in the next step;

[0069] The biological form of the oblate fruit is oblate. The point cloud at the calyx position has a certain adverse effect on obtaining the centroid by fitting the oblate fruit point cloud with RANSAC sphere. Moreover, there is a large color difference between the oblate fruit and the calyx. Through color Euclidean clustering, the calyx can be separated from the oblate fruit. After removing the calyx part, the oblate fruit point cloud is more similar to a sphere, which is beneficial to obtaining a more accurate centroid of the oblate fruit;

[0070] 7) RANSAC sphere fitting

[0071] The RANSAC sphere fitting algorithm is used to fit each bounding box of the point cloud map respectively to obtain the centroid of the tomato;

[0072] The RANSAC sphere fitting algorithm includes:

[0073] ① Randomly select 4 points in the bounding box and calculate the spherical surface where these 4 points are located;

[0074] The calculation formula of the spherical surface is (x - x0) 2 +(y - y0) 2 +(z - z0) 2 =r 2 , where (x, y, z) are the coordinates of the 4 points, (x0, y0, z0) are the coordinates of the center of the sphere, and r is the radius of the spherical surface;

[0075] ② Calculate the distance d from all points in the bounding box to the spherical surface, set a threshold dr. If d < dr, it is recorded as an inlier of the spherical surface, otherwise it is recorded as an outlier of the spherical surface, and record the number of inliers;

[0076] The threshold dr generally takes a value of 0.8 - 1.2 cm, preferably 1.0 cm;

[0077] ③ Iterate the above two steps, select the spherical surface with the largest number of inliers, and calculate the center of the sphere of the spherical surface. The center of the sphere is the centroid of the oblate fruit; Each iteration will calculate an iteration evaluation factor according to the expected error rate, the number of inliers, the total number of samples, and the current iteration number, and determine whether to stop the iteration according to the iteration evaluation factor;

[0078] 8) Key point indexing

[0079] Expand the pixel points around the key points to find the effective point cloud as the 3D key points of the oblate fruit;

[0080] The key points provided by the key point detection model are two-dimensional points, and the three-dimensional coordinate points need to be indexed in the point cloud; However, the depth map obtained by the Realsense L515 cannot obtain the depth of all pixel points, and there are some missing parts in the point cloud map compared with the two-dimensional color map, which will cause the situation where the key points cannot be indexed; Therefore, the pixel points around the key points are traversed. The specific method is to scan line by line in the range of 21×21 (a total of 441 pixel points) with the key point as the center. When the scanned point cloud has depth, it is the effective point cloud, and it is used as the 3D key point; The positions of the pixel points in this range have no obvious change in the three-dimensional space, so the positions of the indexed 3D key points are accurate;

[0081] 9) Connect the 3D key points with the centroid to obtain the 3D pose of the tomato.

[0082] The key point detection model uses the YOLO v5 model. The key point detection model includes a Backbone network, a Neck network, and a Head network.

[0083] Among them, the Backbone network is used to extract image features at different scales, the Neck network plays a role in feature fusion, and the Head network can obtain three types of feature maps, which are used to predict small, medium, and large targets from large to small. These feature maps are divided into grids. For each grid, three prior boxes (anchors) will be used to predict the target boundary. The detection layer is mainly used for the final inference and detection of the model. This network applies the Anchor box to the feature map output by the previous Neck network and finally outputs the class probability of the target object, the object score, the bounding box position vector, and the key point position vector.

[0084] The key point detection model is an improved YOLO v5 model.

[0085] The improvements of the key point detection model are as follows: adding key point regression, optimizing parameters, optimizing the loss function, and adding an attention mechanism. Improvements to the YOLO v5 model

[0086] 1) Adding key point regression

[0087] The key point detection model adds key point regression to the three detection layers of the Head network on the basis of the YOLO v5 model. The output sizes of the three detection layers are 25×25, 50×50, and 100×100 respectively. The formulas for the three detection layers are: 25×25×[(7 + ncls)×3], 50×50×[(7 + ncls)×3], 100×100×[(7 + ncls)×3].

[0088] Among them: 25×25, 50×50, and 100×100 are the sizes of the feature maps; 7 represents the bounding box position, length and width, confidence, and key point position (key point regression), which was 5 before improvement and only included position, length and width, and confidence; each detection layer will output a feature vector, enabling the model to complete both bounding box and key point detection tasks simultaneously. ncls represents the number of classifications.

[0089] 2) Optimizing parameters

[0090] Analyzing the tomato dataset shows that tomato samples all belong to small targets, which are very different from the coco dataset. Therefore, the parameters of SPP are modified from 5, 9, 13 to 5, 7, 9, 13 to improve the detection accuracy of tomatoes.

[0091] 3) Optimizing the loss function

[0092] The bounding box loss function of the key point detection model has been changed from GIoU to SIoU. SIoU redefines the penalty metric considering the vector angle between the required regressions and is one of the best loss functions currently.

[0093] The SIoU loss function consists of 4 Cost functions: Angle cost, Distance cost, Shape cost, IoU cost.

[0094] ①Angle cost:

[0095]

[0096] where b and b gt represent the center points of the predicted box and the ground truth box.

[0097] ②Distance cost:

[0098]

[0099] γ = 2 - Λ

[0100] where c h and c w represent the length and width of the smallest closed interval that can contain both the predicted box and the ground truth box.

[0101] ③Shape cost:

[0102]

[0103] where h gt and w gt represent the length and width of the ground truth box, and h and w represent the length and width of the predicted box.

[0104] ④IoU cost:

[0105] L IoUCost = 1 - IoU

[0106]

[0107] The key point loss function is Wingloss, which can well balance the influence of size errors in the early and late stages of training on key point prediction. For small errors, a logarithmic function with a bias term is used, and for large errors, L1 loss is used, thus balancing the influence of size errors in the initial and late stages of training on key points. The formula is

[0108]

[0109] W is an integer that constrains the range of the non-linear part in the interval [-w, w], ∈ is the curvature of the non-linear boundary region, and C = w - wln(1 + x / ∈) is a constant that can be compared with the linear and non-linear parts of the smooth connection segment.

[0110] 4) Add the attention mechanism

[0111] Since the shape and color of tomatoes have certain differences from the background, in order to improve the accuracy of the key point detection model, a CA attention mechanism module is added between the C3_1_1 module and the Conv2 module to improve the model's ability to extract important features.

[0112] The CA attention mechanism module embeds position information into the channel attention mechanism, decomposes the global pooling into one-to-one feature encoding operations, enabling the attention mechanism module to capture spatial long-range correlations with precise position information. The attention mechanism mainly acts on the feature map, which generates a two-dimensional attention score map by continuously generating two independent attention maps for channels and space, and then applies the attention score map to the input feature map for adaptive feature refinement. The attention mechanism enhances the representation ability of important features in the feature map and suppresses unimportant features.

[0113] The method for establishing the key point detection model is as follows:

[0114] 1) Make a tomato key point detection dataset

[0115] Obtain tomato pictures, and use the labelme annotation tool to manually mark the bounding boxes and key points of tomatoes on the pictures;

[0116] Figure 2 Show the marking methods for the bounding boxes and key points. The annotation file records the coordinates of the center of the bounding box and its width and height, and also records the coordinates of the key points. If the calyx is visible, the key point is marked at the center of the calyx. If the calyx is not visible due to the pose of the tomato, it is marked at the intersection of the tomato boundary and the fruit axis.

[0117] 2) Divide the dataset into a training set and a test set, and input them into the key point detection model for training to obtain a trained key point detection model;

[0118] 3) Deploy the key point detection model to the ros framework;

[0119] The key point detection model communicates with the point cloud processing module through the ros framework, transmitting the coordinate information of the bounding box and key point positions; ros means Robot Operating System, which has a distributed structure and provides an excellent communication mechanism, offering three communication methods: topics, services, and actions, for different communication needs.

[0120] This keypoint detection model can complete keypoint and bounding box detection tasks. Experiments showed that the improved model achieved a mean average pose (MAP) of 92.9%, a 1.8% increase from the original model's 91.1%. The overall 3D pose detection success rate was 93.4%. The 3D pose detection time for an oblate spheroid fruit using RGB-D images was only 0.09 seconds (the detection time for a single tomato), significantly improving detection speed. Existing technologies typically take only seconds, as shown in the following table:

[0121]

[0122] In the present invention, point cloud segmentation, voxel downsampling, outlier removal, color Euclidean clustering, RANSAC sphere fitting, SIoU loss function, Wing loss function, YOLO v5 model, and CA attention mechanism module are all existing technologies.

[0123] Finally, it should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples and is subject to numerous variations. All variations that can be directly derived or conceived by a person of ordinary skill in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A 3D pose detection algorithm for oblate fruits based on key point detection, comprising the following steps: 1) Collect RGB-D images of oblate fruits; 2) Input the RGB image into the key point detection model to obtain the bounding box and key points of the oblate fruit; 3) Generate a point cloud map according to the RGB-D image; 4) Map the bounding box and key points to the point cloud map and crop the point cloud map according to the bounding box; 5) Perform voxel downsampling on the point cloud map; 6) Remove outliers from the point cloud map according to Euclidean clustering; 7) Perform clustering operations on the point cloud map according to color Euclidean clustering; 8) Use the RANSAC sphere fitting algorithm to fit each bounding box respectively to obtain the centroid of the oblate fruit; 9) Expand the pixel points around the key points to find valid point clouds as the 3D key points of the oblate fruit; 10) Connect the 3D key points and the centroid to obtain the 3D pose of the oblate fruit.

2. The 3D pose detection algorithm for oblate fruits based on key point detection according to claim 1, characterized in that: The key point detection model includes a Backbone network, a Neck network, and a Head network; the Head network includes key point regression.

3. The 3D pose detection algorithm for oblate fruits based on key point detection according to claim 2, wherein: In the key point regression, the output sizes of the three detection layers are 25×25, 50×50, and 100×100 respectively; the formulas of the three detection layers are: 25×25×[(7 + ncls)×3], 50×50×[(7 + ncls)×3], 100×100×[(7 + ncls)×3]; ncls represents the number of classifications.

4. The 3D pose detection algorithm for oblate spheroid fruits based on key point detection according to claim 3, wherein: In the Backbone network of the key point detection model, there is also a CA attention mechanism module; the CA attention mechanism module is located between the C3_1_1 module and the Conv2 module.

5. The 3D pose detection algorithm for oblate spherical fruits based on key point detection according to claim 4, wherein: In the Backbone network of the key point detection model, the parameters of the SPP module are 5, 7, 9, and 13.

6. The 3D pose detection algorithm for oblate spheroid fruits based on key point detection according to claim 5, characterized in that: In the key point detection model, the bounding box loss function is the SIoU loss function, and the key point loss function is the Wingloss loss function.

7. An algorithm for 3D pose detection of oblate fruits based on key point detection according to claim 6, characterized in that: The method for establishing the key point detection model is as follows: 1) Make a tomato key point detection dataset; 2) Input the tomato key point detection dataset into the key point detection model for training; 3) Deploy the key point detection model to the ros framework.

8. A 3D pose detection algorithm for oblate spherical fruits based on key point detection according to claim 7, characterized in that: The RANSAC sphere fitting algorithm includes: 1) Randomly select 4 points in the bounding box and calculate the spherical surface where these 4 points are located; 2) Calculate the distance d from all points in the bounding box to the spherical surface, set a threshold dr, if d < dr, it is recorded as an inlier of the spherical surface, otherwise it is recorded as an outlier of the spherical surface, and record the number of inliers; 3) Iterate the first two steps, select the spherical surface with the largest number of inliers, and calculate the center of the spherical surface, which is the centroid of the oblate fruit.

Images