A densely stacked flower bud sorting method and sorting system based on deep learning

Abstract

The present invention provides a densely stacked flower bud sorting method and sorting system based on deep learning, comprising the following steps: completing the teaching of a mechanical arm; collecting RGB images and point cloud depth maps of densely stacked flower buds; improving the deep learning model and pre-training to obtain a target detection model; using the target detection model to perform target detection on the collected RGB images, obtaining the prediction information of the target flower buds and the two-dimensional coordinates of the target flower buds' centroid, and then measuring the spatial coordinates of the target flower bud sorting position in the point cloud depth map; selecting the flower buds to be sorted and sending the spatial coordinates of their sorting positions to the drive controller in sequence; after the drive controller receives the spatial coordinates of the sorting positions of the flower buds to be sorted, it controls the mechanical arm to perform sorting; and sorts different flower buds to be sorted to designated positions. The present invention has a higher accuracy rate for target detection of densely stacked flower buds, requires less computing power, predicts the sorting positions of the flower buds to be sorted more accurately, and has a higher sorting efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a densely stacked flower bud sorting method and a sorting system based on deep learning. Background Art

[0002] In recent years, flower buds (such as jasmine) are often used as embellishments for pastry products, and the flower buds need to be sorted during processing. At present, the sorting of flower buds from densely stacked flower buds is done manually, which restricts production efficiency and poses food safety risks. Object detection methods based on deep learning have been widely used in the field of object sorting. The existing object detection method based on RGB images can well identify flower buds of different categories, but the single RGB image cannot fully reflect the spatial information of the object to be sorted. Although the object detection method using point cloud can better obtain the spatial information of the object to be sorted, when sorting densely stacked flower buds, which are blocked by features and have irregular centroid positions, the accuracy of the existing sorting methods and sorting systems cannot meet the requirements of actual production, resulting in problems such as unstable product quality and low production efficiency. Summary of the invention

[0003] 1. Technical problems to be solved:

[0004] The accuracy of existing sorting methods and sorting systems cannot meet the requirements of actual production, resulting in unstable product quality and low production efficiency.

[0005] 2. Technical solution:

[0006] In order to solve the above problems, the present invention provides a densely stacked flower bud sorting method based on deep learning, which is characterized by comprising the following steps:

[0007] Step S01: perform hand-eye calibration on the camera coordinate system and the robot arm coordinate system, and complete the teaching of the robot arm.

[0008] Step S02: The image acquisition end acquires the RGB image of densely stacked flower buds and the point cloud depth map corresponding to the pixels.

[0009] Step S03: Improve the YOLOv8 deep learning model and perform pre-training to obtain a target detection model.

[0010] Step S04: Use the target detection model to perform target detection on the collected RGB image, obtain the category and prediction box of the target flower bud, use the heat map generated by the gradient weighted class activation mapping to obtain the two-dimensional coordinates of the target flower bud's center of mass, perform depth ranging on the two-dimensional coordinates of the target flower bud's center of mass in the point cloud depth map to obtain the spatial coordinates of the target flower bud's center of mass, and use them as the spatial coordinates of the target flower bud sorting position.

[0011] Step S05: Prioritize sorting and screening of target flower buds, select the flower buds to be sorted and send the spatial coordinates of their sorting positions to the drive controller in sequence.

[0012] Step S06: After the driving controller receives the spatial coordinates of the sorting position of the flower buds to be sorted, it controls the robot arm to perform sorting.

[0013] Step S07: The robot arm performs sorting and sorts the flower buds to be sorted to designated locations.

[0014] Furthermore, in step S01, the specific method is: use a camera to take multiple RGB-D images of the checkerboard calibration plate, and then use the calibration images to perform hand-eye calibration on the camera coordinate system and the robotic arm coordinate system, and calculate and save the relative posture between the object and the robotic arm coordinate system through the grasping and photographing posture relationship to complete the teaching.

[0015] Furthermore, the specific method of step S03 includes the following steps:

[0016] Step S31: Collect sample data and perform image preprocessing; use the image acquisition terminal to collect 2,000 different images of flower buds densely stacked in a material tray, and then preprocess the images.

[0017] Step S32: Label the dataset images: Use rolabelimg to label the dataset images, classify and label them according to qualified flower buds and defective flower buds, save and output the labeling files, and then divide the dataset into training set and test set according to the ratio of 8:2.

[0018] Step S33: Improve the YOLOv8 deep learning model; replace the backbone network with the StarNet network, and replace the activation function in the spatial pyramid pooling structure with the RELU function. The RELU activation function is as follows:

[0019] ReLU(x)=max(0,x).

[0020] In the neck network, the Bottleneck in the C2f module is replaced with Star Blocks, and the BiFPN network is added to the feature fusion module. In the head network, all modules from the detection head with an output scale of 20×20 as the initial to the upsampling layer of this size as the end are eliminated, and the retained detection heads are replaced with directional bounding box detection heads.

[0021] Step S34: Use the labeled data set to perform pre-training in the improved YOLOv8 deep learning model, and take the best training result as the target detection model.

[0022] Furthermore, in step S31, the image preprocessing method is: firstly, adaptive histogram equalization is performed on the collected image to obtain more image detail information, and then weighted average grayscale processing is performed on the image. The definition of the weighted average method is as follows:

[0023] Gray(x,y)=aR(x,y)+bG(x,y)+cB(x,y),

[0024] Where a, b, c are the weights of the R channel, G channel, and B channel respectively, and a+b+c=1, x, y are the coordinate information of different pixels, and the values ​​of a, b, c are finally determined based on multiple experiments.

[0025] Furthermore, the specific method of step S04 is:

[0026] Step S41: predict the category of the target flower bud; input the collected RGB image into the target detection model to obtain the output prediction box and the category of the target flower bud.

[0027] Step S42: predict the equivalent particle system of the target flower bud; perform gradient weighted class activation mapping on the 5th, 11th, and 14th layers of the target detection model to obtain three heat maps, take all positions with a heat value of 1 in the heat map as the equivalent particle system of the target flower bud, and obtain the coordinates of all particles in the equivalent particle system (u i ,v i ).

[0028] Step S43: Predict the two-dimensional coordinates C of the centroid of the target flower bud avg (u,v).

[0029] Step S44: predict the spatial coordinates C of the target bud sorting position space (x,y,z).

[0030] Further, in step S43, the specific method is: the two-dimensional coordinates C(u,v) of the center of mass of the equivalent mass point system can be calculated by the center of mass calculation method of the plane mass point system, as follows:

[0031]

[0032] Among them, m i is the sum of pixels in the particle system, M is the sum of the masses of all particles; after obtaining three different two-dimensional coordinates of the center of mass C1(u1,v1), C2(u2,v2), C3(u3,v3) in the three heat maps, the two-dimensional coordinates of the center of mass of the final target bud C avg (u,v), the specific calculation method is as follows:

[0033]

[0034] Furthermore, the specific method of step S44 is: in the collected point cloud depth map, the two-dimensional coordinate C of the centroid of the target flower bud is calculated. avg (u, v) is used to measure the distance, obtain the corresponding depth information d, and use the point cloud three-dimensional coordinate calculation formula to determine the two-dimensional coordinate C of the target flower bud center of mass avg The spatial coordinates corresponding to (u, v) are determined as the spatial coordinates C of the target bud sorting position. space (x, y, z), the point cloud three-dimensional coordinate calculation formula is as follows:

[0035]

[0036] Among them, f x and f y are the horizontal and vertical focal lengths of the camera, respectively, c x and c y is the x-coordinate of the camera image principal point and the y-coordinate of the image principal point.

[0037] Furthermore, the specific method of step S05 is:

[0038] Step S51: preprocess the point cloud data of densely stacked flower buds; use a straight-through filtering algorithm to remove background point cloud information to obtain point cloud data containing only densely stacked flower buds, and use a Euclidean clustering segmentation algorithm to segment the stacked flower bud point cloud into independent point cloud subsets.

[0039] Step S52: Obtain complete 3D point cloud data of the target flower bud; create a three-dimensional model of the flower bud, and perform point cloud sampling to obtain complete 3D point cloud data of the target flower bud.

[0040] Step S53: Prioritize sorting and screening based on the spatial coordinates of the target flower bud sorting position and the point cloud matching degree.

[0041] Step S54: sending the spatial coordinates of the sorting positions of the flower buds to be sorted to the drive controller in sequence.

[0042] Furthermore, the specific method of step S53 is: comprehensively consider the spatial coordinates of the target flower bud sorting position and the point cloud matching degree to perform sorting priority scoring, and obtain the scores S of different target flower buds. i , then the score S i Normalize and obtain the final sorting score S i ', as follows:

[0043]

[0044] Where w1, w2 and w3 are the weight coefficients of the target bud sorting position z coordinate, the target bud sorting position (x, y coordinate) and the number of point clouds of the target bud complete 3D point cloud data, respectively, and w1+w2+w3=1, z i 、x i ,y i and N i are respectively represented by the z-axis coordinate, x-axis coordinate, y-axis coordinate of the i-th target flower bud sorting position and the number of point clouds of the target flower bud independent point cloud subset, z max is the maximum value of the z-axis coordinate of all target flower bud sorting positions, N is the number of point clouds of the complete 3D point cloud data of the target flower bud, and the weight coefficients w1=0.60, w2=0.10, w3=0.30 are set. Finally, S′ is selected i All flower buds with a value >0.4 are to be sorted.

[0045] The present invention also provides a densely stacked flower bud sorting system based on deep learning, including an image acquisition end, an industrial computer, a drive controller and a robotic arm. The image acquisition end includes a depth camera, which collects RGB images of densely stacked flower buds in a material tray and a point cloud depth map corresponding to the pixels, and sends them to the industrial computer respectively; the industrial computer receives the RGB image sent by the image acquisition end and processes it to obtain the spatial coordinates of the target flower bud sorting position, and performs priority sorting and screening on the target flower buds, selects the flower buds to be sorted according to the priority sorting scores, and sends the spatial coordinates of their sorting positions to the drive controller in sequence; after the drive controller receives the coordinates of the sorting position of the flower buds to be sorted, it controls the robotic arm to perform sorting; the robotic arm performs sorting and sorts the flower buds to be sorted to the designated position.

[0046] 3. Beneficial effects:

[0047] The present invention has a higher accuracy rate in target detection of densely stacked flower buds, and requires less computing power for real-time work; the spatial coordinates of the grasping positions of the flower buds to be sorted are predicted more accurately; and the sorting efficiency of performing the sorting of densely stacked flower buds is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a work flow chart of the densely stacked flower bud sorting method of the present invention.

[0049] Figure 2 It is a network structure diagram of the improved YOLOv8 deep learning model of the present invention.

[0050] Figure 3 This is a structural diagram of the improved C2f module of the present invention.

[0051] Figure 4 It is a workflow diagram for predicting the spatial coordinates of target flower buds of the present invention.

[0052] Figure 5 It is a work flow chart of the present invention for preferentially sorting and screening target flower buds.

[0053] Figure 6 It is a structural schematic diagram of the densely stacked flower bud sorting system of the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 As shown, a densely stacked flower bud sorting method based on deep learning includes step S01: hand-eye calibration of the camera coordinate system and the robot arm coordinate system, and completing the teaching of the robot arm; step S02: the image acquisition end collects the RGB image of the densely stacked flower buds and the point cloud depth map corresponding to the pixels; step S03: improving the YOLOv8 deep learning model and performing pre-training to obtain a target detection model; step S04: using the target detection model to perform target detection on the collected RGB image, obtain the category and prediction box of the target flower bud, and generate a gradient weighted class activation map. The two-dimensional coordinates of the target flower bud mass center are obtained by using the heat map, and the spatial coordinates of the target flower bud mass center are obtained by depth measurement of the two-dimensional coordinates of the target flower bud mass center in the point cloud depth map, and the spatial coordinates are used as the spatial coordinates of the sorting position of the target flower bud; Step S05: Prioritize sorting and screening of the target flower buds, select the flower buds to be sorted and send the spatial coordinates of the sorting position to the drive controller in sequence; Step S06: After the drive controller receives the spatial coordinates of the sorting position of the flower buds to be sorted, it controls the robot arm to perform sorting; Step S07: The robot arm performs sorting and sorts different flower buds to be sorted to the designated positions.

[0056] Step S01: Perform hand-eye calibration on the camera coordinate system and the robotic arm coordinate system, and complete the teaching of the robotic arm: Use the camera to take multiple RGB-D images of the checkerboard calibration plate, and then use the calibration images to perform hand-eye calibration on the camera coordinate system and the robotic arm coordinate system according to the Zhang Zhengyou calibration method. By grasping and photographing the posture relationship, calculate and save the relative posture between the object and the robotic arm coordinate system to complete the teaching.

[0057] Step S02: The image acquisition end acquires the RGB image and point cloud depth map of the densely stacked flower buds: the RGB image of the densely stacked flower buds in the material tray and the point cloud depth map corresponding to the pixels thereof are acquired.

[0058] Step S03: Improve the YOLOv8 deep learning model and perform pre-training to obtain a target detection model.

[0059] Step S03 includes the following steps: Step S31, collecting sample data and performing image preprocessing; using the image acquisition end to collect 2000 different images of flower buds densely stacked in the material tray, and then preprocessing the images.

[0060] In one embodiment, the image preprocessing method is: firstly, adaptive histogram equalization is performed on the collected image to obtain more image detail information, and then weighted average grayscale processing is performed on the image. The definition of the weighted average method is as follows:

[0061] Gray(x,y)=aR(x,y)+bG(x,y)+cB(x,y),

[0062] Among them, a, b, c are the weights of R channel, G channel and B channel respectively, and a+b+c=1, x, y are the coordinate information of different pixels. After multiple experiments, the values ​​of a, b, c are finally determined to be 0.20167, 0.70838, and 0.08995 respectively.

[0063] Step S32, labeling the data set images: using rolabelimg to label the data set images, classifying and labeling them according to qualified flower buds and defective flower buds, saving and outputting the labeling files, and then dividing the data set into a training set and a test set according to a ratio of 8:2;

[0064] Step S33: Figure 2 As shown, in one embodiment, the method for improving the YOLOv8 deep learning model is: replacing the backbone network with the StarNet network, replacing the activation function in the spatial pyramid pooling structure with the RELU function, and the RELU activation function is as follows:

[0065] ReLU(x)=max(0,x).

[0066] In the neck network, Figure 3 As shown in the figure, the Bottleneck in the C2f module is replaced with Star Blocks, and the BiFPN network is added to the feature fusion module. In the head network, all modules from the detection head with an output scale of 20×20 as the initial to the upsampling layer of this size as the end are eliminated, and the retained detection head is replaced with a directional bounding box detection head.

[0067] The number of parameters of the YOLOv8 deep learning model before the improvement is 3151904, and the required computing power is 8.7GFLOPs. The number of parameters of the YOLOv8 deep learning model after the improvement is 1174659, and the required computing power is 4.3GFLOPs.

[0068] Step S34: Pre-training: Use the labeled data set to pre-train in the improved YOLOv8 deep learning model, the number of epochs of training is 200, the number of batches of images in each batch is 8, the size of the input image is 640, Mosaic data enhancement is enabled, other parameters remain default, and the result with the best training effect is taken as the target detection model;

[0069] Step S04: Figure 4 As shown, the target detection model is used to perform target detection on the collected RGB image to obtain the category and prediction box of the target flower bud, and the heat map generated by the gradient weighted class activation mapping is used to obtain the two-dimensional coordinates of the center of mass of the target flower bud. The two-dimensional coordinates of the center of mass of the target flower bud are measured in depth in the point cloud depth map to obtain the spatial coordinates of the center of mass of the target flower bud, which are used as the spatial coordinates of the sorting position of the target flower bud. The main steps are as follows;

[0070] Step S41: The method for predicting the prediction box and category of the target flower bud is: inputting the collected RGB image into the trained target detection model to obtain the output prediction box and the category of the target flower bud.

[0071] Step S42: In one embodiment, the method for predicting the equivalent particle system of the target flower bud is as follows: performing gradient weighted class activation mapping on the 5th, 11th, and 14th layers of the target detection model to obtain three heat maps of the output prediction boxes, taking all positions with a heat value of 1 in the heat map as the equivalent particle system of the target flower bud, and obtaining all particle coordinates (u i ,v i ).

[0072] Step S43: In one embodiment, predict the two-dimensional coordinates C of the target flower bud avg The method of calculating the center of mass of the equivalent particle system is as follows: The two-dimensional coordinates C(u,v) of the center of mass of the equivalent particle system can be calculated by the method of calculating the center of mass of the plane particle system, as follows:

[0073]

[0074] Among them, m i is the sum of pixels in the particle system, and M is the sum of the masses of all particles.

[0075] After obtaining three different centroid two-dimensional coordinates C1(u1,v1), C2(u2,v2), and C3(u3,v3), the two-dimensional coordinates of the final target bud centroid C avg (u,v), the calculation method is as follows.

[0076]

[0077] Step S44: In one embodiment, predict the spatial coordinates C of the target bud sorting position space The method is as follows: in the collected point cloud depth map, the two-dimensional coordinate C of the centroid of the target flower bud is avg (u, v) is measured to obtain the corresponding depth information d, and the two-dimensional coordinate C is determined using the point cloud three-dimensional coordinate calculation formula avg The spatial coordinates corresponding to (u, v) are determined as the spatial coordinates C of the target bud sorting position. space (x, y, z), the formula for calculating the three-dimensional coordinates of the point cloud is as follows.

[0078]

[0079] Among them, f x and f y are the horizontal and vertical focal lengths of the camera, respectively, c x and c y is the x-coordinate of the camera image principal point and the y-coordinate of the image principal point.

[0080] Step S05: Figure 5 As shown, the target flower buds are prioritized for sorting and screening, the flower buds to be sorted are selected and the spatial coordinates of their sorting positions are sent to the drive controller in sequence, which mainly includes the following steps:

[0081] Step S51: preprocess the point cloud data of densely stacked flower buds; use a straight-through filtering algorithm to remove background point cloud information to obtain point cloud data containing only densely stacked flower buds, and use a Euclidean clustering segmentation algorithm to segment the stacked flower bud point cloud into independent point cloud subsets.

[0082] Step S52: Obtain complete 3D point cloud data of the target flower bud; create a three-dimensional model of the flower bud, and perform point cloud sampling to obtain complete 3D point cloud data of the target flower bud.

[0083] Step S53: Prioritize sorting and screening based on the spatial coordinates of the target flower bud sorting position and the point cloud matching degree; perform comprehensive scoring based on the spatial position of the flower bud sorting position and the point cloud matching degree to obtain the scores S of different target flower buds. i , then the score S i Normalize and obtain the final sorting score S′ i , finally select S′ i All flower buds with a value >0.4 are to be sorted.

[0084] In one embodiment, the method for sorting and prioritizing target flower buds is as follows:

[0085]

[0086] Where w1, w2 and w3 are weight coefficients of the target flower bud sorting position z coordinate, the target flower bud sorting position (x, y) coordinate and the number of point clouds of the target flower bud complete 3D point cloud data, respectively, and w1+w2+w3=1. i 、x i ,y i and N i are respectively represented by the z-axis coordinate, x-axis coordinate, y-axis coordinate of the i-th target flower bud sorting position and the number of point clouds of the target flower bud independent point cloud subset, z max is the maximum value of the z-axis coordinate in all target bud sorting positions, and N is the number of point clouds of the complete 3D point cloud data of the target bud. Because the height position is the most important for sorting, the point cloud matching degree is second, and the (x, y) coordinate has the least influence. Therefore, the weight coefficients w1 = 0.60, w2 = 0.10, and w3 = 0.30 are set.

[0087] Step S54: sending the spatial coordinates of the sorting positions of the flower buds to be sorted to the drive controller in sequence; sending the spatial coordinates of the sorting position of the flower bud to be sorted with the highest score to the drive controller according to the priority sorting score, and after completing the sorting of the flower bud, sending the spatial coordinates of the sorting position of the next flower bud to be sorted to the drive controller again until the sorting of all the flower buds to be sorted is completed.

[0088] Step S06: After the driving controller receives the spatial coordinates of the sorting position of the flower buds to be sorted, it controls the robot arm to perform sorting.

[0089] Step S07: The robot arm performs sorting and sorts the different flower buds to be sorted to designated locations.

[0090] The present invention also provides a densely stacked bud sorting system based on deep learning, such as Figure 6 As shown, it includes an image acquisition end, an industrial computer, a drive controller and a mechanical arm. The image acquisition end includes a depth camera, which collects the RGB image of densely stacked flower buds in the material tray and the point cloud depth map corresponding to the pixels, and sends them to the industrial computer respectively; the industrial computer receives the RGB image sent by the image acquisition end and processes it to obtain the spatial coordinates of the sorting position of the target flower buds, and performs priority sorting and screening on the target flower buds, selects the flower buds to be sorted according to the priority sorting scores, and sends the spatial coordinates of their sorting positions to the drive controller in sequence; after the drive controller receives the coordinates of the sorting position of the flower buds to be sorted, it controls the mechanical arm to perform sorting; the mechanical arm performs sorting and sorts the flower buds to be sorted to the specified position.

Claims

1. A densely stacked flower bud sorting method based on deep learning, characterized in that: The following steps are involved: Step S01: perform hand-eye calibration on the camera coordinate system and the robotic arm coordinate system, and complete the teaching of the robotic arm; Step S02: the image acquisition end acquires the RGB image of densely stacked flower buds and the point cloud depth map corresponding to the pixels thereof; Step S03: Improve the YOLOv8 deep learning model and perform pre-training to obtain a target detection model; Step S04: Use the target detection model to perform target detection on the collected RGB image, obtain the category and prediction box of the target flower bud, use the heat map generated by the gradient weighted class activation mapping to obtain the two-dimensional coordinates of the target flower bud mass center, perform depth ranging on the two-dimensional coordinates of the target flower bud mass center in the point cloud depth map to obtain the spatial coordinates of the target flower bud mass center, and use it as the spatial coordinates of the target flower bud sorting position; Step S05: Prioritize sorting and screening of target flower buds, select flower buds to be sorted and send the spatial coordinates of their sorting positions to the drive controller in sequence; Step S06: After the driving controller receives the spatial coordinates of the sorting position of the flower buds to be sorted, it controls the robot arm to perform sorting; Step S07: The robot arm performs sorting and sorts the flower buds to be sorted to designated locations.

2. The densely stacked flower bud sorting method based on deep learning as claimed in claim 1, characterized in that: In step S01, the specific method is: use a camera to take multiple RGB-D images of the checkerboard calibration plate, and then use the calibration images to perform hand-eye calibration on the camera coordinate system and the robotic arm coordinate system, and calculate and save the relative pose between the object and the robotic arm coordinate system through the grasping and photographing pose relationship to complete the teaching.

3. The densely stacked flower bud sorting method based on deep learning as claimed in claim 1, characterized in that: The specific method of step S03 comprises the following steps: Step S3 1: Collect sample data and perform image preprocessing; use the image acquisition terminal to collect 2,000 different images of flower buds densely stacked in a material tray, and then preprocess the images; Step S32: labeling the data set images: using rolabelimg to label the data set images, classifying and labeling them according to qualified flower buds and defective flower buds, saving and outputting the labeling files, and then dividing the data set into a training set and a test set in a ratio of 8:2; Step S33: Improve the YOLOv8 deep learning model; replace the backbone network with the StarNet network, and replace the activation function in the spatial pyramid pooling structure with the RELU function. The RELU activation function is as follows: ReLU(x)=max(0,x), In the neck network, replace Bottleneck in the C2f module with Star Blocks, and add the BiFPN network to the feature fusion module. In the head network, remove all modules from the initial to the end of the upsampling layer with an output scale of 20×20, and replace the retained detection head with a directional bounding box detection head. Step S34: Use the labeled data set to perform pre-training in the improved YOLOv8 deep learning model, and take the best training result as the target detection model.

4. The densely stacked flower bud sorting method based on deep learning as claimed in claim 3, characterized in that: In step S31, the image preprocessing method is: firstly, adaptive histogram equalization is performed on the collected image to obtain more image detail information, and then weighted average grayscale processing is performed on the image. The definition of the weighted average method is as follows: Gray(x,y)=aR(x,y)+bG(x,y)+cB(x,y), Where a, b, c are the weights of the R channel, G channel, and B channel respectively, and a+b+c=1, x, y are the coordinate information of different pixels, and the values ​​of a, b, c are finally determined based on multiple experiments.

5. The densely stacked flower bud sorting method based on deep learning as claimed in claim 1, characterized in that: The specific method of step S04 is: Step S41: predicting the category of the target flower bud; inputting the collected RGB image into the target detection model to obtain an output prediction box and the category of the target flower bud; Step S42: predicting the equivalent particle system of the target flower bud; Gradient weighted class activation mapping is performed on the 5th, 11th, and 14th layers of the target detection model to obtain three heat maps. All positions with a heat value of 1 in the heat map are taken as the equivalent mass point system of the target bud, and the coordinates of all particles in the equivalent mass point system (u i , v i ); Step S43: Predict the two-dimensional coordinates C of the centroid of the target flower bud avg (u, v); Step S44: predict the spatial coordinates C of the target bud sorting position space (x, y, z).

6. The densely stacked flower bud sorting method based on deep learning as claimed in claim 5, characterized in that: In step S43, the specific method is: the two-dimensional coordinates C(u, v) of the center of mass of the equivalent mass point system can be calculated by the center of mass calculation method of the plane mass point system, as follows: Among them, m i is the sum of pixels in the particle system, M is the sum of the masses of all particles; after obtaining three different two-dimensional coordinates of the center of mass C1(u1, v1), C2(u2, v2), C3(u3, v3) in the three heat maps, the two-dimensional coordinates of the center of mass of the final target bud C avg (u, v), the specific calculation method is as follows:

7. The densely stacked flower bud sorting method based on deep learning as claimed in claim 5, characterized in that: The specific method of step S44 is: in the collected point cloud depth map, the two-dimensional coordinate C of the centroid of the target flower bud is avg (u, v) is measured to obtain the corresponding depth information d, and the two-dimensional coordinates C of the target flower bud centroid are determined using the point cloud three-dimensional coordinate calculation formula avg The spatial coordinates corresponding to (u, v) are determined as the spatial coordinates C of the target bud sorting position. space (x, y, z), the point cloud three-dimensional coordinate calculation formula is as follows: Among them, f x and f y are the horizontal and vertical focal lengths of the camera, respectively, c x and c y is the x-coordinate of the camera image principal point and the y-coordinate of the image principal point.

8. The densely stacked flower bud sorting method based on deep learning as claimed in claim 1, characterized in that: The specific method of step S05 is: Step S51: preprocessing the point cloud data of densely stacked flower buds; The straight-through filtering algorithm is used to remove the background point cloud information to obtain the point cloud data containing only densely stacked flower buds. The stacked flower bud point cloud is segmented using the Euclidean clustering segmentation algorithm to segment the stacked flower buds into independent point cloud subsets. Step S52: obtaining complete 3D point cloud data of the target flower bud; making a three-dimensional model of the flower bud, and performing point cloud sampling to obtain complete 3D point cloud data of the target flower bud; Step S53: Prioritize sorting and screening according to the spatial coordinates of the target flower bud sorting position and the point cloud matching degree; Step S54: sending the spatial coordinates of the sorting positions of the flower buds to be sorted to the drive controller in sequence.

9. The densely stacked flower bud sorting method based on deep learning as claimed in claim 8, characterized in that: The specific method of step S53 is: comprehensively consider the spatial coordinates of the target flower bud sorting position and the point cloud matching degree to perform sorting priority scoring, and obtain the scores S of different target flower buds. i , then the score S i Normalize and obtain the final sorting score S′ i , as follows: Where w1, w2 and w3 are the weight coefficients of the target bud sorting position z coordinate, the target bud sorting position (x, y) coordinate and the number of point clouds of the target bud complete 3D point cloud data, respectively, and w1+w2+w3=1, z i 、x i ,y i and N i are respectively represented by the z-axis coordinate, x-axis coordinate, y-axis coordinate of the i-th target flower bud sorting position and the number of point clouds of the target flower bud independent point cloud subset, z max is the maximum value of the z-axis coordinate of all target flower bud sorting positions, N is the number of point clouds of the complete 3D point cloud data of the target flower bud, and the weight coefficients w1=0.60, w2=0.10, w3=0.30 are set. Finally, S′ is selected i All flower buds with a value >0.4 are to be sorted.

Images