Intelligent robot famous high-quality tea picking method based on multi-stage fusion detection and posture analysis

Abstract

The invention relates to an intelligent robot famous high-quality tea picking method based on multi-stage fusion detection and posture analysis, and belongs to the technical field of intelligent tea picking. The method comprises the following steps: firstly, carrying out background point filtering on an obtained RGB-D image and a three-dimensional point cloud of tea leaves; obtaining a tea leaf two-dimensional detection frame from the RGB-D image and obtaining a tea leaf three-dimensional detection frame from the three-dimensional point cloud; matching the two-dimensional detection frame with the three-dimensional detection frame to obtain two-dimensional detection results and three-dimensional detection results which are in one-to-one correspondence; extracting a plurality of tea two-dimensional key points from a tea two-dimensional detection result, and guiding three-dimensional point cloud clustering through the two-dimensional key points; finally, the posture and length of the stalks are obtained through a stalk posture analysis method, and the picking positions and the picking angles of the tea leaves on the stalks are obtained. The robustness and stability of the intelligent picking robot are improved through multiple fusion detection stages, the accurate picking depth and angle are obtained through attitude analysis, and the picking quality of intelligent picking is guaranteed.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent tea picking, and relates to an intelligent robot high-quality tea picking method based on multi-stage fusion detection and posture analysis. Background Art

[0002] As the public's pursuit of a healthy lifestyle and natural products increases, the demand for tea, as a popular natural drink, is also increasing. High-quality tea is more popular with the world because of its better taste than other types of tea. However, the increase in demand for high-quality tea is also accompanied by a large amount of production. In the production of high-quality tea, the labor force for tea picking accounts for more than 60% of the entire tea garden management labor. It is the task with the most operations, the largest labor consumption, and the highest labor intensity in the entire tea production labor. The quality of tea picking directly affects the quality and value of tea. With the increase in labor costs, the lack of experts, and the improvement of tea quality requirements, the simple manual tea picking method can no longer meet the demand. Therefore, the tea industry's demand for innovative picking methods is becoming increasingly urgent. The promotion of mechanized and intelligent picking has become an inevitable choice for the sustainable development of high-quality tea.

[0003] In recent years, with the rapid development of computer vision, machine learning and robotics, the field of smart agriculture has also made great progress. The method of detecting young shoots through deep learning has also been widely studied. Through deep learning models such as YOLO and FasterRCNN, the position of the tea detection frame can be accurately detected, laying the foundation for intelligent picking. For the determination of the picking point position, most of the methods currently use threshold segmentation, key point detection, point cloud processing and other methods to roughly obtain the picking point position. However, the tea garden environment is complex, and tea leaves and miscellaneous leaves are highly similar and occlude each other. These interference factors easily lead to the fuzzy picking position obtained by the algorithm. In addition, most algorithms currently still stay at the determination of the picking position, and do not care about the growth posture of famous tea, which often leads to wrong picking methods. Therefore, the precise picking method of famous tea still needs to be studied. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method for picking high-quality tea with an intelligent robot based on multi-stage fusion detection and posture analysis, so as to achieve the goal of obtaining accurate picking positions and picking angles of high-quality tea using multiple sensors such as RGB-D depth cameras and laser radars in the complex environment of tea gardens. The method uses the RGB-D image, depth 3D image acquired by the RGB-D camera, and the three-dimensional point cloud acquired by the laser radar as input, and adopts preprocessing technology, deep learning technology, and unsupervised machine learning technology to provide a solution for intelligent tea picking. By improving the robustness and stability of picking, it enables intelligent picking of high-quality tea in complex environments.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis includes at least mounting an RGB-D camera and a laser radar on an intelligent robot, and calculating a picking position and angle based on tea information data acquired by the RGB-D camera and the laser radar, which includes the following steps:

[0007] S1. Using GB-D dual-component joint preprocessing method to filter out background points from the acquired RGB-D image and three-dimensional point cloud of tea leaves;

[0008] S2. Establish a tea 2D image target detection model to obtain the tea 2D detection frame from the RGB-D image, and use the point cloud region candidate network algorithm to obtain the tea 3D detection frame from the 3D point cloud;

[0009] S3, using the semantic and spatial multi-constrained adaptive cross non-maximum suppression method to match the two-dimensional detection frame and the three-dimensional detection frame to obtain a one-to-one corresponding two-dimensional tea detection result and a three-dimensional tea detection result;

[0010] S4, using the tea key point detection model to extract the positions of several tea 2D key points from the tea 2D detection results, and integrating the 2D key points into the corresponding tea 3D detection point cloud to guide 3D point cloud clustering, and outputting the tea 3D point cloud clustering results;

[0011] S5. Separate the local point cloud of the stem from the clustering result of the tea 3D point cloud, and then obtain the posture and length of the stem through the stem posture analysis method, and finally obtain the picking position and picking angle of the tea on the stem.

[0012] Further, in step S1, background points are filtered out from the RGB-D image and the point cloud data according to the GB-D dual-component joint preprocessing method, which includes: S11, RGB-D image background point filtering; S12, point cloud data background point filtering;

[0013] In step S11, the RGB-D image background point filtering process includes the following steps:

[0014] S111, extracting the GB component map and the depth D component map in the RGB-D image data, constructing a variance joint estimation function according to the two component maps, and jointly estimating the variance values ​​of the pixels in the image under different thresholds according to the two component maps, which is expressed as:

[0015]

[0016] Among them, λ gb , dRespectively represent the weight coefficients of GB component map and D component map, They represent the probability that the pixel in the GB component image is the foreground and background when the threshold is t, They represent the mean of the foreground and background pixels in the GB component image when the threshold is t, They represent the probability that the pixel in the D component image is the foreground and background when the threshold is t, They represent the mean of the pixel points of the D component image as foreground and background when the threshold is t, and L(t) represents the variance value when the threshold is t;

[0017] S112, using variance joint estimation function to obtain the best segmentation threshold It is expressed as:

[0018]

[0019] S113, the best segmentation threshold is obtained The background points of the original RGB-D image are filtered out, which is expressed as:

[0020]

[0021] Among them, f(i,j) represents the pixel value at the pixel point (i,j), represents the best segmentation threshold; the pixel point (i, j) with a pixel value of 0 represents the background point, which is filtered out;

[0022] In step S12, the point cloud data background point filtering process includes the following steps:

[0023] S121. Transform the pixel points in the RGB-D image to the three-dimensional point coordinates in the laser radar coordinate system:

[0024]

[0025] in They represent the 2D coordinates of the background point (i, j) in the image coordinate system and the 3D coordinates of the background point (i, j) in the lidar coordinate system. i,j Represents the depth value of the pixel coordinate (i, j) in the D component image, f X 、f Y Respectively represent the equivalent focal lengths in the X and Y directions in the camera coordinate system, u0 and v0 represent the coordinates of the center point of the image plane, and R and t represent the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system;

[0026] S122, calculate the distance dis between the mth point in the original three-dimensional point cloud and the nth point after the background point with a pixel value of 0 in the RGB-D image is converted into a three-dimensional point cloud m,n :

[0027]

[0028] Among them, (x m ,y m ,z m )、(x n ,y n ,z n ) respectively represent the three-dimensional coordinates of the m-th point in the point cloud 101 and the three-dimensional coordinates of the n-th point in the point cloud after the projection transformation of the image background pixels, and the total number of the two in the point cloud is M and N respectively;

[0029] S123, set threshold d, if dis m,n If it is smaller than the preset threshold d, it indicates that point m in the original 3D point cloud is a background point and it is filtered out; otherwise, it is retained.

[0030] Further, in step S2, the preprocessed RGB-D image is subjected to a two-dimensional detection frame extraction, which includes: building a tea two-dimensional image target detection model TNet, obtaining a two-dimensional detection result I of the tea in the image coordinate system 2d ; The tea 2D image target detection model TNet includes a backbone network, a feature fusion network, and a prediction layer, among which,

[0031] Preliminary feature extraction is performed in the backbone network. First, different shape convolution parameters are learned for the convolution kernel to adapt to the growth direction of tea stems and buds, which is expressed as:

[0032]

[0033] where v(p0+p n ) means that at p0+p n The shape adaptation coefficient at the position, w(p n ) means in p n The convolution coefficient of the direction, x(p0+p n ) indicates that in the feature map p0+p n The feature value at the position, y(p0) represents the feature extraction result of several points around the position p0, and R represents the coordinate direction set of several points around it; the shape adaptive convolution uses a 3x3 standard size convolution kernel, and uses the convolution coefficient w(p n ) performs convolution operation and learns adaptive coefficient v(p0+p n ), if the convolution coefficient w(p n ) is valid, then v(p0+p n ) is 1, otherwise it is 0;

[0034] The shape adaptive convolution is aggregated in the feature fusion network, which includes a trunk branch and a residual side branch. The trunk branch contains N times of stacking of the residual structure Bottleneck. The residual side branch performs shape adaptive convolution operation, and finally the trunk branch and the residual side branch are connected for output.

[0035] In the prediction layer, detection heads of different scales are used to obtain two-dimensional detection results I 2d .

[0036] Furthermore, in step S2, a loss function of the tea 2D image target detection model TNet is constructed, which is a tea attribute perception enhancement loss function, and the process is as follows:

[0037] First, construct the class attribute perception enhancement function f confidence , by designing the scale parameter μ, the focus parameter α and the adjustment factor τ to deal with the uncertainty of tea attributes in complex environments:

[0038] f confidence =μ|IOU-0.5| α +τ

[0039] Among them, IOU represents the intersection-over-union ratio of each predicted box sample and the corresponding annotation box;

[0040] Then the cross entropy loss is used to construct the final tea attribute perception enhancement loss function L Confidence :

[0041]

[0042] in, and Both represent indicator variables. When the j-th bounding box of cell i is responsible for predicting the object, The value of is 1, The value of is 0, otherwise The value of is 0, The value of c is 1; i represents the confidence label value of cell i, represents the confidence prediction score corresponding to the maximum IOU value between the prediction box and the annotation box of cell i, λ represents the scale parameter of the class attribute, ε represents the adjustment factor of the class attribute, and γ p , γ n are the focusing parameters of positive and negative samples respectively, S represents the total number of grids divided into feature maps, and B represents the number of bounding boxes generated for each grid unit.

[0043] Further, in step S2, the point cloud region candidate network algorithm is used to obtain the tea 3D detection frame J from the 3D point cloud. 3d The process includes:

[0044] Firstly, the input tea 3D point cloud data is processed in blocks through a sliding window of fixed size;

[0045] The irregular point cloud data is then converted into a regular voxel representation;

[0046] Then, candidate boxes of regions where objects may exist are generated, and the features of candidate boxes of each point cloud region are extracted through pooling and convolution operations.

[0047] Finally, target detection and classification are performed on each candidate area to predict the final three-dimensional detection result of tea leaves.

[0048] Further, in step S3, the two-dimensional detection result I of tea leaves is 2d and tea leaves 3D detection results J 3d Matching includes:

[0049] S31, back-projecting the three-dimensional detection result of tea leaves onto the image plane to obtain a two-dimensional detection frame J 2d ;

[0050] S32, characterize the paired frames, two-dimensional detection frame I 2d The i-th box and the two-dimensional detection box J 2d The pairwise box representation function F of the j-th box in i,j As shown below:

[0051] F i,j ={IOU i,j ,S i ,S j},IOU i,j >0,S i ,S j >T s ,i=1,2,...,I,j=1,2,...,J

[0052] Where IOU i,j Represents the two-dimensional detection box J 2d The jth box in is relative to the two-dimensional detection box I 2d The intersection-over-union ratio of the i-th box in i , S j Represents the two-dimensional detection box I 2d The i-th box and the two-dimensional detection box J in 2d The semantic score of the jth box in T s represents the preset semantic score threshold, I and J represent I 2d and J 2d The total number of IOU i,j ≤0, or S i ≤T s , or S j≤T s The boxes will be filtered;

[0053] S32, adaptive reconstruction of two-dimensional detection frame J 2d The semantic score value is expressed as:

[0054]

[0055] In the above formula, firstly, IOU i,j As a spatial constraint factor to make conditional judgments:

[0056] If IOU i,j Less than threshold T or equal to IOU max , then the two-dimensional detection box J 2d The semantic score S of the jth box in j Will be detected by the two-dimensional frame I 2d The semantic score S of the i-th box in i Constrained: If S i -T c >0, indicating that the two-dimensional detection box J 2d The jth box in is a positive sample, and its semantic score is S j It is enhanced by positive weights, otherwise it is judged as a negative sample, and its semantic score S j Suppressed by inverse weighting;

[0057] If IOU i,j is greater than or equal to the threshold T, then the two-dimensional detection box J 2d The semantic score S of the jth box in j Suppressed directly by reverse weight;

[0058] S33, according to the adaptively adjusted S′ j Re-characterization I 2d and J 2d The paired boxes in , filter out J 2d Medium S j ≤T s The box gets the updated J′ 2d ; Then recalculate the two-dimensional detection box I based on this representation set 2d The i-th box in is relative to the two-dimensional detection box J′ 2d The intersection-and-union ratio of the jth box in j to I 2d The semantic score S of each box in i Adaptively adjust to achieve cross suppression; then re-characterize one by one to filter out I 2d Medium S i ≤T s The box gets the updated I′ 2d, and then select the two boxes with the largest intersection-and-union ratio as the valid boxes corresponding to each other as the one-to-one corresponding I2″ d and J2″ d Finally, J2″ d and J 3d Use back projection to perform corresponding processing and filter out J 3d Errors and extra boxes.

[0059] Further, in step S4, the following steps are included:

[0060] S41. Use the tea key point detection model TKPNet to perform key point detection on the matched tea two-dimensional detection frame, and extract the two-dimensional coordinates of several key points on the tea target. The tea key point detection model TKPNet is constructed in the form of an encoder-decoder, and a probability and offset representation model is used to describe the position coordinates of the marked tea key points. The probability representation is shown in the following formula:

[0061]

[0062] In the above formula, if the i pixel coordinates x i Corresponding to the kth key point coordinate l k If the distance is less than or equal to R, then the value f(x i ) is 1, otherwise it is 0; R represents the preset circle radius;

[0063] The offset is expressed as follows:

[0064] D k (x i )=l k -x i

[0065] It represents the coordinate x of the i-th pixel point i Corresponding to the kth key point coordinate l k The two-dimensional offset vector D between k (x i );

[0066] Then the representation model of probability and offset is expressed as:

[0067]

[0068] Where P(·) is a bilinear interpolation function, R represents the preset circle radius, and x i,j Indicates x i is the coordinate of the jth pixel in a circle with R as the radius and Q as the center. i is the total number of pixels in a circle with R as the radius and F k (x i) represents the current i-th pixel coordinate x i The description value corresponding to the probability + offset of the kth key point;

[0069] S42, establish the loss function of the tea key point detection model TKPNet, where the loss function includes a binary cross entropy loss based on key points and an offset regression loss, which is specifically expressed as:

[0070] L=λ f L f +λ d L d

[0071] L f Represents the binary cross entropy loss, which is specifically:

[0072]

[0073] L d represents the offset regression loss, which is specifically:

[0074]

[0075] Among them, λ f , d L f and L d The weight parameter of the loss, N represents the total number of pixels, f(x i ), Represents the coordinates x i The true probability value and predicted probability value, S(·) represents the Smooth smoothing loss, x i , l k They represent the coordinates of the i-th pixel point and the k-th key point respectively, and K represents the total number of key points;

[0076] S43, fusing the acquired two-dimensional key points of the two-dimensional detection results with the corresponding three-dimensional point cloud detection results, and using the key points to guide the three-dimensional point cloud clustering, which includes:

[0077] Project the acquired coordinates of several two-dimensional key points into the three-dimensional coordinate system of the laser radar;

[0078] Position the initial cluster center using the projected coordinates of several two-dimensional key points;

[0079] The point cloud is divided into multiple clusters using the Euclidean clustering algorithm;

[0080] The point cloud clusters containing several key points are extracted as the final tea clustering results.

[0081] Further, in step S41, the extracted key point coordinates include at least the left leaf tip, the right leaf tip and the intersection of the rhizome, which correspond to k=1, 2, 3 respectively.

[0082] Further, in step S5, the following steps are included:

[0083] S51, project the extracted tea stem intersection key points into the laser radar coordinate system to obtain the three-dimensional coordinates Then, the tea point cloud clustering result is clustered along the Z-axis direction of the radar coordinate system with the three-dimensional coordinate point as the starting point to obtain the local point cloud p of the tea stem;

[0084] S52. Obtain the posture of the tea stem point cloud p using the tea stem posture analysis method: Calculate the covariance matrix C of the three-dimensional coordinates of the tea stem point cloud:

[0085]

[0086] in x i ,y i 、z i Respectively represent the coordinate values ​​of the three dimensions of point i, They represent the mean coordinates of the point cloud in three dimensions, and n represents the total number of point clouds;

[0087] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, and λ3, where λ1≥λ2≥λ3, and obtain the eigenvector corresponding to λ1 This is the direction of stem growth;

[0088] S53, determining the origin of the stem growth direction: determining the normal plane at the intersection through the three-dimensional coordinate point k of the stem intersection and the stem growth direction η, the plane equation is as follows:

[0089] δ1x+δ2y+δ3z-(δ1μ1+δ2μ2+δ3μ3)=0

[0090] Where (x, y, z) represents any point on the plane;

[0091] Take the coordinate mean of the tea stem point cloud falling on the normal plane as the origin of the tea growth direction O = (x o ,y o ,z o );

[0092] S54, obtaining the final picking position and picking angle: Calculate the stem length h by the following formula:

[0093]

[0094] Among them, (xo ,y o ,z o ), (x l ,y l ,z l ) represent the origin of the stem point cloud and the coordinates of the lowest point of the stem point cloud respectively;

[0095] Then at the origin O=(x o ,y o ,z o ) moves λh in the opposite direction of the tea stem growth direction η, where λ is the proportional coefficient, and obtains the three-dimensional picking point coordinates in the radar coordinate system

[0096] Finally, the picking angle θ is calculated by the following formula:

[0097]

[0098] in, is the normal vector of the horizontal plane

[0099] The beneficial effects of the present invention are:

[0100] 1) The present invention utilizes the RGB-D component information of tea leaves in a complex environment, removes the redundant R component, constructs a variance joint estimation function with the GB-D dual components, and effectively removes the redundant background points of the tea leaves RGB-D image and the three-dimensional point cloud, thereby effectively improving the detection accuracy as a preprocessing;

[0101] 2) Compared with the tea detection method based on a single sensor, the present invention obtains more accurate and robust detection results by integrating RGB-D camera and LiDAR. In the image detection stage, the SAConv-Bottleneck constructed by shape adaptive convolution, small target detection head and class attribute perception enhancement loss function are used to obtain accurate two-dimensional detection results; in the fusion stage, the adaptive cross non-maximum suppression method is used for post-fusion to obtain accurate two-dimensional and three-dimensional detection results of famous and high-quality tea leaves;

[0102] 3) The present invention adopts a multi-stage detection and posture analysis method. In the first stage, target fusion detection is performed on the tea leaves to obtain fused detection results. In the second stage, based on the first stage, key point detection is performed on the target detection results, and the primary positioning of the tea picking points is performed using the probability + offset key point description method. This key point is then used to guide the clustering of the three-dimensional fusion detection results of the tea leaves to remove outliers. The stems are then cut off at the key points and their postures are analyzed. Finally, not only the accurate tea picking position but also the picking angle can be obtained, thereby achieving simple, reliable and accurate intelligent picking of high-quality teas.

[0103] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0105] Figure 1 This is a main flow chart of the method for picking high-quality tea according to an embodiment of the present invention;

[0106] Figure 2 This is a schematic diagram of the installation of the device according to an embodiment of the present invention;

[0107] Figure 3 The TNet network diagram described in the embodiment of the present invention;

[0108] Figure 4 A TKPNet network diagram according to an embodiment of the present invention;

[0109] Figure 5 This is a diagram showing the process of obtaining the picking position and angle through posture analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0110] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0111] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0112] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0113] See also Figure 1 to Figure 5 , which is a method for picking high-quality tea with an intelligent robot based on multi-stage fusion detection and posture analysis.

[0114] Example

[0115] This embodiment proposes a method for picking high-quality tea with an intelligent robot based on multi-stage fusion detection and posture analysis. Figure 1 As shown, it at least includes mounting an RGB-D camera and a laser radar on an intelligent robot, and calculating the picking position and angle according to the tea information data obtained by the RGB-D camera and the laser radar, which includes the following steps:

[0116] S1. Using GB-D dual-component joint preprocessing method to filter out background points from the acquired RGB-D image and three-dimensional point cloud of tea leaves;

[0117] S2. Establish a tea 2D image target detection model to obtain the tea 2D detection frame from the RGB-D image, and use the point cloud region candidate network algorithm to obtain the tea 3D detection frame from the 3D point cloud;

[0118] S3, using the semantic and spatial multi-constrained adaptive cross non-maximum suppression method to match the two-dimensional detection frame and the three-dimensional detection frame to obtain a one-to-one corresponding two-dimensional tea detection result and a three-dimensional tea detection result;

[0119] S4, using the tea key point detection model to extract the positions of several tea 2D key points from the tea 2D detection results, and integrating the 2D key points into the corresponding tea 3D detection point cloud to guide 3D point cloud clustering, and outputting the tea 3D point cloud clustering results;

[0120] S5. Separate the local point cloud of the stem from the clustering result of the tea 3D point cloud, and then obtain the posture and length of the stem through the stem posture analysis method, and finally obtain the picking position and picking angle of the tea on the stem.

[0121] Step S1: The intelligent robot uses an RGB-D camera and a laser radar to obtain an RGB-D image and a 3D point cloud of tea leaves. A GB-D dual-component joint preprocessing method is proposed based on the RGB-D image data: First, a variance joint estimation function is constructed using the GB component map and the D component map, and then the function is used to obtain the optimal segmentation threshold. Finally, the threshold is used to filter out most of the background points in the RGB-D image and the 3D point cloud at the same time. The specific steps include:

[0122] 101: As Figure 2 As shown, an RGB-D camera is installed at the end of the robotic arm of the intelligent robot to collect RGB-D images of the tea leaves in front; a laser radar is installed on the top of the intelligent robot to collect three-dimensional point cloud data of the surrounding tea leaves.

[0123] 102: A GB-D dual-component joint preprocessing method is proposed to filter out background points from RGB-D images and point cloud data at the same time. First, the GB component map and the depth D component map in the RGB-D image data are extracted, and then a variance joint estimation function is constructed based on the two component maps, as shown in formula (1). The variance values ​​of the pixels in the image under different thresholds are jointly estimated using the two component maps, and then the variance joint estimation function is used to obtain the optimal segmentation threshold. As shown in formula (2), formula (3) is finally used for segmentation, and then acts on the original RGB-D image to filter out most of the background points.

[0124]

[0125]

[0126] where λ gb , d Respectively represent the weight coefficients of GB component map and D component map, They represent the probability that the pixel in the GB component image is the foreground and background when the threshold is t, They represent the mean of the foreground and background pixels in the GB component image when the threshold is t, They represent the probability that the pixel in the D component image is the foreground and background when the threshold is t, They represent the mean of the foreground and background pixels of the D component image when the threshold is t, L(t) represents the variance value when the threshold is t, f(i,j) represents the pixel value at the pixel point (i,j), Represents the best segmentation threshold.

[0127] 103: For the pixel point (i, j) with a pixel value of 0 obtained by equation (3) in 102, it is represented as a background point. It is transformed into a three-dimensional point coordinate in the laser radar coordinate system using equation (4), and equation (5) is used as a criterion: If dis m,nIf it is smaller than the preset threshold d, it indicates that point m in the three-dimensional point cloud of 101 is a background point and is filtered out; otherwise, it is retained, thereby filtering out most of the background points from the three-dimensional point cloud of tea leaves.

[0128]

[0129] in They represent the 2D coordinates of the background point (i, j) in the image coordinate system and the 3D coordinates of the background point (i, j) in the lidar coordinate system. i,j Represents the depth value of the pixel coordinate (i, j) in the D component image, f X 、f Y Respectively represent the equivalent focal lengths in the X and Y directions in the camera coordinate system, u0 and v0 represent the coordinates of the center point of the image plane, R and t represent the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system, (x m ,y m ,z m )、(x n ,y n ,z n ) represent the 3D coordinates of the mth point in the point cloud 101 and the 3D coordinates of the nth point in the point cloud after projection transformation of the background pixels of the image. The total number of points in the two point clouds is M and N respectively. m,n represents the distance between point m and point n, and d represents the preset minimum distance threshold.

[0130] Step S2: First stage detection: Build a tea detection model to detect tea. On the one hand, build an RGB tea image detection model TNet. Figure 3 As shown in the figure, shape adaptive convolution and small target detection head are proposed in TNet to build a tea small target feature aggregation model, and then a class attribute perception enhancement loss function is proposed to enhance the class attributes, and finally the two-dimensional detection frame of the tea in the RGB-D image is effectively obtained; on the other hand, for the pre-processed three-dimensional point cloud data, the PointR-CNN algorithm is used to obtain the three-dimensional detection frame of the tea in the point cloud. The specific steps include:

[0131] 201: First stage detection: target detection. Build a target detection model to detect tea leaves, including a two-dimensional image target detection model and a three-dimensional point cloud target detection model. For the RGB-D image preprocessed in step S1, build a two-dimensional image target detection model TNet for tea leaves to obtain the two-dimensional detection result I of tea leaves in the image coordinate system. 2dTNet consists of a backbone network, a feature fusion network, and a prediction layer. The backbone network extracts preliminary features, and then proposes shape adaptive convolution in the feature fusion network to aggregate and enhance features. Different shape convolution parameters are learned for the convolution kernel to adapt to the growth direction of tea stems and buds, as shown in formula (6). The shape adaptive convolution uses a 3x3 standard size convolution kernel, and uses the convolution coefficient w(p n ) performs convolution operation and learns adaptive coefficient v(p0+p n ) to adapt to the current shape, if the convolution coefficient w(p n ) is valid, then v(p0+p n ) is 1, otherwise it is 0. Then the shape adaptive convolution is embedded in the Bottleneck model to aggregate features, and the SAConv-Bottleneck network module is proposed. This network module has two branches, one is the main branch, which contains N stacks of the residual structure Bottleneck; the other branch is the residual side branch, which only performs a simple shape adaptive convolution operation. Finally, the main branch and the residual side branch are connected for output, thereby realizing the aggregation of small tea target features and effectively adapting to the feature extraction of irregularly shaped tea leaves.

[0132]

[0133] where v(p0+p n ) means that at p0+p n The shape adaptation coefficient at the position, w(p n ) means in p n The convolution coefficient of the direction, x(p0+p n ) indicates that in the feature map p0+p n The eigenvalue at the position, y(p0) represents the feature extraction result of the 9 points around the position p0, and R represents the coordinate direction set of the nine points around it.

[0134] 202: On the prediction layer of TNet, detection heads of different scales are designed to obtain two-dimensional detection results I 2d , I 2d It contains the position coordinates and semantic score of each 2D detection frame. Since tea leaves are small targets, in order to reduce the receptive field of the final output feature map, a larger-scale 160×160 pixel detection head is proposed. The 160×160 pixel mapping to the original image can contain all the information of the 32×32 pixel small target, and effectively output the detection frame of the tea leaf small target.

[0135] 203: In TNet, a tea attribute perception enhancement loss function is proposed. First, the attribute perception enhancement function f is constructed as shown in formula (7): confidence , by designing the scale parameter μ, the focus parameter α and the adjustment factor τ to deal with the uncertainty of tea attributes in complex environments when the IOU is around 0.5. Then the cross entropy loss is used to construct the final tea attribute perception enhancement loss function L Confidence , as shown in formula (8):

[0136] f confidence =μ|IOU-0.5| α +τ (7)

[0137]

[0138] The IOU represents the intersection-over-union ratio of each predicted box sample to the corresponding annotation box. and Both represent indicator variables. When the j-th bounding box of cell i is responsible for predicting the object, The value of is 1, The value of is 0, otherwise The value of is 0, The value of c is 1, i represents the confidence label value of cell i, represents the confidence prediction score corresponding to the maximum IOU value between the prediction box and the annotation box of cell i, λ represents the scale parameter of the class attribute, and its value is negative, ε represents the adjustment factor of the class attribute, and γ p , γ n are the focusing parameters of positive and negative samples respectively, S represents the total number of grids divided into feature maps, and B represents the number of bounding boxes generated for each grid unit.

[0139] 204: For the tea 3D point cloud data preprocessed in step S1, use the Point R-CNN algorithm to obtain its 3D detection result J in the radar coordinate system 3d : First, the input tea 3D point cloud data is divided into blocks through a sliding window of a fixed size, and then the irregular point cloud data is converted into regular voxel representation to facilitate the subsequent extraction of point cloud features. Then, candidate boxes of areas where targets may exist are generated. Through pooling and convolution operations, the features of each point cloud area candidate box are extracted. Finally, target detection and classification are performed on each candidate area to predict the final tea 3D detection result.

[0140] Step S3: Propose a semantic and spatial multi-constrained adaptive cross non-maximum suppression method to match the two detection results obtained in step S2, filter out erroneous and redundant detection results, and finally obtain one-to-one corresponding tea 2D detection results and tea 3D detection results. The specific steps include:

[0141] 301: A semantic and spatial multi-constrained adaptive cross non-maximum suppression method is proposed to improve the tea two-dimensional detection result I obtained in step S2. 2d and tea leaves 3D detection results J 3d Valid matches, where I 2d and J 3d They represent the set of tea 2D detection frames and 3D detection frames respectively. Each detection frame is represented by the coordinate position of the frame and the semantic score S. First, the 3D detection result of tea is back-projected onto the image plane using formula (4) to obtain the 2D detection frame J 2d , and then characterize the paired boxes, the two-dimensional detection box I 2d The i-th box and the two-dimensional detection box J 2d The pairwise box representation function F of the j-th box in i,j As shown in formula (9), where IOU i,j ≤0, or S i ≤T s , or S j ≤T s The boxes will be filtered:

[0142] F i,j ={IOU i,j ,S i ,S j},IOU i,j >0,S i ,S j >T s ,i=1,2,...,I,j=1,2,...,J(9)where IOU i,j Represents the two-dimensional detection box J 2d The jth box in is relative to the two-dimensional detection box I 2d The intersection-over-union ratio of the i-th box in i , S j Represents the two-dimensional detection box I 2d The i-th box and the two-dimensional detection box J in 2d The semantic score of the jth box in T s represents the preset semantic score threshold, I and J represent I 2d and J 2d The total number of .

[0143] 302: Adaptively reconstruct the two-dimensional detection frame J using formula (10) 2d The semantic score value of i,j As a spatial constraint factor for conditional judgment, if IOU i,j Less than threshold T or equal to IOU max , then the two-dimensional detection box J 2dThe semantic score S of the jth box in j Will be detected by the two-dimensional frame I 2d The semantic score S of the i-th box in i Constrained: If S i -T c >0, indicating that the two-dimensional detection box J 2d The jth box in is a positive sample, and its semantic score is S j It is enhanced by positive weights, otherwise it is judged as a negative sample, and its semantic score S j Suppressed by reverse weight; if IOU i,j is greater than or equal to the threshold T, then the two-dimensional detection box J 2d The semantic score S of the jth box in j Directly suppress it through reverse weights.

[0144]

[0145] Where λ and ε represent the scale parameter and adjustment factor respectively, S i Represents a two-dimensional detection box I 2d The semantic score of the i-th box, IOU max Indicates I 2d All boxes with J 2d The maximum value of the IOU of the jth box in, S′ j Represents J after adaptive adjustment 2d The semantic score of the jth box in , T represents the IOU threshold, T c Represents the positive and negative sample threshold.

[0146] 303: Using 302 to obtain the adaptively adjusted S′ j Then, we use formula (9) to represent I again 2d and J 2d The paired boxes in , thereby filtering out J 2d Medium S j ≤T s The updated J′ is obtained 2d , and then recalculate the two-dimensional detection box I based on this representation set 2d The i-th box in is relative to the two-dimensional detection box J′ 2d The intersection-over-union ratio of the j-th box in is then used to calculate the intersection-over-union ratio of the j j And formula (10) for I 2d The semantic score S of each box in i Adaptive adjustment is performed to achieve cross suppression, and then the I 2d Medium S i ≤T s The updated I′ is obtained 2d, and then select the two boxes with the largest intersection-and-union ratio as the valid boxes corresponding to each other, and then we can get the updated correct and one-to-one corresponding I″ 2d and J″ 2d , and finally J″ 2d and J 3d Use back projection to perform corresponding processing and filter out J 3d The errors and redundant frames are removed, and finally the effective matching of the two-dimensional detection results and the three-dimensional detection results is achieved.

[0147] Step S4: Second stage detection: Based on the target fusion detection results obtained in the first stage, a tea key point detection model TKPNet is built to perform key point detection on the image in the two-dimensional frame of tea leaves in step S3. A probability + offset key point representation model is proposed in TKPNet to effectively obtain the positions of the three two-dimensional key points of tea leaves, and then the two-dimensional key points are fused into the corresponding three-dimensional point cloud of tea leaves in step S3, and the two-dimensional key points are used to guide the clustering of the three-dimensional point cloud, so as to obtain the clustering result of the three-dimensional point cloud of tea leaves after the fusion of key points. The specific steps include:

[0148] 401: Second stage detection: key point detection. Build the tea key point detection model TKPNet to perform key point detection on the fused tea 2D frame in step S3, and finally obtain the 2D coordinates of the three key points on the tea target. Figure 4 As shown in Figure 1, TKPNet is constructed in the form of an encoder-decoder, and a probability and offset representation model is proposed to describe the position coordinates of the annotated tea key points. The probability representation is shown in formula (11): In the probability map, if the coordinates of the i-th pixel point x i Corresponding to the kth key point coordinate l k If the distance is less than or equal to R, then the value f(x i ) is 1, otherwise it is 0, where k∈{1,2,3}, the three key points are 1: left leaf tip, 2: right leaf tip, 3: root-stem intersection. The offset is represented by the coordinate x of the ith pixel in the probability map i Corresponding to the kth key point coordinate l k The two-dimensional offset vector D between k (x i ), as shown in formula (12). Finally, it is described in the form of probability and offset, as shown in formula (13):

[0149]

[0150] D k (x i )=l k -x i (12)

[0151]

[0152] Where P(·) is a bilinear interpolation function, R represents the preset circle radius, and x i,j Indicates x i is the center of the circle, R is the radius of the circle, R represents the preset radius of the circle, Q represents the x- i is the center of the circle, R is the total number of pixels in the circle with radius, F k (x i ) represents the current i-th pixel coordinate x i The probability + offset corresponding to the k-th keypoint.

[0153] 402: The probability + offset loss function of TKPNet is proposed. Its loss function consists of two parts, namely the binary cross entropy loss based on key points and the offset regression loss, as shown in formula (14):

[0154] L=λ f L f +λ d L d (14)

[0155] in λ f , d They are the binary cross entropy loss L f and the offset regression loss L d The weight parameter, L represents the loss function of TKPNet in the form of probability + offset, N represents the total number of pixels, f(x i ), Represents the coordinates x i The true probability value and predicted probability value, S(·) represents the Smooth smoothing loss, x i , l k They represent the coordinates of the i-th pixel point and the k-th key point respectively, and K represents the total number of key points, which is 3.

[0156] 403: Fuse the two-dimensional key points with the three-dimensional point cloud detection results matched in step S3, use the key points to guide the three-dimensional point cloud clustering, and remove outliers. First, the three two-dimensional key points obtained in 402 are projected into the radar coordinate system using equation (4), and then the three key points are set as the initial cluster centers to provide accurate prior information for clustering. Then, the point cloud is divided into multiple clusters using the Euclidean clustering algorithm, and finally the point cloud cluster containing the three key points is extracted as the final tea point cloud clustering result.

[0157] Step S5: Project the coordinates of the key points of the stalk intersection in step S4 to the three-dimensional point cloud of the tea leaves clustered in step S4 using spatial transformation, and cut downward at the point position to separate the local point cloud of the stalk. Then, the posture and length of the stalk are obtained through the stalk posture analysis method, and finally the picking position and picking angle of the famous tea on the stalk are obtained, so as to realize intelligent picking, such as Figure 5 As shown, the specific steps include:

[0158] 501: Project the third key point obtained in step S3, i.e., the intersection of the tea stems, using equation (4) to obtain the three-dimensional coordinates of the point in the radar coordinate system. Then, the tea point cloud clustering result obtained in step S4 is cut downward in the Z-axis direction of the radar coordinate system at the position of the point, and only the local point cloud p of the tea stem is obtained.

[0159] 502: Use the tea stem posture analysis method to obtain the posture of the stem point cloud p. First, use formula (15) to calculate the covariance matrix C of the three-dimensional coordinates of the tea stem point cloud, and then perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, λ3, where λ1≥λ2≥λ3, and obtain the eigenvector corresponding to λ1 This is the direction in which the stem grows.

[0160]

[0161] in x i ,y i 、z i Respectively represent the coordinate values ​​of the three dimensions of point i, They represent the mean coordinates of the point cloud in three dimensions, n represents the total number of point clouds, and C is the covariance matrix of the point cloud.

[0162] 503: Determine the origin of the stem growth direction. First, use the three-dimensional coordinate point k of the stem intersection and the stem growth direction η to determine the normal plane at the intersection. The plane equation is shown in equation (16). Then, take the coordinate mean of the tea stem point cloud falling on the normal plane as the origin of the tea growth direction O = (x o ,y o ,z o ).

[0163] δ1x+δ2y+δ3z-(δ1μ1+δ2μ2+δ3μ3)=0 (16)

[0164] Where (x, y, z) represents any point on the plane. represents the three-dimensional coordinates of the key points of the stem intersection, Indicates the direction of stem growth.

[0165] 504: Obtain the final picking position and picking angle. First, obtain the stem length h through equation (17), and then at the origin O = (x o ,y o ,z o ) moves λh in the opposite direction of the tea stem growth direction η, where λ is the proportional coefficient, and the three-dimensional picking point coordinates in the radar coordinate system can be obtained. Finally, the accurate picking angle θ can be obtained through formula (18).

[0166]

[0167]

[0168] Where (x o ,y o ,z o ), (x l ,y l ,z l ) represent the origin and the lowest point coordinates of the stem point cloud, h represents the stem length, is the normal vector of the horizontal plane θ represents the obtained picking angle.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for picking high-quality tea with an intelligent robot based on multi-stage fusion detection and posture analysis, characterized in that: At least the RGB-D camera and the laser radar are mounted on the intelligent robot, and the picking position and angle are calculated according to the tea information data obtained by the RGB-D camera and the laser radar, which includes the following steps: S1. Using GB-D dual-component joint preprocessing method to filter out background points from the acquired RGB-D image and three-dimensional point cloud of tea leaves; S2. Establish a tea 2D image target detection model to obtain the tea 2D detection frame from the RGB-D image, and use the point cloud region candidate network algorithm to obtain the tea 3D detection frame from the 3D point cloud; S3, using the semantic and spatial multi-constrained adaptive cross non-maximum suppression method to match the two-dimensional detection frame and the three-dimensional detection frame to obtain a one-to-one corresponding two-dimensional tea detection result and a three-dimensional tea detection result; S4, using the tea key point detection model to extract the positions of several tea 2D key points from the tea 2D detection results, and integrating the 2D key points into the corresponding tea 3D detection point cloud to guide 3D point cloud clustering, and outputting the tea 3D point cloud clustering results; S5. Separate the local point cloud of the stem from the clustering result of the tea 3D point cloud, and then obtain the posture and length of the stem through the stem posture analysis method, and finally obtain the picking position and picking angle of the tea on the stem.

2. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1 is characterized in that: In step S1, the process of filtering out the background points of the RGB-D image includes the following steps: S111, extracting the GB component map and the depth D component map in the RGB-D image data, constructing a variance joint estimation function according to the two component maps, and jointly estimating the variance values ​​of the pixels in the image under different thresholds according to the two component maps, which is expressed as: Among them, λ gb , d Respectively represent the weight coefficients of GB component map and D component map, They represent the probability that the pixel in the GB component image is the foreground and background when the threshold is t, They represent the mean of the foreground and background pixels in the GB component image when the threshold is t, They represent the probability that the pixel in the D component image is the foreground and background when the threshold is t, They represent the mean of the pixel points of the D component image as foreground and background when the threshold is t, and L(t) represents the variance value when the threshold is t; S112, using variance joint estimation function to obtain the best segmentation threshold It is expressed as: S113, the best segmentation threshold is obtained The background points of the original RGB-D image are filtered out, which is expressed as: Among them, f(i,j) represents the pixel value at the pixel point (i,j), Represents the optimal segmentation threshold; the pixel point (i, j) with a pixel value of 0 represents the background point, which is filtered out.

3. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 2 is characterized in that: The process of filtering out background points from point cloud data includes the following steps: S121. Transform the pixel points in the RGB-D image to the three-dimensional point coordinates in the laser radar coordinate system: in They represent the 2D coordinates of the background point (i, j) in the image coordinate system and the 3D coordinates of the background point (i, j) in the lidar coordinate system. i,j Represents the depth value of the pixel coordinate (i, j) in the D component image, f X 、f Y Respectively represent the equivalent focal lengths in the X and Y directions in the camera coordinate system, u0 and v0 represent the coordinates of the center point of the image plane, and R and t represent the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system; S122, calculate the distance dis between the mth point in the original three-dimensional point cloud and the nth point after the background point with a pixel value of 0 in the RGB-D image is converted into a three-dimensional point cloud m,n : Among them, (x m ,y m ,z m )、(x n ,y n ,z n ) respectively represent the three-dimensional coordinates of the m-th point in the point cloud 101 and the three-dimensional coordinates of the n-th point in the point cloud after the projection transformation of the image background pixels, and the total number of the two in the point cloud is M and N respectively; S123, set threshold d, if dis m,n If it is smaller than the preset threshold d, it indicates that point m in the original 3D point cloud is a background point and it is filtered out; otherwise it is retained.

4. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1 is characterized in that: In step S2, the preprocessed RGB-D image is subjected to a two-dimensional detection frame extraction, which includes: building a tea two-dimensional image target detection model TNet, obtaining a two-dimensional detection result I of the tea in the image coordinate system 2d ; The tea 2D image target detection model TNet includes a backbone network, a feature fusion network, and a prediction layer, among which, Preliminary feature extraction is performed in the backbone network. First, different shape convolution parameters are learned for the convolution kernel to adapt to the growth direction of tea stems and buds, which is expressed as: where v(p0+p n ) means that at p0+p n The shape adaptation coefficient at the position, w(p n ) indicates that in p n The convolution coefficient of the direction, x(p0+p n ) indicates that in the feature map p0+p n The feature value at the position, y(p0) represents the feature extraction result of several points around the position p0, and R represents the coordinate direction set of several points around it; the shape adaptive convolution uses a 3x3 standard size convolution kernel, and uses the convolution coefficient w(p n ) performs convolution operation and learns adaptive coefficient v(p0+p n ), if the convolution coefficient w(p n ) is valid, then v(p0+p n ) is 1, otherwise it is 0; The shape adaptive convolution is aggregated in the feature fusion network, which includes a trunk branch and a residual side branch. The trunk branch contains N times of stacking of the residual structure Bottleneck. The residual side branch performs shape adaptive convolution operation, and finally the trunk branch and the residual side branch are connected for output. In the prediction layer, detection heads of different scales are used to obtain the two-dimensional detection results I 2d .

5. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 4 is characterized in that: In step S2, the loss function of the tea 2D image target detection model TNet is constructed, which is a tea attribute perception enhancement loss function, and the process is as follows: First, construct the class attribute perception enhancement function f confidence , by designing the scale parameter μ, the focus parameter α and the adjustment factor τ to deal with the uncertainty of tea attributes in complex environments: f confidence =μ|IOU-0.5| α +t Among them, IOU represents the intersection-over-union ratio of each predicted box sample and the corresponding annotation box; Then the cross entropy loss is used to construct the final tea attribute perception enhancement loss function L Confidence : in, and Both represent indicator variables. When the j-th bounding box of cell i is responsible for predicting the object, The value of is 1, The value of is 0, otherwise The value of is 0, The value of c is 1; i represents the confidence label value of cell i, represents the confidence prediction score corresponding to the maximum IOU value between the prediction box and the annotation box of cell i, λ represents the scale parameter of the class attribute, ε represents the adjustment factor of the class attribute, and γ p , γ n are the focusing parameters of positive and negative samples respectively, S represents the total number of grids divided into feature maps, and B represents the number of bounding boxes generated for each grid unit.

6. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1 is characterized in that: In step S2, the point cloud region candidate network algorithm is used to obtain the tea 3D detection box J from the 3D point cloud. 3d The process includes: Firstly, the input tea 3D point cloud data is processed in blocks through a sliding window of fixed size; The irregular point cloud data is then converted into a regular voxel representation; Then, candidate boxes of regions where objects may exist are generated, and the features of candidate boxes of each point cloud region are extracted through pooling and convolution operations. Finally, target detection and classification are performed on each candidate area to predict the final three-dimensional detection result of tea leaves.

7. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1 is characterized in that: In step S3, the two-dimensional detection result I of tea leaves is 2d and tea leaves 3D detection results J 3d Matching includes: S31, back-projecting the three-dimensional detection result of tea leaves onto the image plane to obtain a two-dimensional detection frame J 2d ; S32, characterize the paired frames, two-dimensional detection frame I 2d The i-th box and the two-dimensional detection box J 2d The pairwise box representation function F of the j-th box in i,j As shown below: F i,j ={IOU i,j ,S i ,S j },IOU i,j >0,S i ,S j >T s ,i=1,2,...,I,j=1,2,...,Jwhere IOU i,j Represents the two-dimensional detection box J 2d The jth box in is relative to the two-dimensional detection box I 2d The intersection-over-union ratio of the i-th box in i , S j Represents the two-dimensional detection box I 2d The i-th box and the two-dimensional detection box J in 2d The semantic score of the jth box in T s represents the preset semantic score threshold, I and J represent I 2d and J 2d The total number of IOU i,j ≤0, or S i ≤T s , or S j ≤T s The boxes will be filtered; S32, adaptive reconstruction of two-dimensional detection frame J 2d The semantic score value is expressed as: In the above formula, firstly, IOU i,j As a spatial constraint factor to make conditional judgments: If IOU i,j Less than threshold T or equal to IOU max , then the two-dimensional detection box J 2d The semantic score S of the jth box in j Will be detected by the two-dimensional frame I 2d The semantic score S of the i-th box in i Constrained: If S i -T c >0, indicating that the two-dimensional detection box J 2d The jth box in is a positive sample, and its semantic score is S j It is enhanced by positive weights, otherwise it is judged as a negative sample, and its semantic score S j Suppressed by inverse weighting; If IOU i,j is greater than or equal to the threshold T, then the two-dimensional detection box J 2d The semantic score S of the jth box in j Suppressed directly by reverse weight; S33, according to the adaptively adjusted S j 'Re-characterize I 2d and J 2d The paired boxes in , filter out J 2d Medium S j ≤T s The box gets the updated J2′ d ; Then recalculate the two-dimensional detection box I based on this representation set 2d The i-th box in is relative to the two-dimensional detection box J2′ d The intersection-and-union ratio of the jth box in j ′ to I 2d The semantic score S of each box in i Adaptively adjust to achieve cross suppression; then re-characterize one by one to filter out I 2d Medium S i ≤T s The box gets the updated I2′ d , and then select the two boxes with the largest intersection-and-union ratio as the valid boxes corresponding to each other as the one-to-one corresponding I2″ d and J2″ d Finally, J2″ d and J 3d Use back projection to perform corresponding processing and filter out J 3d Errors and extra boxes.

8. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1 is characterized in that: In step S4, the following steps are included: S41. Use the tea key point detection model TKPNet to perform key point detection on the matched tea two-dimensional detection frame, and extract the two-dimensional coordinates of several key points on the tea target. The tea key point detection model TKPNet is constructed in the form of an encoder-decoder, and a probability and offset representation model is used to describe the position coordinates of the marked tea key points. The probability representation is shown in the following formula: In the above formula, if the i pixel coordinates x i Corresponding to the kth key point coordinate l k If the distance is less than or equal to R, then the value f(x i ) is 1, otherwise it is 0; R represents the preset circle radius; The offset is expressed as follows: D k (x i )=l k -x i It represents the coordinate x of the i-th pixel point i Corresponding to the kth key point coordinate l k The two-dimensional offset vector D between k (x i ); Then the representation model of probability and offset is expressed as: Where P(·) is a bilinear interpolation function, R represents the preset circle radius, and x i,j Indicates x i is the coordinate of the jth pixel in a circle with R as the radius and Q as the center. i is the total number of pixels in a circle with R as the radius and F k (x i ) represents the current i-th pixel coordinate x i The description value corresponding to the probability + offset of the kth key point; S42, establish the loss function of the tea key point detection model TKPNet, where the loss function includes a binary cross entropy loss based on key points and an offset regression loss, which is specifically expressed as: L=λ f L f +λ d L d L f Represents the binary cross entropy loss, which is specifically: L d represents the offset regression loss, which is specifically: Among them, λ f , d L f and L d The weight parameter of the loss, N represents the total number of pixels, f(x i ), Represents the coordinates x i The true probability value and predicted probability value, S(·) represents the Smooth smoothing loss, x i , l k They represent the coordinates of the i-th pixel point and the k-th key point respectively, and K represents the total number of key points; S43, fusing the acquired two-dimensional key points of the two-dimensional detection results with the corresponding three-dimensional point cloud detection results, and using the key points to guide the three-dimensional point cloud clustering, which includes: Project the acquired coordinates of several two-dimensional key points into the three-dimensional coordinate system of the laser radar; Position the initial cluster center using the projected coordinates of several two-dimensional key points; The point cloud is divided into multiple clusters using the Euclidean clustering algorithm; The point cloud clusters containing several key points are extracted as the final tea clustering results.

9. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 8, characterized in that: In step S41, the extracted key point coordinates include at least the left leaf tip, the right leaf tip and the intersection of the rhizome, which correspond to k=1, 2, 3 respectively.

10. The method for picking high-quality tea using an intelligent robot based on multi-stage fusion detection and posture analysis according to claim 1, characterized in that: In step S5, the following steps are included: S51, project the extracted tea stem intersection key points into the laser radar coordinate system to obtain the three-dimensional coordinates Then, the tea point cloud clustering result is clustered along the Z-axis direction of the radar coordinate system with the three-dimensional coordinate point as the starting point to obtain the local point cloud p of the tea stem; S52. Obtain the posture of the tea stem point cloud p using the tea stem posture analysis method: Calculate the covariance matrix C of the three-dimensional coordinates of the tea stem point cloud: in x i ,y i 、z i Respectively represent the coordinate values ​​of the three dimensions of point i, They represent the mean coordinates of the point cloud in three dimensions, and n represents the total number of point clouds; Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​λ1, λ2, and λ3, where λ1≥λ2≥λ3, and obtain the eigenvector corresponding to λ1 This is the direction of stem growth; S53, determining the origin of the stem growth direction: determining the normal plane at the intersection through the three-dimensional coordinate point k of the stem intersection and the stem growth direction η, the plane equation is as follows: δ1x+δ2y+δ3z-(δ1μ1+δ2μ2+δ3μ3)=0 Where (x, y, z) represents any point on the plane; Take the coordinate mean of the tea stem point cloud falling on the normal plane as the origin of the tea growth direction O = (x o ,y o ,z o ); S54, obtaining the final picking position and picking angle: Calculate the stem length h by the following formula: Among them, (x o ,y o ,z o ), (x l ,y l ,z l ) represent the origin of the stem point cloud and the coordinates of the lowest point of the stem point cloud respectively; Then at the origin O=(x o ,y o ,z o ) moves λh in the opposite direction of the tea stem growth direction η, where λ is the proportional coefficient, and obtains the three-dimensional picking point coordinates in the radar coordinate system Finally, the picking angle θ is calculated by the following formula: in, is the normal vector of the horizontal plane

Images