A method for identifying and positioning picking points of tea tender shoots

Abstract

The application discloses a kind of tea tender shoot identification and picking point positioning method, obtain the image with tea tender shoot, and utilize YOLOv7 target detection model to the image obtained tea tender shoot identification, output the detection frame coordinate information of target to be picked;According to whether the picking part in detection frame exists shielding, it is classified into two categories, with shielding and without shielding;For the tea tender shoot without shielding of picking part, three-dimensional picking point positioning is carried out;For the tea tender shoot with shielding of picking part, after obtaining the depth information of its picking part by the method of point cloud completion, three-dimensional picking point positioning is carried out again.The method can solve the picking error caused by the existence of shielding of picking part when positioning tea tender shoot in complex scene, improve picking efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a method for identifying tea tender shoots and positioning picking points. BACKGROUND

[0002] In 2022, the total production value of dry tea in China reached a new high of 318.68 billion yuan, an increase of 252.53 billion yuan compared with the previous year, an increase of 8.62%, and the total output was 318.10 million tons. Among them, the economic benefits of famous and high-quality tea are much higher than those of bulk tea. The production value of famous and high-quality tea is about 2226 billion yuan, and the production value of bulk tea is about 954 billion yuan, with a ratio of about 7:3. One of the reasons why famous and high-quality tea has excellent taste and better economic benefits is its raw materials. Famous and high-quality tea has very high requirements for raw materials. The picking standard for high-grade famous and high-quality tea is mostly "fine and tender picking", that is, the tea raw material is tea bud, one bud and one leaf, and one bud and two leaves. The raw material of bulk tea is three to five leaf mature open tea.

[0003] To meet the growing demand for famous and high-quality tea, the tea planting area in China continues to expand. However, the picking standard of famous and high-quality tea is high, and the current picking method of famous and high-quality tea is almost pure manual picking. Manual picking has the disadvantages of low efficiency, high cost, and labor shortage. In the face of many problems in the production of famous and high-quality tea raw materials, researching and implementing mechanized picking of famous and high-quality tea may be a way out of the current situation.

[0004] In the process of realizing the mechanization of famous and high-quality tea picking, the recognition of the visual system and the positioning of the picking point are of great importance. At present, there are invention patents in the direction of tea target recognition and picking point positioning: Chinese patent "Tea tender shoot recognition method based on improved YOLOv5 target detection" inputs the tea tender bud image after image preprocessing into the improved YOLOv5 target detection module to obtain the prediction result and output the prediction frame of the recognition image; Chinese patent "Precise identification method for tea tender shoot grade in complex environment" acquires tea tender shoot images through a binocular camera, performs size normalization and noise reduction smoothing processing, extracts tea tender shoot features, designs an SVM classifier to segment tea tender shoot images, and uses an improved YOLOv5s model to detect the grade of tea tender shoots; Chinese patent "Tea picking method and system based on RGB differentiation" acquires the depth information and RGB color image of tea, extracts the recognition target of the current frame using an improved salient object detection algorithm, acquires the leaf grade image and calculates the real-time average depth information in the pre-selected frame center and edge area as the basis for determining the cutting tool pose information.

[0005] Although the above-mentioned inventions effectively realize the recognition of tea tender shoots and the positioning of picking points, the disorderly growth of tea in the natural growth environment inevitably leads to the shielding of part of the to-be-picked tender shoots, which will cause the mechanical arm to operate to the wrong position and pick incorrectly, affecting the accuracy and efficiency of picking. SUMMARY

[0006] In order to solve the problems existing in the prior art, the present application proposes a tea tender shoot recognition and picking point positioning method, aiming to solve the picking errors caused by the shielding of the picking part during the positioning of the tea tender shoots in a complex scene, and further the problem of reduced picking efficiency.

[0007] The technical scheme adopted by the present application is as follows:

[0008] A tea tender shoot recognition and picking point positioning method, comprising the following steps:

[0009] Step 1: Obtain an image with tea tender shoots, and use a YOLOv7 target detection model to recognize the tea tender shoots in the obtained image, and output the detection box coordinate information of the to-be-picked target; classify according to whether the picking part in the detection box is shielded, into two categories: with shielding and without shielding;

[0010] Step 2: For tea tender shoots with no shielding of the picking part, perform three-dimensional picking point positioning; for tea tender shoots with shielding of the picking part, obtain the depth information of the picking part by the point cloud completion method, and then perform three-dimensional picking point positioning.

[0011] Further, for tea tender shoots with no shielding of the picking part in the detection box, first perform skeleton extraction, and scan the skeleton points at the bottom of the detection box to determine the skeleton points as the picking points, then combine the depth information to obtain the three-dimensional coordinates of the tea tender shoot picking points, and realize three-dimensional picking point positioning of the unshielded tea tender shoots.

[0012] Further, the method for three-dimensional picking point positioning of tea tender shoots with shielding of the picking part is as follows:

[0013] S1, align the detection box information of the tea tender shoots with shielding of the picking part with the depth map to generate 3D point cloud data;

[0014] S2, denoising and smoothing processing is performed on the obtained point cloud data;

[0015] S3, clustering the point cloud data, segmenting the shielding object, the incomplete tea tender shoots and the background three kinds of point cloud data, and extracting the incomplete tea tender shoots point cloud data;

[0016] S4, the extracted incomplete tea shoot point cloud data is transmitted to the PF-Net point cloud completion network as input for point cloud completion, the missing picking site point cloud data is completed, and complete tea shoot point cloud data is obtained;

[0017] S5, the center of the lower trunk is calculated based on the complete tea shoot point cloud data obtained through point cloud completion, and the three-dimensional coordinates of the tea shoot picking point with occlusion are obtained.

[0018] Further, the PF-Net point cloud completion network includes a generator part and a discriminator part, the generator extracts features from point cloud data and completes point cloud, and the discriminator is used for training process, the reliability of the completed point cloud data is identified by calculating the loss of the completed point cloud data and the actual data of the data set, and the point cloud completion effect is optimized through continuous training iteration to convergence.

[0019] Further, the generator includes an encoder part and a decoder part, the encoder performs feature extraction, and the decoder is responsible for predicting point cloud according to the feature extraction information.

[0020] Further, the decoder part uses FPN feature pyramid as the decoder network to obtain incomplete point cloud data.

[0021] Further, the point cloud data is processed by bilateral filtering.

[0022] Further, the z value of the point cloud data is clustered by K-Means clustering algorithm.

[0023] Further, the RGB-D camera is used to collect tea shoot images.

[0024] Further, the YOLOv7 target detection model includes a backbone network, a feature pyramid and a detection head, the backbone network extracts features through convolution, standardization operation and activation function stacking; the feature pyramid is used for multi-scale target detection; the detection head is composed of multiple convolution layers and fully connected layers, uses convolution and activation function to extract the position and category information of the target, and then uses non-maximum suppression to screen the most possible target detection result.

[0025] The present application has the following advantages:

[0026] The present application reduces the shielding problem in the machine vision recognition and positioning process caused by natural factors of tea leaves, improves the full picking rate of tea leaves, reduces the inaccurate positioning and wrong picking problem of tea picking machine caused by shielding, and improves the accurate picking rate of tea leaves and the tea picking efficiency of the machine. Based on this, in order to apply the method to mobile devices in outdoor scenes, the improved YOLOv7 network greatly reduces the complexity of the model, reduces the parameter amount of the model, and speeds up the inference speed of the model. Based on the present application, the method can quickly and effectively identify tea shoots, use the 3D point cloud completion method to process the shielded tea buds in the picking position, solve the inevitable shielding problem in the machine vision recognition process of tea shoots in the natural growth state, reduce the wrong picking problem in the machine picking process, and improve the full picking rate, accurate picking rate and working efficiency in the process of mechanical picking of tea leaves. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the overall operation logic diagram of the method of the present application;

[0028] Figure 2 is the improved YOLOv7 algorithm network structure diagram of the method of the present application;

[0029] Figure 3 is a supplementary explanation of the improved YOLOv7 algorithm network structure of the present application;

[0030] Figure 4 is the GhostNetV2 module structure used in the present application;

[0031] Figure 5 is the Decoupled module structure used in the present application;

[0032] Figure 6 is the PF-Net point cloud completion network structure used in the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0034] A tea shoot identification and picking point positioning method, the flowchart of which is shown in Figure 1 , comprising the following steps:

[0035] Step 1: Obtain an image with tea shoots from a tea tree, and use a YOLOv7 target detection model to recognize the tea shoots in the obtained image, output the detection box coordinate information (center_x, center_y, width, height) of the target to be picked, where center_x and center_y are the midpoint coordinates of the detection box, and width and height are the width and height of the detection box. Based on the recognition result, classify according to whether there is an occlusion in the picking part in the detection box, into two categories: with occlusion and without occlusion.

[0036] In this embodiment, an RGB-D camera is used to collect tea shoot images to form an image dataset, and the collected image dataset is subjected to image preprocessing, which specifically includes rotation, scaling, and image sharpening.

[0037] In this embodiment, the YOLOv7 target detection model extracts features from the image and fuses multi-scale features to recognize tea shoots and obtain detection box information.

[0038] In this embodiment, the YOLOv7 target detection model mainly consists of three parts: a backbone network, a feature pyramid, and a detection head. The backbone network extracts features through convolution, standardization operations, and activation function stacking. A feature pyramid is introduced for multi-scale target detection. By adding additional branches and feature fusion operations in the backbone network, rich semantic features can be extracted from different levels of feature maps while maintaining sensitivity to targets of different scales. The detection head is composed of multiple convolution layers and fully connected layers. The detection head extracts the position and class information of the target by using convolution and activation functions, and then uses non-maximum suppression to filter the most likely target detection results.

[0039] In the backbone network for feature extraction, GhostNetV2 architecture is used for fast and efficient extraction of useful features, referring to Figure 4 The GhostNetV2 architecture adds a DFC attention mechanism based on the Ghost module. The DFC attention mechanism learns to fuse feature maps at different levels, providing richer and more expressive feature representations, thereby improving the performance of the model in the target detection task. The DFC attention mechanism is used in conjunction with the Ghost module to capture long-range spatial position dependencies and better capture global information. The GhostNetV2 architecture not only makes the network model more lightweight and speeds up the inference, but also facilitates the deployment of mobile devices.

[0040] In the target bounding box prediction process, there are 3 branches respectively for detecting large targets, medium targets, and small targets. Referring to Figure 5Each branch adopts a Decoupled head to extract the target position and category information respectively, and learns through different network branches, and then fuses. The Decoupled head can effectively reduce the parameter quantity and computational complexity, and enhance the generalization ability and robustness of the model.

[0041] Step 2: For tea shoots with unobstructed picking parts, three-dimensional picking point positioning is performed; for tea shoots with obstructed picking parts, the depth information of the picking part is obtained through the point cloud completion method, and then three-dimensional picking point positioning is performed. Specifically as follows:

[0042] For tea shoots with unobstructed picking parts in the detection frame, first perform skeleton extraction, and scan the skeleton points at the bottom of the detection frame to determine them as picking points, then combine the depth information to obtain the three-dimensional coordinates of the tea shoot picking points, and realize three-dimensional picking point positioning of unobstructed tea shoots.

[0043] The method for three-dimensional picking point positioning of tea shoots with obstructed picking parts is as follows:

[0044] S1, first align the detection frame information of the tea shoot with obstructed picking part to the depth map to generate 3D point cloud data.

[0045] The depth image can be calculated as point cloud data after coordinate transformation, and the image coordinate system can be converted to the world coordinate system according to the constraint conditions of the camera internal parameter, i.e. coordinate system transformation.

[0046] The point cloud conversion formula is:

[0047]

[0048] Where x, y, z are the point cloud coordinate system, x', y' are the image coordinate system, and D is the depth value.

[0049] S2, perform bilateral filtering processing on the obtained point cloud data to remove noise points and outliers in the point cloud data and smooth processing, to prevent the original data noise points, outliers and sparsity fluctuations in the point cloud from affecting the subsequent point cloud completion network effect.

[0050] The bilateral filtering formula is:

[0051] p' = p + δp n p

[0052]

[0053] Where p' is the point corrected by the bilateral filtering formula, p is a point in the point cloud, δp is the weight of the normal movement, n p is the neighborhood N r(p) a unit normal vector of the regression plane, q is a neighborhood N r (p) a point in N r (p) is a neighborhood centered at the point p with radius r, ω d and ω n is the distance weight and the normal direction projection module length weight.

[0054] S3, the pretreated point cloud data is clustered by K-Means clustering algorithm according to z value, and three kinds of point cloud data of shelter, defective tea shoots and background are segmented, and the point cloud data of defective tea shoots is extracted.

[0055] S4, referring to Figure 6 The extracted point cloud data of defective tea shoots is transmitted to the PF-Net point cloud completion network as input, the missing picking part point cloud data is completed, and the complete tea shoot point cloud data is obtained.

[0056] The PF-Net point cloud completion network generates only the point cloud data of the defective part without changing the original data, and the network mainly consists of two parts: generator part and discriminator part. The generator extracts features from point cloud data and completes point cloud, and the discriminator is mainly used in the training process. The reliability of the completed point cloud data is identified by calculating the loss between the completed point cloud data and the actual data set, and the point cloud completion effect is optimized by continuous training iteration and convergence.

[0057] The entire generator part is divided into encoder part and decoder part, the encoder performs feature extraction, and the decoder is responsible for predicting point cloud according to the feature extraction information.

[0058] Before the feature extraction process, in order to strengthen the feature extraction effect, the characteristics of multi-scale feature fusion structure with excellent performance in image feature extraction process are extracted, the down sampling method of I farthest point sampling (IFPS) is adopted, and three scales of point cloud data are obtained.

[0059] In the feature extraction process of the encoder part, three scales of point cloud data will pass through the CMLP module for feature extraction, and then three scales of feature extraction results are spliced, and the whole feature extraction result Final Feature Vector V is obtained through MLP operation and convolution operation.

[0060] In the decoder part, in order to better preserve details, the network uses FPN feature pyramid as the decoder network to obtain three different scales of defective point cloud data.

[0061] In order to make the trained network model have better point cloud completion effect and make the generated incomplete data more real, a discriminator is added to the incomplete point cloud prediction in the original scale in the network training process, and a loss function is designed to make it converge.

[0062] S5, calculate the center of the lower trunk of the complete tea shoot point cloud data obtained by point cloud completion to obtain the three-dimensional coordinates of the tea shoot picking point with occlusion in the picking part.

[0063] The above examples are only used to illustrate the design idea and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made in accordance with the principles and design ideas disclosed by the present application are within the scope of protection of the present application.

Claims

1. A method for identifying and locating picking points of tea shoots, characterized in that, The method comprises the following steps: Step 1: acquiring an image of tea tender shoots, and using a YOLOv7 target detection model to recognize the tea tender shoots in the acquired image, and outputting detection box coordinate information of a target to be picked; according to whether the picking part in the detection box is shielded, the target is classified into two categories: shielded and unshielded; the YOLOv7 target detection model comprises a backbone network, a feature pyramid and a detection head, the backbone network performs feature extraction through convolution, standardization operation and activation function stacking; the feature pyramid is used for multi-scale target detection; the detection head is composed of multiple convolution layers and fully connected layers, uses convolution and activation function to extract the position and category information of the target, and then uses non-maximum suppression to screen the most likely target detection result; in the backbone network for feature extraction, a GhostNetV2 architecture is used, and the GhostNetV2 architecture adds a DFC attention mechanism on the basis of a Ghost module; in the target box prediction process, there are three branches for detecting large targets, medium targets and small targets respectively, each branch uses a Decoupled head to extract the target position and category information respectively, and learns through different network branches, and then is fused; Step 2: for the tea tender shoots with unshielded picking parts, three-dimensional picking point positioning is performed; for the tea tender shoots with shielded picking parts, after obtaining the depth information of the picking parts through a point cloud completion method, three-dimensional picking point positioning is performed; the method for three-dimensional picking point positioning of the tea tender shoots with shielded picking parts is as follows: S1, aligning the detection box information of the tea tender shoots with shielded picking parts to a depth map to generate 3D point cloud data; S2, denoising and smoothing the obtained point cloud data; S3, clustering the point cloud data, segmenting three types of point cloud data of the shielded object, the incomplete tea tender shoots and the background, and extracting the incomplete tea tender shoots point cloud data; S4, inputting the extracted incomplete tea tender shoots point cloud data into a PF-Net point cloud completion network for point cloud completion, completing the missing picking part point cloud data, and obtaining complete tea tender shoots point cloud data; the PF-Net point cloud completion network comprises a generator part and a discriminator part, the generator extracts features from the point cloud data and completes the point cloud, the discriminator is used in the training process, calculates the loss of the completed point cloud data and the actual data in the data set, discriminates the reliability of the completed point cloud data, and iterates continuously until convergence to optimize the point cloud completion effect; S5, calculating the center of the lower trunk of the complete tea tender shoots point cloud data obtained through point cloud completion, and obtaining the three-dimensional coordinates of the picking point of the tea tender shoots with shielded picking parts.

2. The method according to claim 1, wherein, For the tea tender shoots with unshielded picking parts in the detection box, first, the skeleton is extracted, the skeleton points at the bottom of the detection box are scanned, the skeleton points are determined as the picking points, the three-dimensional coordinates of the picking points of the tea tender shoots are obtained in combination with the depth information, and the three-dimensional picking point positioning of the unshielded tea tender shoots is realized.

3. The method according to claim 1, wherein, The generator comprises an encoder part and a decoder part, the encoder replicates feature extraction work, and the decoder is responsible for predicting the generation of point clouds according to the information of feature extraction.

4. The method according to claim 3, wherein The decoder part uses FPN feature pyramid as the decoder network to obtain the incomplete point cloud data.

5. The method for identifying and positioning the picking point of tea tender shoots according to any one of claims 1-4, characterized in that, The point cloud data is processed by bilateral filtering.

6. The method according to claim 5, wherein The z value of the point cloud data is clustered by K-Means clustering algorithm.

7. The method according to claim 5, wherein The RGB-D camera is used to collect the images of tea tender shoots.

Images