A method for accurately identifying the grade of tea tender shoots in a complex environment

Abstract

The application discloses a kind of complex environment under tea tender shoot grade accurate identification method, comprising: using binocular vision camera to collect tea tender shoot original image containing;The original image collected is preprocessed: size normalization, noise reduction smoothing processing;Extract the image characteristics of tea tender shoot, and carry out data dimension reduction and feature fusion;Design SVM classifier to carry out image segmentation to tea tender shoot;The segmented image is made dataset;Dataset is as the input of deep learning model, trains the improved YOLOv5s model proposed in the application, and carries out grade target detection to tea tender shoot.The application proposes the identification method of first rough then fine, first using image segmentation step to remove the interference of ground, old leaf and branch under complex environment, then further fine identification tea tender shoot grade using deep learning target detection algorithm, improve the final recognition accuracy, realize complex environment under tea tender shoot grade accurate identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target recognition of tea leaf picking robots, and in particular to a method for accurately identifying the grade of tea leaf tender shoots in a complex environment. BACKGROUND

[0002] With the steady growth of China's national economy, the demand for tea at home and abroad is also increasing day by day. In order to increase the yield of high-quality tea, it is necessary to modernize the tea picking industry. At present, most of the tea picking robots at home and abroad are based on the "one-size-fits-all" mode for bulk tea, which cannot preserve the integrity of the tea leaves and meet the requirements of picking high-grade tea. A small number of tea picking robots for famous and high-quality tea have low recognition accuracy and do not take into account the factors of the actual environment, so the actual application rate is not high.

[0003] At present, most tea recognition technologies are based on machine vision, which mainly includes tea image feature-based recognition technology and deep learning-based tea target detection technology.

[0004] Chinese patent document CN111784764A discloses a tea bud recognition and positioning algorithm, which mainly uses the extracted color features of tea buds to segment the tea bud image based on the threshold of super green features. Although this method has fast recognition speed, it is relatively rough to segment the buds only by color features, and the recognition accuracy is low, and there is no further classification of bud grade.

[0005] Patent document CN102013021A discloses a tea bud segmentation and recognition method based on color and region growing, which uses the HSI color features of tea and the region growing method to segment the tea image. Although this method can better preserve the overall contour of the tea, it is not accurate in segmentation accuracy, and it is difficult to cope with complex field environments such as light conditions, rain interference, and branch obstruction.

[0006] Patent document CN112633212A discloses a method for recognizing and classifying the grade of tea buds based on computer vision. This method uses an improved watershed algorithm to segment the RGB image of tea, and then classifies the recognition results of the improved watershed algorithm into one bud and one leaf, one bud and two leaves, and one bud and multiple leaves for classification and grade classification marking. The LeNet-5 convolutional neural network is used to train and test these categories. Although this invention uses a deep learning network to further identify the grade of tea, there is still room for improvement in the recognition accuracy of the model for multiple poses and multiple targets of tea.

[0007] Although these patents can realize tea bud recognition and even tea grade recognition, they generally do not have a refined recognition process, the recognition accuracy is not high, and the actual picking environment is a non-structured complex and changeable environment, which cannot be truly applied to practical applications. SUMMARY

[0008] In view of the above problems, the present application provides a tea tender shoot grade accurate identification method in a complex environment, which improves the final identification accuracy and realizes the tea tender shoot grade accurate identification in a complex environment.

[0009] The technical solution for achieving the object of the present application is as follows: a tea tender shoot grade accurate identification method in a complex environment, which comprises tea tender shoot image segmentation and tea tender shoot grade identification, and specifically comprises:

[0010] An original image is collected by using a binocular camera, the original image is preprocessed and image segmentation is performed;

[0011] A tea tender shoot grade data set is made based on the segmented image;

[0012] A YOLOv5s model is constructed, and the YOLOv5s model is trained through the tea tender shoot grade data set until a YOLOv5s model that converges and meets the accuracy requirement is obtained; the YOLOv5s model adopts an FPN+PAN structure, in the lateral splicing process of the FPN, an AAM attention mechanism module is embedded to generate a feature mapping, and a MDCM module is introduced, in the top-down propagation path, the feature mapping is first subjected to the MDCM module before down-sampling, the receptive field is expanded through the hollow convolution, and the multi-scale feature extraction capability is enhanced, and then the feature mapping is fused with other features at a shallow level;

[0013] The tea tender shoot grade identification is performed on the real-time collected tea original image through the YOLOv5s model, and the specific grade of the tender shoot is identified: special grade (single bud), first grade (one bud one leaf) and second grade (one bud two leaves, one bud three leaves), and the posture of the tea tender shoot: front view, side view and downward view.

[0014] Further, the pre-processing and image segmentation of the original image specifically comprises:

[0015] Step 1-1: the original image is subjected to denoising, smoothing and enhancement processing through a bilateral filtering algorithm, so as to improve the discrimination degree of the tea tender shoot and the background;

[0016] Step 1-2: color, texture and shape features are extracted from the tea tender shoot respectively;

[0017] Step 1-3: the extracted feature set is subjected to data dimension reduction by using PCA;

[0018] Step 1-4: The image containing tea shoots is segmented by a heterogeneous multi-feature weighted fusion model and an SVM classifier, and the background such as old leaves, branches and ground is removed, wherein the heterogeneous multi-feature weighted fusion model better fuses the features of tea shoots.

[0019] Compared with the prior art, the present application has the beneficial effects that:

[0020] (1) The YOLOv5s model is constructed, the FPN+PAN structure is improved, the AAM attention mechanism is introduced, the loss of local information of tea shoots is reduced, the MDCM module is introduced, the receptive field is expanded through dilated convolution, the multi-scale feature extraction capability is enhanced, the other features of shallow level are fused, the PANet shortens the path from the bottom layer to the high layer, enhances the liquidity of information, and is helpful for better fusion of different hierarchical features, and improves the detection precision of small target tea single bud;

[0021] (2) The original image is processed by using the bilateral filtering algorithm, most of the Gaussian and salt and pepper noise can be filtered out, and the boundary shape of the tea shoots can be reserved to a certain extent;

[0022] (3) A heterogeneous multi-feature weighted fusion model is proposed, which better fuses the features of tea shoots, and the SVM classifier is designed, RBF kernel is selected as the kernel function after a large number of experimental comparison and analysis, and the prediction accuracy is improved.

[0023] The present application will be further described below in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is a complex environment tea shoots grade accurate identification method flow chart of the present application.

[0025] Figure 2 It is a complex environment tea shoots image segmentation flow chart of the present application.

[0026] Figure 3 It is a SVM classifier design flow chart of the present application.

[0027] Figure 4 It is a tea shoots different grade, different attitude schematic diagram of the present application.

[0028] Figure 5 It is an improved FPN+PAN tea shoots grade feature extraction network structure diagram of the present application.

[0029] Figure 6 It is an AAM attention mechanism structure diagram of the present application.

[0030] Figure 7 It is a multi-scale dilated convolution feature extraction module structure diagram of the present application. DETAILED DESCRIPTION

[0031] The present application is based on machine vision, combined with image processing technology, machine learning technology and deep learning technology to achieve accurate identification of tea tender shoot grade in complex environment. In order to better understand the characteristics, advantages and operation details of the present application, further description will be made in conjunction with the drawings of the specification, but some examples in the detailed description are intended to explain the present application and cannot be understood as limiting the present application.

[0032] In combination Figure 1 , a method for accurate identification of tea tender shoot grade in complex environment includes two stages, the first stage is to segment the tea tender shoot image in complex environment, and the second stage is to identify the tea tender shoot grade by an improved YOLOv5s model;

[0033] The first stage of the present application segments the tea tender shoot image in complex environment, in combination Figure 2 as shown, there are several steps as follows:

[0034] Step 1: Use binocular cameras to collect raw image data under different light conditions (normal light, front light, back light), different viewing angles (front view, side view, overhead view), and different backgrounds (ground, branches, old leaves), and go to step 2;

[0035] Step 2: Denoising and smoothing processing is performed on the collected raw image, and a bilateral filtering algorithm is used, which can filter out most of the Gaussian and salt and pepper noise, and also retains the boundary shape of the tea tender leaf to a certain extent. The filtered image goes to step 3;

[0036] Step 3: Extract color, texture and shape features of tea tender shoots, and go to step 4;

[0037] Step 3.1: Considering the RGB model, HIS model and Lab model in color feature extraction, color features are extracted by partitioning feature points of the original image data, and 50 feature points are selected for each tea leaf image of tender leaf, old leaf, branch and ground area to extract R, G, B, H, S, I, L, a, b color components;

[0038] Step 3.2: Considering the comprehensive consideration of tea texture characteristics, implementation difficulty and computing capacity, gray level co-occurrence matrix (GLCM) is used to extract texture features of tender shoot image, and the reference texture feature parameters are energy, entropy and contrast;

[0039] Step 3.3: A region-based morphological feature extraction method is used to obtain the shape feature vector of the tender leaf, and the reference shape feature parameters are area, perimeter, major axis, minor axis, narrow length, shape coefficient, circularity and rectangularity;

[0040] Step 4: Principal Component Analysis (PCA) is used to reduce the dimensionality of the high-dimensional feature set extracted in Step 3 to address the "curse of dimensionality" problem and reduce computational complexity.

[0041] Step 5: A heterogeneous multi-feature weighted fusion model is proposed to better integrate tea shoot features. The specific weighted fusion method is as follows:

[0042] Assume that the color feature set of the tea shoot after PCA dimensionality reduction is F1, the shape feature is F2, and the texture feature is F3. Each is input into the SVM classifier to establish a classification model, and the training accuracy R1, R2, and R3 are obtained. According to the obtained accuracy, the feature weights are calculated, and the calculation formula is as follows:

[0043]

[0044] Multiply the sub-feature sets F1, F2, and F3 by the corresponding weights W i , perform linear weighted combination, and input the fused features into the SVM classifier again to predict the probability of the test sample belonging to which class. The expression of the fused feature set S is as follows:

[0045]

[0046] In particular, since the physical meaning and value range of each feature set are not the same, it is not appropriate to compare the advantages and disadvantages based on numerical values, so normalization operation is needed for the feature set. Gaussian normalization is used to normalize the collected features to the [-1, 1] interval:

[0047] t'(i) = ((t(i) - μ) / 3σ + 1) / 2 (3)

[0048] where t'(i) is the normalized i-th feature, t(i) is the i-th feature of the image, μ is the mean value of the feature, and σ is the standard deviation of the feature.

[0049] Step 6: Use machine learning algorithm to design SVM classifier to segment the image containing tea shoots, remove old leaves, branches, and ground background, and convert the segmented image to Step 7.

[0050] The design process of the SVM classifier combines Figure 3 and can be summarized as follows:

[0051] Step 6.1: Select training set and test set in a certain proportion, and the class images of the training set should be as many as possible. The test set cannot contain images of the training set.

[0052] Step 6.2: Feature vector extraction: low-dimensional features of the image are extracted, feature normalization and dimension reduction are performed, and on the basis of ensuring that the main features are not lost, the calculation amount is greatly reduced and the training speed is accelerated.

[0053] Step 6.3: Selecting a suitable nonlinear transformation kernel function and error cost coefficient C; in particular, the SVM classifier designed in the present application selects RBF kernel as the kernel function after a large number of experimental comparison and analysis

[0054] Step 6.4: Realize the space mapping from low-dimensional to high-dimensional, realize the transformation of the quadratic optimization problem;

[0055] Step 6.5: Training of the training set, obtaining the SVM classifier model;

[0056] Step 6.6: Using the trained model to predict the class of the test set sample.

[0057] The second stage of the present application is to identify the target of tea tender shoot grade, which continues the above steps:

[0058] Step 7: Using the tea tender shoot image segmented in step 6, combined with Figure 4 different grade and different posture schematic diagram of tea tender shoot, making tea tender shoot grade data set, entering step 8;

[0059] In particular, according to the actual tea multi-posture situation, 5 kinds of recognition categories are labeled in the data set: single bud, one bud and one leaf, one bud and two leaves, side view with shielding tender bud and downward view without tea stem tender bud, and the training set and the verification set are randomly selected according to the proportion of 9:1.

[0060] Step 8: Input the data set into the improved YOLOv5s model for training, and enter step 9;

[0061] In particular, in order to improve the accuracy of tea tender shoot grade target detection, the present application improves the current YOLOv5s model, and the specific improvement details are as follows:

[0062] combined with Figure 5The FPN+PAN structure in the YOLOv5s target detection model is improved to construct a feature extraction network suitable for tea tender shoot grade. In the original FPN lateral splicing process, an AAM attention mechanism module is embedded to generate feature maps {M1, M2, M3}. In the top-down propagation path, the feature maps {M1, M2, M3} will first pass through the MDCM module before downsampling. Through the dilated convolution expansion receptive field, the multi-scale feature extraction capability is enhanced, and then fused with other shallow features to generate feature maps {P1, P2, P3}. PANet shortens the path from the bottom to the high layer, enhances the flowability of information, and helps better fusion of different level features, improving the detection accuracy of small target tea single bud.

[0063] In particular, in combination with Figure 6 An AAM attention mechanism structure diagram, further understanding the self-attention mechanism. The AAM structure is roughly divided into two parts: adaptive average pooling operation and spatial weight map generation.

[0064] The first part, the feature map M1 obtains multi-scale semantic feature information through an adaptive pooling layer, and then uses the bilinear interpolation method to upsample these multi-scale features to a unified scale WxH, preparing for subsequent fusion.

[0065] The second part, the spatial attention mechanism uses the Concat layer to merge the previous multiple channel features, and then sequentially passes through the 1x1 convolution layer, the ReLU layer, the 3x3 convolution layer and the Sigmoid activation layer, to obtain the spatial attention weight. The generated weight map and the feature map after the Concat layer are fused by Add. The weight map is mapped to the feature map M1 by Hadamard product operation, and the context is aggregated to generate M2. The finally generated feature map M2 has rich multi-scale context information, strengthens the correlation between local features and global features, and highlights the weight of local features, effectively solving the information loss problem caused by the reduction of channel number.

[0066] In particular, in combination with Figure 7 A multi-scale dilated convolution feature extraction module structure diagram, further understanding the MDCM module. The network structure is divided into two parts, the first part is to extract the multi-scale features of tea tender shoots using dilated convolution, and the second part is to fuse features. This part fuses the features of different sizes obtained in the previous step using Add fusion, element-wise adds the feature information extracted by the three parallel dilated convolution branches, and outputs the fused tea tender shoot features.

[0067] Furthermore, dilated convolution expands the receptive field by adding holes (inserting 0s) between each pixel of the convolution kernel, achieving a larger receptive field compared to standard convolution with the same number of parameters and computational cost. The kernel size of dilated convolution is k, which is equivalent to the kernel size k of a standard convolution. * The relationship between them is as follows:

[0068] k * = k + (k-1) × (d-1) (4)

[0069] Where d is the expansion rate.

[0070] Receptive field R of the current layer i+1 The calculation formula is as follows:

[0071] R i+1 =R i +(k * -1)×S i (5)

[0072] in, R i This indicates the sensory field of the next higher level. This represents the product of the stride lengths of all previous layers.

[0073] The dilated convolutional modules in the MDCM module all use 3×3 convolutional kernels with dilation rates r of 3, 5, and 7, respectively.

[0074] Step 9: Use the trained model to identify tender shoots in the original tea leaf image. Use pre-selected boxes to mark the categories to be identified, mainly: premium grade (single bud), grade 1 (one bud and one leaf), and grade 2 (one bud and two leaves, one bud and three leaves), as well as the posture of the tea shoot: front view, side view, and top view. Output the specific category and prediction confidence in the upper left corner.

[0075] This invention proposes a coarse-to-fine identification method. First, image segmentation is used to remove interference from the ground, old leaves, and branches in complex environments. Then, a deep learning object detection algorithm is used to further refine the identification of the grade of tea shoots, thereby improving the final identification accuracy and achieving accurate identification of the grade of tea shoots in complex environments.

[0076] For illustrative purposes, the specific embodiments described above are merely exemplary and are intended to help those skilled in the art better understand this patent. They should not be construed as limiting the scope of this patent. Any technical solutions obtained by means of equivalent substitution or equivalent transformation fall within the protection scope of this invention.

Claims

1. A method for accurately identifying the grade of tea tender shoots in a complex environment, characterized in that, Specifically comprising: Collecting original images by using a binocular camera, pre-processing the original images and performing image segmentation; Making a tea tender shoot grade dataset based on the segmented images; Constructing a YOLOv5s model and training the YOLOv5s model through the tea tender shoot grade dataset until a YOLOv5s model meeting the accuracy requirement is obtained; the YOLOv5s model adopts an FPN+PAN structure, in the lateral splicing process of the FPN, an AAM attention mechanism module is embedded to generate feature mapping, and a MDCM module is introduced, in the top-down propagation path, the feature mapping is first subjected to the MDCM module before down-sampling, through dilated convolution expansion of the receptive field, the multi-scale feature extraction capability is enhanced, and then the feature mapping is fused with other features at a shallow level; Performing tender shoot grade recognition on real-time collected tea original images through the YOLOv5s model; The AAM attention mechanism module comprises an adaptive average pooling operation module and a spatial weight map generation module; The adaptive average pooling operation module adopts a bilinear interpolation method to up-sample the multi-scale semantic features of the obtained initial feature map M1 to a unified size; The spatial weight map generation module merges a plurality of channel features through a Concat layer, and then sequentially subjects the features to a 1*1 convolution layer, a ReLU layer, a 3*3 convolution layer and a Sigmoid activation layer to obtain spatial attention weights, fuses the generated spatial attention weights and the feature map after the Concat layer, and generates a feature map M2 by using Hadamard product operation to map the weight feature map to the initial feature map M1 after context aggregation; The MDCM module extracts multi-scale features of tea tender shoots by using a plurality of parallel dilated convolution branches, and fuses the features by using an Add fusion mode, and outputs the fused tea tender shoot features by element-wise addition.

2. The method according to claim 1, wherein, The pre-processing of the original images and the image segmentation specifically comprises: Step 1-1: performing denoising, smoothing and enhancement processing on the original images by using a bilateral filtering algorithm; Step 1-2: extracting color, texture and shape features of the tea tender shoots respectively; Step 1-3: performing data dimension reduction on the extracted feature set by using PCA; Step 1-4: segmenting the image containing the tea tender shoots by using a heterogeneous multi-feature weighted fusion model and an SVM classifier.

3. The method according to claim 2, wherein, The step 1-2 of extracting color, texture and shape features of the tea tender shoots respectively specifically comprises: Step 1-2-1: extracting color features in the form of partition feature point labeling of image data, and selecting 50 feature points for the tender leaf, old leaf, branch and ground area of each tea image to extract R, G, B, H, S, I, L, a and b color components; Step 1-2-2: extracting texture features of the tender shoot image by using a gray level co-occurrence matrix, wherein the reference texture feature parameters include energy, entropy and contrast; Step 1-2-3: obtaining shape feature vectors of the tender leaf by using a region-based morphological feature extraction method, wherein the reference shape feature parameters include area, perimeter, major axis, minor axis, narrow length, shape coefficient, circularity and rectangularity.

4. The method according to claim 2, wherein, The steps 1-4 specifically include segmenting the image containing the tea shoot by a heterogeneous multi-feature weighted fusion model and an SVM classifier: Suppose that the color feature set of the tea shoot after PCA dimension reduction is F1, the shape feature is F2, and the texture feature is F3, which are respectively input into the SVM classifier to establish a classification model, and the training accuracy R1, R2, and R3 are obtained, and the feature weight of each is calculated according to the obtained accuracy, and the calculation formula is: The sub-feature sets F1, F2, F3 are multiplied by the corresponding weights W i , respectively, and linearly combined to obtain the fused feature set S, and the expression of the fused feature set S is as follows: S = F i • W i i = 1, 2, 3 s.t.W i >0, The fused features are input into the SVM classifier again to predict the probability of the to-be-tested sample belonging to which class; Before the feature weighted fusion, the collected features are normalized to the [-1, 1] interval by using Gaussian normalization: t'(i) = ((t(i)-μ) / 3σ+1) / 2 In the formula, t'(i) is the normalized ith feature, t(i) is the ith feature of the image, μ is the average value of the feature, and σ is the standard deviation of the feature.

5. The method according to claim 2, wherein the method is characterized by, The SVM classifier is designed by using a machine learning algorithm, and the SVM classifier uses an RBF kernel as a kernel function.

6. The method according to claim 1, wherein, The dilated convolution is in the form of expanding the receptive field by adding a hole between each pixel of the convolution kernel, and the size of the convolution kernel of the dilated convolution is k, and the size of the equivalent convolution kernel k * The relationship between them is as follows: k * = k + (k - 1) x (d - 1) Where d is the dilation rate. Receptive field R of the current layer i+1 is: R i+1 = R i + (k * - 1) x S i wherein, R i denotes the receptive field of the previous layer, denotes the product of the strides of all previous layers. The dilated convolution branch adopts a 3x3 convolution kernel, and the dilation rates r are 3, 5, and 7, respectively.

Images