A tea bud detection method based on deep learning and its model building method

Abstract

The present invention provides a tea bud detection method based on deep learning and a model building method thereof, the main part of which is a model building method, comprising the following steps: obtaining a tea data set in a real scene and dividing it into a training set and a test set, and preprocessing the data; building a YOLOv5_DC model that can focus on the slender morphological features of tea buds, realizing adaptive extraction of tea bud features by introducing deformable convolution in the backbone network, and retaining the semantic and spatial information of multi-layer features by introducing a lightweight upsampling operator in the neck network; training and testing the model, and using it for tea bud detection after the model is built. At the same time, in order to improve the detection capability of the model, a cross entropy function is used to improve the loss function. The method proposed by the present invention has high detection accuracy, low computational cost and fewer parameters, and is an effective solution for tea bud detection by tea picking robots.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual detection of agricultural picking robots, and in particular relates to a tea bud detection method based on deep learning and a model building method thereof. Background Art

[0002] Tea is one of the most widely consumed beverages in the world, and is favored by consumers for its rich nutritional value and unique flavor. The traditional method of picking tea buds manually is a typical labor-intensive operation with high labor intensity, low production efficiency and unstable quality. Due to the disappearance of the demographic dividend and the rapid development of information technology, there is an urgent need to research and develop automated and intelligent picking robots for needle-shaped famous tea production.

[0003] Object detection technology is of great significance to meet the needs of intelligent tea picking relying on vision-guided manipulators. However, compared with other crops, tea bud detection faces certain challenges due to its extreme aspect ratio, small size, and inevitable occlusion and illumination changes, which make it difficult to complete this task using conventional methods. Therefore, research and development of tea bud recognition and detection methods are of great significance for intelligent picking.

[0004] In recent years, the effectiveness of deep learning algorithms in target detection has been fully demonstrated. Generally speaking, target detection technology based on deep learning can be divided into two categories: end-to-end one-stage detection algorithms and two-stage detection algorithms that require screening candidate boxes. Among them, the two-stage target detection algorithm represented by Faster R-CNN has a slow detection speed and is not suitable for deployment on picking robots; while the one-stage algorithm represented by the YOLO series achieves end-to-end fast detection and is more suitable for target detection of intelligent picking robots.

[0005] As a newer version of the YOLO series, YOLOv5 has been greatly improved in accuracy and speed, showing strong detection performance in commonly used target detection networks and having strong advantages in rapid model deployment. However, it works better on general target datasets. In the detection of tea buds, which are slender in shape, small in size, with unclear features and easily affected by occlusion and light changes, the effect of directly using the YOLOv5 model is not ideal. Summary of the invention

[0006] In view of the above problems, the first aspect of the present invention provides a method for building a tea bud detection model based on deep learning, comprising the following steps:

[0007] Step 1: Collect and screen the original image data of tea leaves in real scenes, create a data set, and divide it into a training set and a test set according to a certain ratio;

[0008] Step 2: annotate and amplify the training set images to obtain a preprocessed training set;

[0009] Step 3, based on the YOLOv5 model structure, a YOLOv5_DC model that can focus on the slender morphological features of tea buds is built; the YOLOv5_DC model introduces deformable convolution DCNv2 in the backbone network to realize adaptive extraction of tea bud features, and introduces a lightweight upsampling operator CARAFE in the neck network to retain the semantic and spatial information of multi-layer features;

[0010] Step 4: Use the cross entropy function Varifocal Loss to improve the loss function, and use the training set preprocessed in step 2 to train the YOLOv5_DC model in step 3;

[0011] Step 5: Use the image data in the test set to test the trained YOLOv5_DC model and complete the construction of the final model.

[0012] Preferably, after the YOLOv5_DC model training is completed, an image enhancement judgment module and an image enhancement module are further provided at the image input end, which are used to perform image enhancement judgment and image enhancement during testing and practical application. The specific process is as follows:

[0013] S1, calculate the average grayscale value AG of all tea images. If the AG value is greater than or equal to the preset value T1, the image is adaptively corrected for brightness and enters S2; if the AG value is less than T1, it is directly input into the trained YOLOv5_DC model;

[0014] S2, using the SLIC superpixel block algorithm for image segmentation, fully considering the similarity of features within the region and the difference of features between regions to improve the effectiveness of regional adaptive correction;

[0015] S3, calculating the regional average gray value AG for each superpixel block, if the regional AG value is greater than or equal to the preset value T2, then adaptive brightness correction is performed on the region, otherwise the regional brightness remains unchanged;

[0016] S4, merging the corrected area and the uncorrected area to obtain a corrected and enhanced image to input into the trained YOLOv5_DC model.

[0017] Preferably, the SLIC superpixel block algorithm is used for image segmentation, combining color information and position information, and the clustering process uses a weighted distance measurement value D, which is specifically calculated as follows:

[0018]

[0019]

[0020]

[0021] Among them, l T Represents the color space in a grayscale image; [xy] T It represents the pixel position, [l k x k y k ] T represents the cluster center; d s is the spatial similarity; d c is the color similarity;

[0022] The adaptive brightness correction of the region is a method of estimating an appropriate gamma value based on average brightness. The gamma value γ required for correction is calculated as follows:

[0023]

[0024]

[0025]

[0026]

[0027] Among them, X∈[0 1] is the average brightness of the image, is the center of the histogram brightness, all pixels in the output image will be enhanced using the estimated gamma value.

[0028] Preferably, in step 2, the training set images are labeled and amplified, specifically:

[0029] The original tea images collected in natural scenes were cropped with a length-width ratio of 1:1 according to the principle of retaining as many tea bud samples as possible, and their length and width were adjusted to 640×640 pixels and then divided into training and test sets;

[0030] Data annotation: Use image annotation tools to manually annotate images. The annotation standard is single bud annotation. The annotated label file is saved in TXT format.

[0031] Data enhancement: The annotated images are augmented by operations such as 90-degree and 180-degree flipping, vertical and horizontal mirroring, brightness enhancement, and adding Gaussian noise, and the corresponding annotation files are generated synchronously.

[0032] Preferably, the YOLOv5_DC model includes a backbone network, a neck network and a prediction end; the backbone network includes a DBS module, a C3 module and an SPPF module, the DBS module is composed of a variable convolution DCNv2, a batch normalization layer BN and a SiLU activation function layer, and is intended to achieve adaptive extraction of tea bud features; the C3 module adopts a residual structure to reduce model parameters and extract finer-grained features; the SPPF module is improved from the SPP module, and the SPPF module fuses local features and global features to enrich the expression ability of the output feature map; the neck network uses a feature pyramid network FPN and a path aggregation network PAN for feature fusion, so that the model obtains rich feature information; the prediction end detects targets of different sizes through three detection layers with feature maps of different sizes.

[0033] Preferably, in step 3, the deformable convolution DCNv2 is introduced into the backbone network to realize the adaptive extraction of tea bud features as follows:

[0034] The deformable convolution DCNv2 is introduced to design the DBS module, and the DBS module is used to replace the CBS module in the original YOLOv5 backbone network; the DCNv2 can achieve random sampling near the current position by introducing an offset variable at the position of each sampling point in the convolution kernel, and enable the sampling grid to deform freely, so as to improve the model's ability to focus on the target; at the same time, the modulation mechanism is used to learn the weight of each sampling point and adjust the amplitude of the input feature at different spatial positions to reduce the influence of irrelevant image content on feature extraction, so as to improve the model's ability to focus on features;

[0035] During the training process, the convolution kernel and offset are learned at the same time. The specific calculation formula is as follows:

[0036] R={(-1,-1),(-1,0),...,(0,1),(1,1)}

[0037]

[0038]

[0039] g(a,b)=max(0,1-|ab|)

[0040] Where R is the size and expansion of the receptive field; P represents an arbitrary position, q enumerates all the integral space positions in the feature map x; x(p) represents the value of the point at all decimal positions after adding the offset; Δm is the modulation scalar.

[0041] Preferably, in step 3, a lightweight upsampling operator CARAFE is introduced into the neck network, specifically: when building the model, a lightweight upsampling operator CARAFE is introduced to replace the nearest neighbor interpolation in the neck network for upsampling, CAREFE includes a content-aware reconstruction module and a kernel prediction module, and the kernel prediction module consists of three sub-modules: a channel compressor, a content encoder, and a kernel normalizer; first, the channel compression sub-module is used to compress the input feature channel, and secondly, the content encoder sub-module encodes the content of the compressed feature map to generate a reconstructed kernel, and finally, the kernel normalization sub-module applies a softmax function to each reconstructed kernel.

[0042] Preferably, in step 4, the cross entropy function Varifocal Loss is used to improve the loss function, specifically:

[0043] The cross entropy function Varifocal Loss is used to calculate the loss value of the anchor box, which can be expressed as:

[0044]

[0045] Among them, q is the intersection-and-union ratio between the target anchor box and the annotation box. When q=0, it means that the anchor box is a negative sample. When q>0, it means that the anchor box is a positive sample. p is the prediction probability of the anchor box for tea buds. α and γ are adjustable coefficients.

[0046] The second aspect of the present invention provides a tea bud detection method based on deep learning, comprising the following process:

[0047] Collect tea image data in real time;

[0048] Inputting the tea image data into the tea bud detection model constructed by the construction method described in the first aspect to perform tea bud detection;

[0049] Output detection information, including the specific location information of the tea buds in the tea image and various confidence levels;

[0050] Confidence screening is performed according to the pre-set threshold to obtain the final tea bud detection results.

[0051] The third aspect of the present invention also provides a tea bud detection device based on deep learning, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the tea bud detection model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor can execute the tea bud detection method.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] First, the present invention uses a deep learning algorithm to achieve accurate target detection of the tea picking robot, which solves the problem of low accuracy in detecting slender tea buds based on the current deep learning algorithm, provides strong support for the practical application of the tea picking robot, and has high practical application value;

[0054] Second, the present invention introduces the deformable convolution DCNv2 in the backbone network of the model to replace the traditional convolution operation, allowing the model to learn geometric deformation by adjusting the offset, achieving feature adaptation and enhancing the feature extraction capability of the model;

[0055] Third, in the neck structure of the model, the present invention adopts content-aware feature reorganization CARAFE instead of the nearest neighbor interpolation in the neck network for upsampling, which helps to retain the semantic and spatial information of multi-layer features and prevent the loss of valuable information;

[0056] Fourth, the present invention uses the Varifocal Loss function to balance the number of positive and negative samples, which helps to reduce target omissions due to occlusion and improve the performance of the model;

[0057] Fifth, the present invention proposes an image enhancement technology that uses an adaptive correction method to adjust regional brightness, thereby improving image quality and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a diagram of the data enhancement effect of Example 1 of the present invention.

[0059] Figure 2 This is the overall structure diagram of the YOLOv5_DC learning model proposed in the present invention.

[0060] Figure 3 This is a structural diagram of the improved DBS module proposed in the present invention.

[0061] Figure 4 FIG. 4 is an overall framework diagram of the CARAFE used in the present invention.

[0062] Figure 5 This is a workflow diagram of the image enhancement judgment and image enhancement module proposed in the present invention.

[0063] Figure 6 This is a comparison chart of the comprehensive performance of different models in Example 1 of the present invention.

[0064] Figure 7 This is a comparison chart of tea bud detection results using different models in an embodiment of the present invention.

[0065] Figure 8 This is a simplified structural block diagram of the tea bud detection device in Example 2. DETAILED DESCRIPTION

[0066] The invention will be further described below in conjunction with specific embodiments.

[0067] Embodiment 1:

[0068] In order to better realize the intelligence of the tea picking process and solve the technical difficulties of the current intelligent tea picking robot in target detection, the present invention proposes a tea bud detection model based on deep learning, which can be used for cloud-based picking robots to improve the detection accuracy of needle-shaped tea buds. The main invention content is the YOLOv5_DC model construction method. Compared with the original YOLOv5 model, the newly proposed model can greatly improve the accuracy of tea bud detection without increasing the complexity of the model, providing strong support for the precise detection and positioning of the tea picking robot, and can solve the problem that the existing deep learning model has poor detection ability for slender objects to a certain extent.

[0069] The method of building a tea bud detection model based on deep learning generally includes the following steps:

[0070] Step 1: Collect and screen the original image data of tea leaves in real scenes, create a data set, and divide it into a training set and a test set according to a certain ratio;

[0071] Step 2: annotate and amplify the training set images to obtain a preprocessed training set;

[0072] Step 3, based on the YOLOv5 model structure, a YOLOv5_DC model that can focus on the slender morphological features of tea buds is built; the YOLOv5_DC model introduces deformable convolution DCNv2 in the backbone network to realize adaptive extraction of tea bud features, and introduces a lightweight upsampling operator CARAFE in the neck network to retain the semantic and spatial information of multi-layer features;

[0073] Step 4: Use the cross entropy function Varifocal Loss to improve the loss function, and use the training set preprocessed in step 2 to train the YOLOv5_DC model in step 3;

[0074] Step 5: Use the image data in the test set to test the trained YOLOv5_DC model and complete the construction of the final model.

[0075] This embodiment further illustrates the specific process of building the tea bud detection model of the present invention through a specific experimental process.

[0076] 1. Get the original data

[0077] Tea images were collected in real tea gardens, including tea images of different types, angles, backgrounds, and light intensities, which were used as raw data. The shooting angles included the side and top, and the shooting distance was 30-50 cm. The tea images were cropped with a length and width of 1:1 in accordance with the principle of retaining as many tea bud samples as possible, and their length and width were adjusted to 640×640 pixels. They were divided into training and test sets in an 8:2 ratio for model training and testing.

[0078] 2. Image Data Preprocessing

[0079] After image preprocessing such as image cutting, screening, and compression, the collected image datasets were annotated with the image annotation tool “Make Sense”. The annotation standard was single buds, and the annotated label files were saved in TXT format.

[0080] The annotated images are enhanced by performing data augmentation through operations such as vertical and horizontal mirroring, rotation at different angles, brightness enhancement, and adding Gaussian noise. The corresponding annotation files are generated synchronously with the image enhancement, such as Figure 1 As shown in Figure 2. After data augmentation, the dataset has a total of 1,330 images.

[0081] 3. Model building

[0082] Based on the YOLOv5 model structure, a YOLOv5_DC model is built that can focus on the slender morphological characteristics of tea buds. The YOLOv5_DC model is mainly composed of the backbone network, the neck network and the prediction end, such as Figure 2 As shown. The backbone network includes DBS, C3, and SPPF modules. The DBS module consists of a variable convolution DCNv2, a batch normalization layer, and a SiLU activation function layer, which aims to achieve adaptive extraction of tea bud features. C3 uses a residual structure to reduce model parameters and extract finer-grained features. The SPPF module is improved from the SPP module. The SPPF module fuses local features and global features to enrich the expressive power of the output feature map. The neck network uses the feature pyramid network FPN and the path aggregation network PAN for feature fusion, so that the model obtains rich feature information. The prediction end detects targets of different sizes through three detection layers with feature maps of different sizes.

[0083] (1) In the backbone network, the conventional convolution used in the backbone network of the original YOLOv5 has a very limited ability to model geometric transformations. It relies on a fixed geometric structure, the convolution unit samples the input feature map at a fixed position, and the pooling layer reduces the spatial resolution at a fixed ratio, which limits its ability to capture various geometric transformations. In order to improve the detection accuracy of slender sprouts, the present invention introduces a deformable convolution DCNv2 design DBS (DCNv2-BN-SiLU) module and uses the DBS module to replace the CBS module in the original YOLOv5 backbone network, such as Figure 3 As shown in the figure. DCNv2 introduces an offset variable at each sampling point in the convolution kernel, which can achieve random sampling near the current position and enable the sampling grid to deform freely, thereby improving the model's ability to focus on the target and enhancing the model's performance. At the same time, the modulation mechanism is used to learn the weight of each sampling point and adjust the amplitude of the input feature at different spatial positions to reduce the impact of irrelevant image content on feature extraction and improve the model's ability to focus on features.

[0084] Furthermore, during the training process, the convolution kernel and offset are learned simultaneously. The specific calculation formula is as follows:

[0085] R={(-1,-1),(-1,0),...,(0,1),(1,1)}

[0086]

[0087]

[0088] g(a,b)=max(0,1-|ab|)

[0089] Where R is the size and expansion of the receptive field; P represents an arbitrary position, q enumerates all the integral spatial positions in the feature map x; x(p) represents the value of the point at all decimal positions after adding the offset; Δm is the modulation scalar;

[0090] (2) Neck network part. Feature upsampling is an important operation of deep neural networks. In the original YOLOv5, the feature pyramid network FPN is used to promote the propagation of semantic information from high to low. However, FPN has two obvious limitations. First, it cannot capture the necessary semantic information required for dense prediction tasks, resulting in potential mixing of target positions and noise positions during the upsampling process; second, its small receptive field limits its ability to utilize global feature information and may affect its expression ability and performance. In order to overcome these limitations, the present invention introduces the content-aware feature reconstruction operator CARAFE to replace the nearest neighbor interpolation in the neck network for upsampling during model construction. CAREFE includes two main components: a content-aware reconstruction module and a kernel prediction module. The kernel prediction module consists of three sub-modules: a channel compressor, a content encoder, and a kernel normalizer. The specific structure is as follows: Figure 4 As shown. First, the channel compression submodule is used to compress the input feature channel. Second, the content encoder submodule encodes the content of the compressed feature map to generate a reorganized kernel. Finally, the kernel normalization submodule applies a softmax function to each reorganized kernel. CARAFE has three major advantages. The first is a large receptive range. Information can be collected within a large receptive field, making better use of surrounding information. The second is high adaptability, which can dynamically adapt to instance-specific content. The third is lightweight and has high computational efficiency.

[0091] Furthermore, the working principle of the CARAFE upsampling operator is as follows: for an input image of shape H×W×C, a 1×1 convolution is used to compress its channels to H×W×C m ; For the compressed input feature map, use k encoder ×k encoder The convolutional layer predicts the upsampling kernel, the number of input channels is Cm, and the number of output channels is Expand the channel dimension in the space dimension to get the shape The upsampling kernel is normalized using softmax so that the sum of the convolution kernel weights is 1.

[0092] 4. Model Training

[0093] The cross entropy function Varifocal Loss is used to improve the loss function, and the preprocessed training set is used to train the constructed YOLOv5_DC model.

[0094] Experimental platform: Pytorch deep learning framework, Win10 64 operating system, GPU is RTX 3090 (24GB), CPU is 16vCPU Intel(R)Xeon(R)Platinum 8350C CPU@2.60GHz, memory is 43G. Set the number of training images per batch to 16, Epoch to 200, learning rate initialization to 0.01, momentum to 0.937, weight decay coefficient to 0.0005, and use stochastic gradient descent strategy SGD to optimize network parameters during training.

[0095] In this embodiment, the following process is included:

[0096] Initial weights are obtained by training part of the data set, which are used as pre-trained weights for subsequent models. Based on the idea of ​​transfer learning, the pre-trained weights are loaded into the YOLOv5_DC model as initial values ​​to avoid large fluctuations in loss values ​​during training and improve network convergence speed.

[0097] The training set is loaded into the YOLOv5_DC network model for training, and the loss function is improved using Varifocal Loss. The loss function plays a vital role in determining the performance of the model by measuring the difference between the predicted value and the actual value. In order to alleviate the class imbalance problem between positive and negative samples, the YOLOv5_DC constructed by the present invention uses the cross entropy function Varifocal Loss to calculate the loss value of the anchor box, which can be specifically expressed as:

[0098]

[0099] Among them, q is the intersection-and-union ratio between the target anchor box and the annotation box. When q=0, it means that the anchor box is a negative sample. When q>0, it means that the anchor box is a positive sample. p is the prediction probability of the anchor box for tea buds. α and γ are adjustable coefficients.

[0100] 5. Model testing and application instructions

[0101] After the YOLOv5_DC model training is completed, an image enhancement judgment module and an image enhancement module are also set at the image input end to perform image enhancement judgment and image enhancement during testing and actual application. Figure 5 As shown, the specific process is:

[0102] S1, calculate the average grayscale value AG of all tea images. If the AG value is greater than or equal to the preset value T1, the image is adaptively corrected for brightness and enters S2; if the AG value is less than T1, it is directly input into the trained YOLOv5_DC model;

[0103] S2, using the SLIC superpixel block algorithm for image segmentation, fully considering the similarity of features within the region and the difference of features between regions to improve the effectiveness of regional adaptive correction;

[0104] The SLIC algorithm combines color information and location information. The clustering process uses a weighted distance measurement value D, which is calculated as follows:

[0105]

[0106]

[0107]

[0108] Among them, l T Represents the color space in a grayscale image; [xy] T It represents the pixel position, [l k x k y k ] T represents the cluster center; d s is the spatial similarity; d c is the color similarity;

[0109] S3, calculating the regional average gray value AG for each superpixel block, if the regional AG value is greater than or equal to the preset value T2, then adaptive brightness correction is performed on the region, otherwise the regional brightness remains unchanged;

[0110] Adaptive brightness correction is performed on the area. This is a method of estimating the appropriate gamma value based on the average brightness. The gamma value γ required for correction is calculated as follows:

[0111]

[0112]

[0113]

[0114]

[0115] Among them, X∈[0 1] is the average brightness of the image, is the center of the histogram brightness, all pixels in the output image will be enhanced using the estimated gamma value.

[0116] S4, merging the corrected area and the uncorrected area to obtain a corrected and enhanced image to input into the trained YOLOv5_DC model.

[0117] 6. Experimental results

[0118] In this embodiment, the model use control scheme is: the original YOLOv5 network model, the YOLOv5 network model in which the second and third CBS in the backbone network are replaced with DBS, the YOLOv5 network model in which the nearest neighbor interpolation in the neck structure is replaced with the CARAFE operator, the YOLOv5 network model that uses the Varifocal Loss function instead of the Focal Loss function to calculate the loss value of the anchor frame, and the complete improved YOLOv5_DC network model proposed in this embodiment. The comparison verifies that the overall performance of the method of the present invention is the best.

[0119] Table 1 Ablation test results of the model on the tea dataset

[0120] Model DCN CARAFE VFL P(％) R(％) AP(%) Params(M) GFLOPs YOLOv5s × × × 83.8 77.8 85.1 7.02 15.9 YOLOv5s_D √ × × 87.6 82.5 88.5 7.07 14.3 YOLOv5s_C × √ × 87.2 81.3 87.9 7.15 16.2 YOLOv5s_V × × √ 86.4 80.5 87.2 7.02 15.9 YOLOv5s_DC √ √ × 88.2 82.3 88.8 7.20 14.8 YOLOv5s_DCV √ √ √ 89.8 84.2 90.6 7.20 14.7

[0121] As shown in Table 1, the results of the ablation experiment demonstrate the effectiveness of the proposed YOLOv5_DCV module. Specifically, the model accuracy is improved by 5.0%, the recall rate is improved by 6.4%, and the average precision (AP) is improved by 5.5%.

[0122] In this example, in order to verify the effectiveness of the model proposed in the present invention, the comprehensive performance of the current mainstream one-stage model is compared. The models involved in the comparison are: SSD model and YOLO series models (YOLOv3\YOLOv4\YOLOv5\YOLOX\YOLOv7\YOLOv8). Figure 6 , Figure 7 It can be seen that the YOLOv5_DC model proposed in the present invention has high average precision, small GLOPs value and best performance. The comparative experimental results in Table 2 show that compared with the mainstream single-stage target detection algorithms (YOLOv8, YOLOv7, YOLOX, YOLOv5, YOLOv4, YOLOv3, SSD), the AP of the method of the present invention is improved by 2.5%, 8.6%, 5.8%, 5.5%, 7.2%, 6.9%, and 5.9%, respectively.

[0123] Table 2 Performance comparison of different detection models

[0124] Model AP(%) P(％) R(％) Params(M) GFLOPs SSD 84.7 78.7 81.3 26.8 36.3 YOLOv3 83.7 81.4 76.9 61.5 155.3 YOLOv4 83.4 80.2 77.9 63.9 141.9 YOLOv5s 85.1 83.8 77.8 7.0 15.9 YOLOX_s 84.8 84.6 80.5 13.7 26.8 YOLOv7 82.0 86.6 83.6 9.3 26.7 YOLOv8s 88.1 87.3 78.6 11.2 28.8 YOLOv5s_DCV 90.6 89.8 84.2 7.2 14.8

[0125] The YOLOv5_DC model built in this embodiment can be applied to specific application scenarios after completing this step of testing, and includes the following processes:

[0126] Collect tea image data in real time;

[0127] Input the tea image data into the tea bud detection model constructed by the above construction method to perform tea bud detection;

[0128] Output detection information, including the specific location information of the tea buds in the tea image and various confidence levels;

[0129] Confidence screening is performed according to the pre-set threshold to obtain the final tea bud detection results.

[0130] Embodiment 2:

[0131] like Figure 8 As shown, the present invention also provides a tea bud detection device based on deep learning, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; a computer execution program is stored in the memory; a computer execution program of a tea bud detection model built by the construction method described in Example 1 is stored in the memory; when the processor executes the computer execution program stored in the memory, the processor can execute the tea bud detection method. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0132] The device may be provided as a terminal, a server or other forms of devices.

[0133] Figure 8 is a block diagram of a device shown as an example. The device may include one or more of the following components: a processing component, a memory, a power component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component generally controls the overall operation of the electronic device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component may include one or more processors to execute instructions to complete all or part of the steps of the above method. In addition, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0134] The memory is configured to store various types of data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0135] The power supply assembly provides power to various components of the electronic device. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device. The multimedia assembly includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia assembly includes a front camera and / or a rear camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0136] The audio component is configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. The I / O interface provides an interface between the processing component and the peripheral interface module, and the above-mentioned peripheral interface module can be a keyboard, a click wheel, a button, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

[0137] The sensor assembly includes one or more sensors for providing various aspects of status assessment for the electronic device. For example, the sensor assembly can detect the on / off state of the electronic device, the relative positioning of components, such as the display and keypad of the electronic device, and the sensor assembly can also detect the position change of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device and the temperature change of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0138] The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0139] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0140] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0141] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for building a tea bud detection model based on deep learning, characterized in that: The following steps are involved: Step 1: Collect and screen the original image data of tea leaves in real scenes, make a data set, and divide it into a training set and a test set according to a certain ratio; Step 2: annotate and amplify the training set images to obtain a preprocessed training set; Step 3, based on the YOLOv5 model structure, a YOLOv5_DC model that can focus on the slender morphological features of tea buds is built; the YOLOv5_DC model introduces deformable convolution DCNv2 in the backbone network to realize adaptive extraction of tea bud features, and introduces a lightweight upsampling operator CARAFE in the neck network to retain the semantic and spatial information of multi-layer features; The YOLOv5_DC model includes a backbone network, a neck network and a prediction end; the backbone network includes a DBS module, a C3 module and an SPPF module, the DBS module is composed of a variable convolution DCNv2, a batch normalization layer BN and a SiLU activation function layer, and is intended to achieve adaptive extraction of tea bud features; the C3 module uses a residual structure to reduce model parameters and extract more fine-grained features; the SPPF module is improved from the SPP module, and the SPPF module fuses local features and global features to enrich the expression ability of the output feature map; the neck network uses a feature pyramid network FPN and a path aggregation network PAN for feature fusion, so that the model obtains rich feature information; the prediction end detects targets of different sizes through three detection layers with feature maps of different sizes; Introducing deformable convolution DCNv2 into the backbone network to achieve adaptive extraction of tea bud features is as follows: The deformable convolution DCNv2 is introduced to design the DBS module, and the DBS module is used to replace the CBS module in the original YOLOv5 backbone network; the DCNv2 can achieve random sampling near the current position by introducing an offset variable at the position of each sampling point in the convolution kernel, and enable the sampling grid to deform freely, so as to improve the model's ability to focus on the target; at the same time, the modulation mechanism is used to learn the weight of each sampling point and adjust the amplitude of the input feature at different spatial positions to reduce the influence of irrelevant image content on feature extraction, so as to improve the model's ability to focus on features; During the training process, the convolution kernel and offset are learned at the same time. The specific calculation formula is as follows: Where R is the size and expansion of the receptive field; P represents an arbitrary position, and q enumerates all the integral spatial positions in the feature map x; Indicates the value of the point at all decimal places after adding the offset; is the modulation scalar; A lightweight upsampling operator CARAFE is introduced into the neck network. Specifically, when building the model, a lightweight upsampling operator CARAFE is introduced to replace the nearest neighbor interpolation in the neck network for upsampling. CAREFE includes a content-aware reorganization module and a kernel prediction module. The kernel prediction module consists of three submodules: a channel compressor, a content encoder, and a kernel normalizer. First, the channel compression submodule is used to compress the input feature channel. Second, the content encoder submodule encodes the content of the compressed feature map to generate a reorganized kernel. Finally, the kernel normalization submodule applies a softmax function to each reorganized kernel. Step 4: Use the cross entropy function Varifocal Loss to improve the loss function, and use the training set preprocessed in step 2 to train the YOLOv5_DC model in step 3; Step 5: Use the image data in the test set to test the trained YOLOv5_DC model and complete the construction of the final model.

2. The method for building a tea bud detection model based on deep learning as claimed in claim 1, characterized in that: After the YOLOv5_DC model training is completed, an image enhancement judgment module and an image enhancement module are also provided at the image input end, which are used to perform image enhancement judgment and image enhancement during testing and actual application. The specific process is as follows: S1, calculate the average grayscale value AG of all tea images. If the AG value is greater than or equal to the preset value T1, the image is adaptively corrected for brightness and enters S2; if the AG value is less than T1, it is directly input into the trained YOLOv5_DC model; S2, using the SLIC superpixel block algorithm for image segmentation, fully considering the similarity of features within the region and the difference of features between regions to improve the effectiveness of regional adaptive correction; S3, calculating the regional average gray value AG for each superpixel block, if the regional AG value is greater than or equal to the preset value T2, then adaptive brightness correction is performed on the region, otherwise the regional brightness remains unchanged; S4, merging the corrected area and the uncorrected area to obtain a corrected and enhanced image to input into the trained YOLOv5_DC model.

3. The method for building a tea bud detection model based on deep learning as claimed in claim 2, characterized in that: The SLIC superpixel block algorithm is used for image segmentation, combining color information and position information. The clustering process uses a weighted distance measurement value D, and the specific calculation method is: in, Represents the color space in grayscale images; It represents the pixel position. represents the cluster center; is the spatial similarity; is the color similarity; The adaptive brightness correction of the region is a method of estimating the appropriate gamma value based on the average brightness, and correcting the required gamma value The calculation method is: Among them, X∈[0 1] is the average brightness of the image, is the center of the histogram brightness, and all pixels in the output image will use the estimated Value is enhanced.

4. The method for building a tea bud detection model based on deep learning as claimed in claim 1, characterized in that: In step 2, the training set images are labeled and amplified, specifically: The original tea images collected in natural scenes were cropped with a length-width ratio of 1:1 according to the principle of retaining as many tea bud samples as possible, and their length and width were adjusted to 640×640 pixels and then divided into training and test sets; Data annotation: Use image annotation tools to manually annotate images. The annotation standard is single bud annotation. The annotated label file is saved in TXT format. Data enhancement: The annotated images are augmented by operations such as 90-degree and 180-degree flipping, vertical and horizontal mirroring, brightness enhancement, and adding Gaussian noise, and the corresponding annotation files are generated synchronously.

5. The method for building a tea bud detection model based on deep learning as claimed in claim 1, characterized in that: In step 4, the cross entropy function Varifocal Loss is used to improve the loss function, specifically: The cross entropy function Varifocal Loss is used to calculate the loss value of the anchor box, which can be expressed as: Among them, q is the intersection-and-union ratio between the target anchor box and the annotation box. When q=0, it means that the anchor box is a negative sample. When q>0, it means that the anchor box is a positive sample. p is the prediction probability of the anchor box for tea buds. α and γ are adjustable coefficients.

6. A tea bud detection method based on deep learning, characterized in that: The process includes: Collect tea image data in real time; Inputting the tea image data into the tea bud detection model constructed by the construction method according to any one of claims 1 to 5 to perform tea bud detection; Output detection information, including the specific location information of the tea buds in the tea image and various confidence levels; Confidence screening is performed according to the pre-set threshold to obtain the final tea bud detection results.

7. A tea bud detection device based on deep learning, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the tea bud detection model constructed by the construction method as described in any one of claims 1 to 5; when the processor executes the computer execution program stored in the memory, the processor executes the tea bud detection method.

Images