Tobacco plant number statistical method based on YOLOv8 and unmanned aerial vehicle data

Abstract

The invention relates to a tobacco plant number automatic counting method based on target detection. The method relates to the technical field of intelligent agriculture, and is different from a traditional plant counting method in that automatic counting is carried out on a multispectral tobacco leaf data set through a YOLOv8 target detection network. According to the method, a multispectral tobacco leaf data set shot by an unmanned aerial vehicle is selected as an image to be researched. Specifically, after tobacco leaf data set images of a multispectral unmanned aerial vehicle at different growth stages of tobacco leaves are collected, the collected data are subjected to splicing, cutting, rejection, synthesis of different wave bands and the like, then, an original YOLOv8 network is adopted to train RGB wave band images processed by Step2 and synthesized RGN wave band images, RGB wave band and RGN wave band models generated after training are obtained, and the obtained RGB wave band and RGN wave band models are subjected to multi-spectral unmanned aerial vehicle imaging. Obtaining a final network model; for a single-channel data set and a seven-channel data set in a spectrum, an original YOLOv8 network architecture needs to be modified to adapt to different input data, a model is trained, an optimal generated network model is selected, and finally a test set is utilized to evaluate a network generated after training and count the number of single tobacco plants. The invention provides a method for counting the plant number of the whole tobacco field.

Description

(I) Technical field

[0001] The present invention is an automatic counting method for the number of tobacco plants based on target detection. Specifically, a YOLOv8 target detection network and a modified YOLOv8 network are used to identify a multispectral tobacco leaf dataset and count the number of tobacco plants in the entire tobacco field. The present invention belongs to the field of smart agricultural technology. (II) Background technology

[0002] Precision agriculture (PA), also known as precision farming or site-specific crop management, is an approach that uses advanced technology to optimize agricultural practices and improve productivity. It involves observing, measuring, and responding to within-field and intra-field variability in crops. PA provides a data-driven and technology-enabled approach to optimize agricultural practices, improve sustainability, and increase productivity despite farmland variability. It enables farmers to make accurate and informed decisions that lead to higher crop yields.

[0003] In recent years, with the rapid development of artificial intelligence, computer vision, machine learning, and deep learning, these advanced technologies have been widely used in the agricultural field. Traditional agricultural tasks, such as plant counting and plant disease identification, have also been revolutionized. Deep learning, a data-driven method, usually requires large-scale data for network training, and using drones for data collection is not only simpler but also faster. Combined with computer vision technology, the use of deep learning methods to achieve automatic plant counting has broad application prospects.

[0004] The fundamental problem of plant counting is plant detection. In the field of deep learning, both OD (object detection) and semantic segmentation can accomplish this task, and OD is more commonly used. OD is a computer vision task used to identify and locate objects in images or videos.

[0005] In recent years, deep learning has significantly improved OD. There are anchor-free methods and anchor-based methods. The mainstream methods of anchor-based methods can be roughly divided into two categories: one-stage methods and two-stage methods. One-stage methods just do OD directly without explicitly proposing any region of interest proposals in advance, split the image into a grid, and predict the existence of objects and their bounding box coordinates on each grid cell. For two-stage methods, they first propose potential ROIs for the image, then refine the features of these regions, and finally classify them. Typical two-stage methods include R-CNN, Fast R-CNN, FastR-CNN, Mask R-CNN, FPN, Libra R-CNN, etc.

[0006] Transfer learning is a technique in deep learning. It uses large-scale pre-training data to obtain a pre-trained model of a deep neural network, which is the starting point for training in the target domain. Therefore, it can be regarded as a training strategy. Unlike training from scratch, the pre-trained model already has basic or general knowledge obtained from pre-training. Transfer learning has limited requirements for data. For some applications where it is difficult to collect enough data, transfer learning helps to avoid overfitting. At the same time, the generalization ability of the model is improved through pre-training.

[0007] Through the study of existing plant counting methods, we found that most of the current work on plant counting uses deep learning methods, mainly semantic segmentation methods and OD methods. The main crops are cereal crops, such as corn, maize, wheat, etc. Another common feature is the use of RGB visible light images as materials.

[0008] In existing plant counting work, the most commonly used images are three visible light bands (VIS): blue band (400-500nm), green band (500-600nm) and red band (600-700nm), which cover the wavelength range visible to the human eye. If a drone is equipped with a multispectral sensor, it can capture multispectral bands in addition to the visible light band. The essence of multispectral data is the reflection of light at different discrete wavelengths, which reflects the reflectivity of a point for a specific wavelength. Therefore, multispectral data can be regarded as images. Deep learning techniques for image processing can also be applied to multispectral images. Some bands are of great significance to agriculture and provide valuable information for crop monitoring and analysis. For example, the near-infrared (NIR) band (700-1300nm) can provide insights into plant health, photosynthetic activity, and water content. It is particularly useful in monitoring vegetation vitality, detecting stress conditions, and estimating crop biomass. The Red Edge (RE) band (700 -750nm) captures subtle changes in chlorophyll absorption that can indicate crop growth stage, nutritional status, and stress conditions. This is useful for distinguishing healthy from stressed vegetation. Therefore, for the specific purpose of plant counting, we recommend utilizing more relevant bands within the multispectral range.

[0009] YOLO is a series of OD networks. Unlike traditional OD methods involving multiple stages, the YOLO series of networks unifies detection into a single neural network. After one pass, the predicted bounding box coordinates and class probabilities are directly given. YOLO's general framework introduces a groundbreaking approach by defining OD as a regression problem. YOLO takes advantage of the advantages of One-Stage and achieves real-time detection, achieving a balance between speed and accuracy. For our specific purpose of plant counting, we only need to detect one class of plants, and the real-time requirement is very high. Therefore, YOLO is a suitable choice. In this work, we chose the last version of the series, YOLOv8.

[0010] This patent involves a tobacco plant counting method based on YOLOv8 and drone data. In this work, we propose to use images captured by drones to automatically count tobacco plants. The images include not only visible light images, but also multispectral images. (III) Summary of the invention

[0011] The technical problem to be solved by the present invention is:

[0012] 1) It is necessary to use multispectral images for plant counting;

[0013] 2) A dedicated multispectral tobacco plant dataset needs to be created;

[0014] 3) The architecture of YOLOv8 needs to be modified to adapt to different input data.

[0015] This invention focuses on a tobacco plant number counting method based on YOLOv8 and drone data. The purpose of this invention is achieved as follows:

[0016] 1) Step 1: Collect the dataset, collect the tobacco leaf dataset images taken by multispectral drones at different growth stages of tobacco leaves, and obtain 5 different spectral images for each photo, namely RGB image, NG image, NR image, RE image and NIR image;

[0017] 2) Step 2: Data preprocessing, including splicing, cutting, eliminating, and synthesizing different bands of the collected data;

[0018] 3) Step 3: Use the original YOLOv8 network to train the RGB bands processed in Step 2 and the synthesized RGN band images to obtain the RGB bands and RGN band models generated after training;

[0019] 4) Step 4: Improve the original YOLOv8 network architecture to adapt to different input data (single channel, 7 channels), train the improved model, and select the generated optimal network model;

[0020] 5) Step 5: Use the test set to evaluate the network generated after training and count the number of tobacco plants in a single field, and propose a method for counting the number of plants in the entire tobacco field.

[0021] Compared with the prior art, the advantages of the present invention are:

[0022] 1) Use deep learning-based object detection (OD) methods to automatically count plants in multispectral images;

[0023] 2) The advanced YOLOv8 network was used, and the network structure was modified to adapt to different band combinations, and a lot of data preprocessing was performed;

[0024] 3) To make our method more practical, we designed an algorithm that processes the entire plot image that needs to be observed. The counting accuracy is as high as 99.53%;

[0025] 4) To make our method more practical, we designed an algorithm to process the entire plot image that needs to be observed;

[0026] 5) The combination of drones, multispectral data and powerful deep learning methods shows good application prospects. (IV) Description of the drawings

[0027] Figure 1 It is a flowchart of the steps of a tobacco plant number counting method based on YOLOv8 and drone data of the present invention;

[0028] Figure 2 It is a comparison diagram of tobacco leaf images collected at different periods in the present invention;

[0029] Figure 3 These are 5 different spectral images in the present invention, (a) RGB visible light band image. (b) green band image (c) near infrared band image, (d) red band image (e) red edge band image. ;

[0030] Figure 4 These are different spectrum combination diagrams. (a) is a combination of seven bands; (b) is a combination of three bands of visible light; (c) is a combination of three bands of R, G, and NIR;

[0031] Figure 5 This is the structure diagram of the YOLOv8 network;

[0032] Figure 6 It is to modify the first layer of the YOLOv8 network to adapt to different inputs;

[0033] Figure 7 It is a combination of pictures testing an entire piece of land;

[0034] Figure 8 is a graphic representation of a slice. (V) Specific implementation methods

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0036] like Figure 1 It is a flowchart of the steps of a tobacco plant-tree statistical method based on YOLOv8 and drone data of the present invention. Figure 2 , Figure 3 This is the tobacco leaf drone multispectral dataset we collected. Figure 4 This is a different spectrum combination diagram. Figure 5 , Figure 6 The present invention is described in detail with reference to the network architecture diagram of FIG. The present invention proposes a tobacco plant number counting method based on the YOLOv8 network, including the following contents:

[0037] The specific implementation steps are as follows:

[0038] 1)Step 1: Collect dataset.

[0039] We need to visit the fields every 15 days or so during the 2023 field farming season to collect images of tobacco leaves at different growth stages. A complete dataset is a guarantee of model robustness. At the same time, continuous observation of the growth process of tobacco plants is conducive to estimating yields. Figure 2 As shown in the figure, we collected five different growth stages of tobacco leaves. Each photo was taken to obtain five different spectral images, namely RGB image, NG image, NR image, RE image and NIR image, as shown in the figure. Figure 3 shown.

[0040] 2)Step 2: Data preprocessing.

[0041] Step 2.1: Since the two cameras are located at different positions of the drone, there is still a slight deviation between the RGB image and other multispectral images under the same lens. Therefore, the coordinates are first calibrated when merging. Because the size of the spliced ​​image is too large to be input into the deep learning neural network, the spliced ​​image is divided into small blocks (640*640) before training. Then use Labelme to mark the plants in the picture.

[0042] Step2.2.: In order to utilize more bands besides RGB, we merge RGB, NR, NG, RE, and NIR together. Another band combination is G, R, and NIR, such as Figure 4 (c) is a combination commonly used in agriculture to detect vegetation, synthesizing the RGN three-band image.

[0043] 3)Step3: Use YOLOv8 network to train three-channel images.

[0044] Step 3.1: RGN band synthesis,In order to utilize more bands besides RGB, we merge RGB, NR, NG, RE and NIR together, as shown in Figure 4 (a). Figure 4 As shown in (c), another band combination is G, R and NIR. It is a commonly used combination in agriculture for detecting vegetation.

[0045] Step 3.2: Three-channel model training. In our invention, we used the YOLOv8 network to train the RGB band and the RGN band, and obtained the optimal model of the network. The YOLOv8 network architecture is as follows: Figure 5 As shown. YOLOv8 is an OD algorithm based on deep learning, which can achieve fast, accurate and robust OD and instance segmentation on high-resolution images. It consists of three parts: backbone part, neck part and head part.

[0046] 4)Step 4: Improve the original YOLOv8 network architecture.

[0047] Step 4.1: In addition to the traditional RGB image with three input channels, our data has more channel combinations. Visible RGB color images, as well as RGN three-channel images of red, green and near infrared, single-channel input images of narrow red band (NR), narrow green band (NG), near infrared band (NIR) and red edge band (RE), and all bands are combined together to become a 7-band combination.

[0048] Step 4.2: Training of single-channel and 7-channel models. Improve the YOLOv8 network to adapt to inputs from different channels. Here we make adjustments to the single-channel and 7-channel networks and improve the first layer of the backbone network. Figure 6 As shown in Figure 1, the specific improvements include the optimization of the first layer of the backbone network. To avoid overfitting, we adopted transfer learning and loaded the pre-trained weights before training with our target domain data. Since the structure of YOLOv8 has been modified, the weights cannot fully match this modified structure. Therefore, except for the first convolutional layer, we only loaded the remaining weights of the trained model.

[0049] 5) Step 5: Use the test set to evaluate the network generated after training and to count the number of tobacco plants in a single field and the number of plants in the entire tobacco field.

[0050] Step 5.1 Evaluate network performance.

[0051] This group of methods aims to explore the performance of different spectra. In addition to the five types of images obtained by drones (narrow red band (NR), narrow green band (NG), near infrared band (NIR) and red edge band (RE) and RGB), we evaluated the 7-channel and RGN channels in our methods. The performance was evaluated by using three common metrics: precision, recall, mAP and accuracy of plant counting.

[0052] Step 5.2 Statistical method of single tobacco tree.

[0053] In addition to quantitative analysis as an OB method, plant counting is another key method of our task. Accordingly, after the model training obtains the corresponding model, we will count the number of tobacco plants and trees in different bands;

[0054] Step 5.3: Statistical method for tobacco plants in the entire tobacco field.

[0055] In the previous method, we performed a statistical study on images as fragments. In addition, the count results were reported as the sum of all image fragments. From the application point of view, the user is more interested in knowing the number of plants in a given field. For this reason, we designed another study using the entire field. Select a plot called plot-s9, such as Figure 7 (a). It can be cut into 9 slices of 640×640 size. The reason for using 9 slices is that they cover the left adjacent, right adjacent, top adjacent and bottom adjacent cases.

[0056] In training, it is still segmented into small-sized fragments (640×640). In testing, it is not reasonable to input the entire image into the neural network. Whether resizing or keeping the initial size of the image, the detection accuracy will decrease. The detection accuracy is highest when the slice has the same size as the training image. However, the plant part located at the edge of the image is detected as an instance. A plant is cut into two parts, and these two parts are detected in adjacent images. If the plant is segmented into two slices, they may be detected as two instances. For the entire land, it is no longer accurate to count the number of plants by simple summation, because the plant at (the boundary of the slice) can be counted twice. Therefore, we designed an algorithm to count the plants on a large piece of land. Figure 8 As shown in (a), a tobacco leaf is located at the cut line between two slices, and it is divided into two halves or one large and one small. YOLOv8 is powerful in detecting plants even if it is only a small part. When the slices are put together, a plant will be counted twice because it is detected twice in two slices and two detected non-intersecting bounding boxes. It is difficult to tell whether one plant is detected twice or two plants are detected separately.

[0057] To address this problem, in our approach, the entire image is segmented into slices with overlapping areas. Segmenting the image in this way has two benefits. First, if the plant happens to be located on the slice line, the overlapping slices make each of the parts larger and guarantee a higher detection confidence. Second, when the detection results of all slices are put together, the bounding boxes of the two parts of the plant detected from one slice are overlapping, such as Figure 8 (b) If this happens, only one plant is counted. We conducted two sets of experiments to evaluate the post-processing algorithm. The final results show that the RGN band has a better advantage in the statistics of the entire tobacco tree.

[0058] The improvement goal of the present invention is to use the YOLOv8 network to automatically count tobacco trees, thereby optimizing agricultural practices, improving sustainability, and increasing productivity in the face of changes in farmland. It enables farmers to make accurate and informed decisions, thereby increasing crop yields.

[0059] The present invention uses a multispectral tobacco leaf dataset as a test image, with single channel, three channels and seven channels. The size of a single tobacco field image is 640×640 and the size of a whole tobacco field image is 1800×1800. The present invention is trained on different channels to obtain the optimal model corresponding to different channels, and these models are used to detect the number of tobacco plants in different band images. In addition, the present invention uses the evaluation indicators of accuracy, racall, and mAP, and a key indicator is the accuracy of plant counting. Before defining these indicators, the true positive (TP), false positive (FP), false negative (FN) and true negative (TN) are defined. In OD, TP refers to correct detection when the predicted class matches and the IoU is greater than the threshold. FP refers to false detection when the class does not match or the IoU is less than the threshold. FN is the number of positive instances that are not detected. TN is not applicable to OD. In this work, since there is only one detection class: tobacco plant, only positive is meaningful and negative is meaningless. TP indicates that the tobacco plant has been detected as a tobacco plant. FP indicates that other substances have been detected in the tobacco plant.

[0060] The precision is calculated by dividing the number of TPs by the total number of positive detections, as shown in Formula 1. It indicates the ability of the model to identify only relevant objects and also provides insights into the ability of the model to avoid false positive detections. A high precision indicates a low probability of misidentifying background or non-target objects as positive detections.

[0061]

[0062] The second metric is recall. It is the ability of a model to find all relevant cases. The number of true positive detections divided by the sum of true positive detections and false negative detections is shown in Formula 2. It represents the proportion of correctly identified positive instances to all actual positive instances of a given dataset. A higher recall value indicates that the model has a lower missing rate of relevant objects and is able to effectively capture most positive instances.

[0063]

[0064] In addition to the three commonly used OD indices, plant counting accuracy is the metric we are most concerned about, as shown in Formula 3. This ratio is the predicted stand number divided by the actual stand number.

[0065]

[0066] In addition to quantitative analysis as an OB method, plant counting is also a method. In a set of counting methods, our confidence threshold is set to 0.25 and the IoU threshold is set to 0.5. The data used in this group is a summary of all data taken in five periods, of which 1818 images are used for training and 456 images are used for testing the method. By comparing various indicators, the results obtained are shown in Table 1. Table 1 shows that the RGN spectral combination in the present invention maintains the best plant counting results.

[0067] Table 1. Comparison of accuracy in different bands

[0068]

[0069] Note: The format of training data and test data is band-date-number of data images. "Channels" indicates the number of channels of input data, "Precision" indicates the average precision on the predicted box, and "Recall" indicates the recall rate.

[0070] Correspondingly, after the model training obtained the corresponding model, we counted the tobacco plants and trees in different bands. The statistical results are shown in Table 2. The results show that the RGN channel in the present invention still maintains the best plant and tree counting results.

[0071] Table 2: Results of tobacco tree counting in different bands

[0072]

[0073] Note: The format of training data and test data is band-date-number of data images. "Channels" means the number of channels of input data. "True Plants" means there are actually 3109 plants in my test images (456 images). "Plants test" means the number of plants counted by the network after I gave these 456 images to the network.

[0074] The present invention compares the tobacco plant statistics of different bands in the whole tobacco field. As shown in Table 3, the actual number of plants in plot 9 is 233. Obviously, the plants in the overlapping bands are counted repeatedly. If two detected bounding boxes overlap, the counting algorithm we designed only counts once. Figure 8 As shown in (b), the plant marked with a circle should have been one, but two were detected in two slices at the same time. They are marked because two or more detected bounding boxes overlap. 43 plants were counted repeatedly. After repeated counting, the final count was 229. After post-processing, the detection accuracy in the present invention was as high as 98.28%. We found that 0602 obtained the best counting accuracy and 0629 obtained the second best counting accuracy. Then, the accuracy decreased along the timeline. The reason is that the tobacco plants grew larger and overlapped along the timeline. Therefore, the detection accuracy of the instance was affected. However, even on the last day of 0823, the result still remained at a high level of 95.65%.

[0075] Table 3: Test results for a large plot of land

[0076]

[0077] Note: GT-1 is the sum of 9 GTs. Test-1 is the total number of plants detected in 9 plots. “Test-o” refers to the number of plants detected multiple times. GT-2 is the actual number of plants in plot-s9. Test-2 is the calculation result of our post-processing algorithm.

Claims

1. A tobacco plant number counting method based on YOLOv8, characterized in that: The method specifically comprises the following steps: 1) Step 1: Collect the dataset, collect the tobacco leaf dataset images taken by multispectral drones at different growth stages of tobacco leaves, and obtain 5 different spectral images for each photo, namely RGB image, NG image, NR image, RE image and NIR image; 2) Step 2: Data preprocessing, splicing, cutting, and eliminating the collected data; 3) Step 3: Use the original YOLOv8 network to train the RGB bands processed in Step 2 and the synthesized RGN band images to obtain the RGB bands and RGN band models generated after training; 4) Step 4: Improve the original YOLOv8 network architecture to adapt to different input data (single channel, 7 channels), train the improved model, and select the generated optimal network model; 5) Step 5: Use the test set to evaluate the network generated after training and count the number of tobacco plants in a single field, and propose a method for counting the number of plants in the entire tobacco field.

2. The tobacco plant tree statistics method based on YOLOv8 according to claim 1 is characterized in that: The other two channel (single channel, 7 channel) model training included in Step 4 is specifically as follows: Step 4.1: In addition to the traditional RGB image with three input channels, our data has more channel combinations. Visible RGB color images, as well as RGN three-channel images of red, green and near infrared, single-channel input images of narrow red band (NR), narrow green band (NG), near infrared band (NIR) and red edge band (RE), and all bands are combined together to become a 7-band combination. Step 4.2: Training of single-channel and 7-channel models. We improved the YOLOv8 network to adapt to different channel inputs. The specific improvements include the optimization of the first layer of the backbone network. To avoid overfitting, we used transfer learning and loaded the pre-trained weights before training with our target domain data. Since the structure of YOLOv8 has been modified, the weights cannot fully match this modified structure. Therefore, except for the first convolutional layer, we only loaded the remaining weights of the trained model.

3. The tobacco plant tree statistics method based on YOLOv8 according to claim 1, characterized in that: Step 5 includes evaluating network performance and counting the number of tobacco plants in a single block and the number of tobacco plants in the entire tobacco field. Specifically, it includes: Step 5.1 Evaluate network performance. This group of methods aims to explore the performance of different spectra. In addition to the five images obtained by the UAV, four single-channel images of narrow red band (NR), narrow green band (NG), near infrared band (NIR) and red edge band (RE) and RGB three-channel images, we evaluated the method on synthetic 7-channel and RGN three-channel images. Step 5.2 Statistical method of single tobacco tree. Plant counting is another key aspect of our task. We tackle the challenge of plant identification in different bands by training multiple band models to automatically identify and count the number of individual tobacco plants. Step 5.3: Statistical method for tobacco plants in the entire tobacco field. In the previous two methods, we performed statistical studies on the image as a single piece. From the application point of view, the user is more interested in knowing the number of plants in a given field. To this end, we designed two methods to count tobacco plants in a large field. In our method, the whole image is segmented into slices with overlapping areas. An image covering 9 RGN slices of left adjacent, right adjacent, upper adjacent and lower adjacent is used as the research object of this method. The two methods are: a. Slice overlap: To improve detection accuracy, the entire image is segmented into slices with overlapping areas. This has two benefits: first, if the plant is located on the slice line, overlapping slices can ensure that the plant in each section is larger, improving detection confidence; second, when the detection results of all slices are merged, plants located in the overlapping area of ​​the slices are prevented from being counted repeatedly. b. Double counting handling: An algorithm is designed to handle the double counting problem. When two detected bounding boxes overlap, they are counted only once. This ensures the accuracy of the number of plants in a field. These two methods allow for accurate plant counts across the entire field, addressing the challenges of plant overlap and duplicate counts, ensuring the reliability of the results.

Images