Method and system for tobacco plant segmentation and counting based on large-scale image of unmanned aerial vehicle

Abstract

The present application belongs to the technical field of tobacco plant quantity statistics, and discloses a tobacco plant segmentation and counting method and system based on large-scale images of unmanned aerial vehicles. The present application is based on large-scale tobacco field segmentation image data sets, integrates MobileNetV2 and channel attention mechanism ECA on DeeplabV3+ network, constructs MED-Net used for tobacco field segmentation framework, so as to realize tobacco field segmentation; further, a tobacco plant background removal method is used to remove background information; in addition, the present application also establishes a pixel-level-based tobacco plant contour screening method, which is conducive to screening small target contours such as weeds and improving the accuracy of tobacco plant contour extraction. The present application can effectively realize tobacco field segmentation and tobacco plant counting in large-scale unmanned aerial vehicle remote sensing images, reduce the over-segmentation problem caused by the need for a large amount of cutting of unmanned aerial vehicle remote sensing images due to insufficient device performance, realize the statistics of the number of tobacco plants without a large amount of data labeling, and improve the efficiency and accuracy of tobacco plant statistics.

Description

Technical Field

[0001] This invention belongs to the field of tobacco plant quantity statistics technology, specifically involving a method and system for tobacco plant segmentation and counting based on large-scale images from unmanned aerial vehicles (UAVs). Background Technology

[0002] Tobacco plant count refers to the accurate counting of tobacco seedlings after transplanting, which is crucial for ensuring the quality and yield of tobacco. Traditional tobacco plant count mainly relies on manual counting, which suffers from problems such as large workload, low efficiency, high cost, and difficulty in guaranteeing statistical accuracy, resulting in inaccurate and incomplete information on tobacco planting monitoring.

[0003] In the statistical analysis of tobacco plant counts based on UAV imagery, the UAV's flight altitude not only affects the detail and accuracy of image reconstruction but also processing efficiency and tobacco plant detection accuracy. Higher flight altitudes mean a greater number of targets in a single image, resulting in higher data acquisition and algorithm processing efficiency, but less detailed information, which tests the algorithm's ability to acquire target features. Lower flight altitudes yield richer detailed information and are easier for the algorithm to process, but they reduce data acquisition efficiency, increasing the number of images collected from the same area and thus reducing processing efficiency.

[0004] Currently, some crop quantity statistics based on multispectral and hyperspectral data have limitations in practical applications due to high collection costs, complex data processing, and difficulties in data integration. Furthermore, some deep learning-based tobacco plant detection methods require extensive data annotation and detection model training, and have failed to achieve tobacco field segmentation and tobacco plant region extraction.

[0005] Currently, some target detection algorithms, in order to detect crops in UAV images, can only reduce the UAV's flight altitude to obtain more detailed image information, or, with limited computing resources, can only crop the large-scale remote sensing images acquired by the UAV into small-scale remote sensing images before detecting tobacco plants. This target detection method is prone to incomplete plant outlines and has problems such as over-segmentation, duplicate statistics, and low statistical efficiency. Summary of the Invention

[0006] The purpose of this invention is to propose a method for tobacco plant segmentation and counting based on large-scale images from unmanned aerial vehicles (UAVs). This method combines UAV remote sensing image technology with deep learning and image processing methods, which can effectively segment tobacco fields in large-scale UAV images, further remove the background of tobacco plants in the segmented tobacco fields, and facilitate the extraction and screening of tobacco plant outlines, thereby achieving effective counting of tobacco plant numbers.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for tobacco plant segmentation and counting based on large-scale images from unmanned aerial vehicles (UAVs) includes the following steps:

[0009] Step 1. Build a tobacco field segmentation framework MED-Net based on DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve segmentation of tobacco field areas and non-tobacco planting areas;

[0010] Step 2. Based on the HSV color space, a tobacco plant background mask is created. Based on the tobacco field segmentation results obtained in Step 1, small background areas, including soil and plastic film, are further removed to obtain an image that retains only the tobacco plants.

[0011] Step 3. Perform Gaussian filtering, morphological denoising, and opening operation denoising on the image obtained in Step 2 that only retains tobacco plants. Then extract the tobacco plant contours. After filtering and counting the extracted tobacco plant contours, obtain the number of tobacco plants.

[0012] Furthermore, based on the aforementioned method for tobacco plant segmentation and counting using large-scale UAV images, this invention also proposes a corresponding tobacco plant segmentation and counting system based on large-scale UAV images, which adopts the following technical solution:

[0013] A tobacco plant segmentation and counting system based on large-scale images from unmanned aerial vehicles (UAVs) includes the following modules:

[0014] The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas.

[0015] The tobacco field background removal module is used to create a tobacco plant background mask based on the HSV color space. Based on the obtained tobacco field segmentation results, it further removes small background areas, including soil and mulch, to obtain an image that retains only the tobacco plants.

[0016] The tobacco plant count module is used to perform Gaussian filtering, morphological denoising, and opening operation denoising on the image that retains only tobacco plants. Then, the tobacco plant contours are extracted, and the extracted tobacco plant contours are filtered and counted to obtain the number of tobacco plants.

[0017] The present invention has the following advantages:

[0018] As described above, this invention discloses a method and system for tobacco plant segmentation and counting based on large-scale UAV images. The method establishes a UAV-based tobacco field segmentation image dataset and integrates the MobileNetV2 extraction network and the channel attention mechanism ECA on the DeeplabV3+ network to construct the MED-Net framework for tobacco field segmentation, suitable for segmenting tobacco fields in large-format UAV images. Furthermore, this invention proposes a method for tobacco plant background removal, which removes background information such as soil and plastic film, avoiding incomplete outlines of some tobacco plants caused by over-segmentation and improving the accuracy and efficiency of tobacco plant statistics. Additionally, this invention establishes a pixel-level tobacco plant outline filtering method to filter the outlines of small targets such as weeds, improving the accuracy of tobacco plant outline extraction and enabling tobacco plant counting without requiring extensive data annotation. This invention effectively achieves tobacco field segmentation and tobacco plant counting in large-scale UAV remote sensing images, reducing the over-segmentation problem caused by excessive cropping of UAV remote sensing images due to insufficient equipment performance, and improving the efficiency and accuracy of tobacco plant statistics. Compared to tobacco plant counts based on target detection algorithms, the method of this invention does not require extensive tobacco plant data annotation and tobacco plant detection model training, making it easier to count the number of tobacco plants over a large area. Attached Figure Description

[0019] Figure 1 This is a flowchart of a tobacco plant segmentation and counting method based on large-scale images from UAVs, as described in an embodiment of the present invention.

[0020] Figure 2 This is a network structure diagram of the tobacco field segmentation framework MED-Net constructed in an embodiment of the present invention.

[0021] Figure 3 This is a flowchart of tobacco plant background removal in an embodiment of the present invention.

[0022] Figure 4 This is a flowchart of tobacco plant outline screening and counting in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram showing the variation of the loss function values ​​of the model of this invention and the comparison model with Epoch.

[0024] Figure 6 This is a schematic diagram illustrating segmentation using different models. Among them, Figure 6 In the diagram, (a) represents the original image, (b) represents the label file of the corresponding image, and the green areas in (c) to (g) represent the segmentation results of different models such as PSPnet, Unet, DeeplabV3+_Xception, DeeplabV3+_MobileNetV2, and MED-Net.

[0025] Figure 7 This is a schematic diagram illustrating the background removal effect of tobacco plants in different plots. Figure 7 In the diagram, (a) to (e) represent different plots of land, row (i) represents the tobacco field division map, row (ii) represents the tobacco plant background removal map, and row (iii) represents the magnified effect of a local area of ​​the tobacco plant background removal map.

[0026] Figure 8 This is a schematic diagram showing the statistical results of the number of tobacco plants in different plots. Among them, Figure 8 In the diagram, (a) to (e) represent the statistical results of the number of tobacco plants in different plots. The first line (i) shows the statistical results of the number of tobacco plants in different plots, the second line (ii) shows the overall statistical results of the number of tobacco plants, and the third line (iii) shows a magnified view of a local area of ​​the tobacco plant statistical results. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0028] Example 1

[0029] This embodiment 1 describes a method for tobacco plant segmentation and counting based on large-scale images from UAVs. This method uses the acquired large-scale UAV images to construct a tobacco field segmentation framework, MED-Net, to segment tobacco field areas from non-tobacco planting areas. On this basis, a tobacco plant background mask is further established based on the HSV color space to remove small-sized background areas, including soil and mulch, resulting in an image that retains only tobacco plants. Finally, the image is subjected to Gaussian filtering, morphological denoising, and opening denoising, and the tobacco plant contours are extracted. After filtering and counting the extracted tobacco plant contours, the number of tobacco plants is obtained.

[0030] The large-scale images mentioned here specifically refer to images with large dimensions. Typically, drones fly at an altitude of about 100 meters, and traditional target detection algorithms struggle to detect such a large number of targets.

[0031] like Figure 1 As shown, the tobacco plant segmentation and counting method based on large-scale UAV images in this embodiment includes the following steps:

[0032] Step 1. Build a tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve segmentation between tobacco field areas and non-tobacco planting areas.

[0033] Segmentation of tobacco fields in UAV imagery is a binary classification problem. This invention uses the DeepLabV3+ network to segment tobacco fields, but its segmentation performance is poor for complex field boundaries and similar crops such as cornfields, which cannot meet the needs of tobacco field segmentation. Improvements are needed to address the problems of segmentation models in tobacco field segmentation applications.

[0034] Based on this, this invention further integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA on the foundation of the DeepLabV3+ network framework, constructing the MED-Net framework for tobacco field segmentation, such as... Figure 2 As shown.

[0035] The encoder of the tobacco field segmentation framework MED-Net includes the MobileNetV2 module, the ECA-Net module, and the ASPP module.

[0036] In the encoder section, the Xception network in the DeepLabV3+ network is first replaced with the MobileNetV2 module to reduce the number of feature extraction network layers, parameters, model complexity, and running speed. Secondly, the high-level semantic features obtained by the MobileNetV2 module are sent to the ECA-Net module for adaptive attention weighting of the channel dimensions of the feature maps, helping the model better understand and utilize information from different channels and increasing the model's attention to different channels. Then, the deep features are enhanced and extracted using dilated convolutions of different rates through the ASPP module, followed by concat stacking. Finally, the features are compressed using 1×1 convolution operations, and the feature channels are adjusted before the enhanced features are transmitted to the MED-Net decoder.

[0037] In the decoder section, the low-level semantic features acquired by the MobileNetV2 module in the encoding stage are first processed by the ECA-Net module to calculate the attention weight of each feature channel, suppressing minor feature channels and increasing the weight of important feature channels. Then, the weighted features are concatenated with the upsampled enhanced features output from the encoding stage after adjusting the feature channels through a 1×1 convolution. After adjusting the feature channels through a 3×3 convolution, bilinear interpolation upsampling is performed to obtain the output prediction result with the same size as the original input image of the input MED-Net.

[0038] The feature extraction network MobileNetV2 reduces the number of model parameters by 20% compared to MobileNetV1, while achieving significantly better accuracy. The main improvements over MobileNetV1 are as follows:

[0039] 1. Added the Inverted Residuals structure.

[0040] The inverse residual structure first uses 1×1 convolutions to increase dimensionality, then extracts features through 3×3 depthwise convolutions (DW), and finally uses 1×1 convolutions to reduce dimensionality. The entire process is "expansion-convolution-compression," resembling a spindle shape. The convolution operation uses 3×3 DW instead of the standard 3×3 convolution. The first two ReLU activation functions of MobileNetV1 are replaced with ReLU6. The ReLU6 function is shown in the following formula:

[0041] y=ReLU6(x)=min(max(x,0),6).

[0042] The inverse residual structure in MobileNetV2 deepens the network layers by first increasing the dimensionality and then decreasing it, which enhances gradient propagation and feature representation, helping the network to better capture details and features in images.

[0043] Meanwhile, ReLU6 can provide better nonlinear expression capabilities compared to ReLU, and can directly truncate the part of the input value greater than 6 to 6, reducing the computational load of the network and improving performance. By replacing the first two ReLUs with ReLU6, MobileNetV2 can further improve computational efficiency while maintaining a certain level of accuracy.

[0044] 2. Add linear bottleneck structures (Linear Bottlenecks).

[0045] The linear bottleneck structure maps a high-dimensional space to a low-dimensional space and uses linear activation in the final convolutional layer to reduce the loss of low-dimensional feature information. The linear bottleneck structure consists of two convolutional layers: the first 1×1 convolutional layer reduces the number of input channels to 1 / 6 of the original, and the second 3×3 depthwise convolutional layer performs feature extraction and expansion, restoring the number of channels back to the original size. Simultaneously, linear activation is used in the final layer to reduce the loss of low-dimensional feature information, reducing computational cost and the number of model parameters, thus improving the network's computational efficiency while maintaining a certain level of accuracy.

[0046] The ECA-Net module learns a weight vector to weight features across different channels, enhancing the model's ability to represent features from different channels and showing great potential for improving model performance. The ECA-Net module first processes the input features using Global Average Pooling (GAP) to obtain all features without dimensionality reduction. Then, it performs a fast one-dimensional convolution of size k with weight sharing after all features to learn the features and generate the weight percentages for each channel. Finally, it combines the original input features with the channel weights to obtain features with channel attention.

[0047] While avoiding dimensionality reduction, ECA-Net proposes a local cross-channel interaction strategy and an adaptive method for selecting the size of the one-dimensional convolution kernel, which effectively captures cross-channel interaction information while significantly reducing model complexity.

[0048] The ECA-Net module only calculates the feature channel y. i Information exchange between it and its k neighboring channels, and through the Sigmoid function. σ Generate the weight percentage for each channel, and the channel attention weight w. i The calculation formula is as follows:

[0049]

[0050] In the formula, w i This represents the weight of the i-th channel. Indicates y i The set of k adjacent channels, j = 1, 2, ..., k, ω j Represents the learning parameters, Indicates adjacent channels.

[0051] To further improve the feature extraction performance of ECA-Net, a 1D convolution C1D with a kernel size of k is used to realize information exchange between channels and achieve the sharing of the same learning parameters. The above formula is then rewritten as follows:

[0052] w=σ(C 1D k (y)).

[0053] Where w represents the weight, C1D k (y) represents a 1D convolution C1D with kernel size k performing a convolution operation on channel y.

[0054] Since the ECA module is designed to capture local cross-channel interactions appropriately, it is necessary to determine the local cross-channel interaction coverage, i.e., the kernel size k of the 1D convolution, where k is proportional to the channel dimension C.

[0055] Given the channel dimension C, the adaptive function for calculating the kernel size k is:

[0056]

[0057] Among them, |t| odd Let r = 2 and b = 1 represent the odd number closest to t.

[0058] Before training the tobacco field segmentation framework MED-Net, the dataset is prepared as follows:

[0059] Step 1.1. Data Acquisition: The data collection period is the early stage of tobacco growth. Data collection is conducted using different flight routes planned according to different flight altitudes to obtain raw image data of the tobacco field. Simultaneously, due to the need to ensure the data acquired by the drone can be stitched together (e.g., using DJI Terra to stitch flight data), some areas of the different raw image data overlap. To avoid double counting of the same areas when counting tobacco plants in the experimental data, the collected raw image data needs to be stitched to remove overlapping areas. The stitched data is then cropped to obtain the tobacco plant counting experimental data. To reduce the amount of data processing, Global Mapper software is used for data preprocessing to obtain drone tobacco field data, such as... Figure 3 As shown in (a).

[0060] Step 1.2. Select data with differences from the original tobacco field image data collected by the UAV to construct training data for the tobacco field segmentation model; after compressing the selected training data, use the image annotation tool Labelme to annotate the data, and use Mosaic and Augmentor tools to enhance the annotated data.

[0061] Mosaic data augmentation involves randomly selecting four original images, adjusting their size and position, and then stitching them together to obtain new data. The Augmentor tool uses various methods such as rotation, scaling, cropping, and deformation to achieve data augmentation.

[0062] Step 1.3. After data augmentation, divide the training set and test set according to the ratio for MED-Net training.

[0063] The dataset obtained here is a large-scale image dataset from UAVs. This dataset is used to train the tobacco field segmentation framework MED-Net, and the trained MED-Net is used for tobacco field image segmentation.

[0064] After segmenting tobacco field images based on MED-Net, most non-tobacco field areas such as cornfields, grasslands, forests, and roads can be removed, resulting in images like... Figure 3The tobacco field division diagrams shown in (b) to (c) are as follows.

[0065] Step 2. Based on the HSV color space, a tobacco plant background mask is created. Based on the tobacco field segmentation results obtained in Step 1, small background areas, including soil and plastic film, are further removed to obtain an image that retains only the tobacco plants.

[0066] After completing tobacco field image segmentation based on MED-Net, most non-tobacco field areas such as cornfields, grasslands, forests, and roads can be removed. However, the presence of small background areas such as mulch film, soil, and weeds between tobacco plants can interfere with subsequent tobacco plant counts, and the MED-Net image segmentation model struggles to separate these areas from the tobacco plants. Therefore, this invention constructs a method for removing small background areas to achieve fine segmentation of tobacco fields and background regions.

[0067] Step 2.1. First, convert the RGB format tobacco field segmentation image to HSV color space to obtain the following: Figure 3 The image shows the HSV color space diagram of the tobacco field (d). Next, by adjusting the hue (H), saturation (S), and value (V) components, the background removal effect of the tobacco plants was observed, and the corresponding H, S, and V values ​​were statistically analyzed. H, S, and V threshold ranges were established, resulting in the following... Figure 3 The color mask of the tobacco field shown in (e) is shown in the middle.

[0068] Step 2.2. Overlay the obtained tobacco field color mask image with the RGB format tobacco field segmentation image to mask the background area of ​​tobacco plants in the remote sensing image. When the two images have the same size, define the tobacco field segmentation image as the first input array src1, the tobacco field color mask image as the second input array src2, and dst as the output array of the same size as the input arrays, where the default value of all elements is 0, i.e., a black mask image; then we have:

[0069]

[0070] Where I represents the image, mask(I) represents the mask image, and dst(I) represents the output image.

[0071] By performing an AND operation between the tobacco field color mask and the tobacco field segmentation image, the content of the corresponding area in the tobacco field segmentation image is displayed in the white areas of the mask image (mask(I)≠0), and the corresponding area in the tobacco field segmentation image is occluded in the black areas of the mask image (mask(I)=0), thus separating the tobacco plants from the background area, as shown below. Figure 3 The background of the tobacco field shown in (f) has been removed.

[0072] Step 3. Perform Gaussian filtering, morphological denoising, and opening operation denoising on the image obtained in Step 2 that only retains tobacco plants. Then extract the tobacco plant contours. After filtering and counting the extracted tobacco plant contours, obtain the number of tobacco plants.

[0073] Step 3.1. Image Grayscale Conversion: Grayscale conversion compresses the original image data, improves data processing speed, reduces interference, and avoids striping distortion. The grayscale value Y of each component is obtained using the following formula.

[0074] Y = 0.229R + 0.587G + 0.114B;

[0075] In the above formula, R, G, and B represent the three channel components corresponding to the RGB color space.

[0076] Step 3.2. Gaussian filtering to smooth the image: This smoothing process reduces the variation between pixels in the image and is used to remove noise from the image before edge detection, thereby improving image quality and accuracy.

[0077] The input image is defined as S(u,v). After processing with a two-dimensional Gaussian function, G(u,v) is obtained. The value of the center pixel of the template is replaced by the weighted average gray value of the pixels in the neighborhood. The two-dimensional Gaussian function processing is shown below:

[0078]

[0079] Step 3.3. Image binarization operation: As shown in the following formula, through threshold binarization operation, set the grayscale threshold T1. When the grayscale value is less than or equal to the threshold T1, set its grayscale value to 0. When it is greater than the threshold T1, set its grayscale value to 255.

[0080]

[0081] After grayscale conversion, Gaussian filtering, and binarization, the dimensionality of the image data is reduced, and the variation between pixels in the image is decreased. This separates the tobacco plants from the background, highlighting their outlines. The processing result is as follows: Figure 4 As shown in (a).

[0082] Step 3.4. Image morphology and opening operation for denoising:

[0083] First, define a 3×3 cross-shaped structural element M as shown in the following formula. Perform morphological operations on the image through the structural element M to return a structural element of the specified shape and size.

[0084]

[0085] Then, opening operations are used to perform erosion followed by dilation to separate the tobacco plants from part of the background area. Irrelevant external white pixels around the tobacco plants are removed, while other parts remain unchanged, resulting in the following: Figure 4 The result is shown in (b).

[0086] Step 3.5. Contour Extraction:

[0087] By compressing elements in the horizontal, vertical, and diagonal directions, and retaining only the endpoint coordinates of those directions, the outer contour information of the tobacco plant is output. The tobacco plant contour is then drawn using this contour information. Figure 4 As shown in (c).

[0088] Step 3.6. Contour Filtering:

[0089] Contour filtering consists of two processes: the first is based on the Euclidean distance between the centroids of the contours, and the second is based on the pixel area inside the contour.

[0090] The centroid of the contour refers to the geometric center of the closed contour.

[0091] Specifically, the contour selection process based on the Euclidean distance between the centroids of the contours is as follows:

[0092] The coordinates of the centroid are determined by calculating the rectangular moments of the profile, as shown in the following formula:

[0093]

[0094] Where, x i and y i M represents the x-coordinate and y-coordinate of the centroid, respectively. 00 M represents the zeroth moment of the profile. 01 and M 10 These represent the first moments of the profile.

[0095] Let there be two contours C i C j The Euclidean distance between the centers of mass is d i,j The distance threshold is T d The pixel areas of the outlines are A i A j When the Euclidean distance d between the centroids of two contours i,j Greater than or equal to the distance threshold T d When the distance is less than the distance threshold T, retain the contour information of both contours; d When comparing two contours C i C jThe pixel area size is determined, and the outline information of the larger pixel area is retained to ensure that only the outline information of one tobacco plant is retained within the range of less than the distance threshold.

[0096] The specific screening strategy is shown in the following formula:

[0097]

[0098] The contour results after filtering by the centroid distance of the contour, such as Figure 4 As shown in (d).

[0099] The contour filtering process based on the pixel area inside the contour is as follows:

[0100] Set the pixel area threshold for the tobacco plant outline to T. min and T max To eliminate outlines that are too large or too small; when the pixel area of ​​the tobacco plant outline meets T min <A i ≤T max At the same time, retain the tobacco plant outline information C i Otherwise, delete the tobacco plant outline information C. i .

[0101] The contour result after filtering based on contour area is as follows: Figure 4 As shown in (e).

[0102] The number of tobacco plants was obtained by statistically analyzing the number of filtered contours. Based on the filtered contour information, the coordinates of the upper left and lower right corners of the smallest bounding rectangle of the tobacco plant contours were detected, and the locations of the tobacco plants were circled with rectangles in the remote sensing image of the tobacco field, resulting in the following: Figure 4 The statistical results of the number of tobacco plants shown in (f) are as follows.

[0103] In addition, to verify the effectiveness of the method proposed in this invention, the following specific experimental procedure is also given.

[0104] 1. MED-Net tobacco field segmentation experiment.

[0105] In order to achieve the segmentation of tobacco fields at the plot scale and remove non-tobacco planting areas such as roads, forests, and non-tobacco crops in UAV remote sensing images, this invention builds the MED-Net model based on the above-mentioned experimental software and hardware environment, and allocates the training set and test set in a 9:1 ratio for model training and testing. The data allocation and model training parameters are shown in Table 1.

[0106] Table 1 Model training parameters

[0107]

[0108] As shown in Table 1, the tobacco field segmentation dataset contains 600 images, with 540 in the training set and 60 in the test set. To compare the performance metrics of MED-Net, PSPnet, Unet, DeeplabV3+_Xception, and DeeplabV3+_MobileNetV2 models were introduced for comparison. The learning rate was uniformly set to 7e-3, batch size to 4, momentum parameter to 0.9, and weight decay to 1e-4. The optimizer used was SGD, resulting in the following... Figure 5 The curves showing the loss function values ​​of the five models as a function of Epoch are presented. When the performance indicators of the loss functions of the five models no longer improve after approximately 112 training iterations, a qualitative comparative test analysis of the tobacco field segmentation performance of the five models is conducted, yielding the following results: Figure 6 The results of the tobacco field segmentation comparison are shown.

[0109] like Figure 6 As shown, Figure 6 In the diagram, (a) represents the original map, which contains five plots, labeled (i), (ii), (iii), (iv), and (v). Figure 6 (b) in the text represents the tag file for the corresponding plot image. Figure 6 In (c) to (g), the green areas represent the segmentation results of different algorithms. In plot (i), the main components include large areas of uncultivated farmland, greenhouses, and corn planting areas. Tobacco fields occupy only a small portion of the UAV remote sensing image. MED-Net can completely segment the tobacco fields, while DeeplabV3+_MobileNetV2 can segment larger tobacco field areas but fails to segment smaller areas. The other three algorithms failed to segment the tobacco fields. In plot (ii), all the compared algorithms can completely segment the tobacco fields, but MED-Net has the best segmentation results for the tobacco field ridges. In plot (iii), the tobacco fields are relatively scattered and small in area, interspersed with different crop planting areas and green backgrounds such as grasslands and forests. MED-Net can segment the tobacco planting areas more completely than the other algorithms. In plot (iv), PSPnet, DeeplabV3+_Xception, and DeeplabV3+_MobileNetV2 were able to segment the tobacco field area, but the segmentation of the field ridges between tobacco fields was poor, with some areas having incomplete boundary segmentation. Unet also exhibited incorrect segmentation. MED-Net, while ensuring complete segmentation boundaries, achieved effective segmentation of the field ridges between tobacco fields. In plot (v), there were numerous corn-grown areas with similar mulching and planting methods to the tobacco-grown areas. The two types of plots showed little difference in the UAV remote sensing imagery, and all tested algorithms could correctly segment the tobacco field area, but MED-Net achieved the best segmentation results.

[0110] 2. HSV background removal experiment.

[0111] To address the removal of small background areas such as mulch and soil between tobacco plants, based on MED-Net tobacco field segmentation, experiments were conducted to determine the optimal HSV threshold for background removal. This threshold can maximize the removal of background pixels while preserving pixel information of the tobacco plant portion, thus achieving the removal of small background areas. Some processing results are shown below. Figure 7 As shown.

[0112] Figure 7 In the text, (a) to (e) represent different plots of land. Figure 7 The first row (i) represents the tobacco field segmentation map, the second row (ii) represents the tobacco plant background removal map, and the third row (iii) represents the magnified effect of a local area of ​​the tobacco plant background removal map.

[0113] 3. Experiment on the number of tobacco plants.

[0114] like Figure 8 As shown, plot (a) has a relatively simple background with little background interference, but some areas have vigorous tobacco plant growth, with overlapping leaves that obscure the plant. The algorithm struggles to separate these overlapping areas, leading to missed detections. Plot (b) has excessive weeds in some areas, interfering with tobacco plant outline extraction. Additionally, some tobacco plants grow slowly, and even manual observation of smaller plants requires repeated verification, resulting in low prediction accuracy. Plot (c) has trees shading tobacco plants. When removing background based on the HSV color space, these shaded plants are easily mistaken for background information and removed, causing the outline extraction of shaded areas to fail, leading to missed detections in tobacco plant count. Plots (d) and (e) have relatively simple backgrounds and uniform tobacco plant growth, resulting in relatively high tobacco plant counting accuracy. Figure 2 The table shows the statistical data on the number of tobacco plants in plots (a) to (e). In Table 2, U_C (Unfiltered contours) represents the number of contours that have not been filtered, D_T (Distance threshold) represents the number of contours removed by filtering based on the pixel distance threshold between tobacco plants, and A_T (Area threshold) represents the number of contours removed by filtering based on the pixel area of ​​tobacco plants.

[0115] Table 2 Statistical results of tobacco plant quantity

[0116]

[0117] As shown in Table 2, the MED-Net framework proposed in this invention can achieve accurate segmentation of tobacco fields in large-scale UAV images with high similarity and complex backgrounds. In addition, the tobacco plant contour extraction and screening strategy proposed in this invention can effectively obtain tobacco plant contours. While reducing the large amount of data annotation and model training work brought about by deep learning-based crop counting, it can achieve accurate statistics of tobacco plant numbers and meet the needs of tobacco agriculture tobacco plant statistics.

[0118] Example 2

[0119] This embodiment 2 describes a tobacco plant segmentation and counting system based on large-scale images from UAVs. This system is based on the same inventive concept as the tobacco plant segmentation and counting method based on large-scale images from UAVs described in embodiment 1 above.

[0120] A tobacco plant segmentation and counting system based on large-scale images from unmanned aerial vehicles (UAVs) includes the following modules:

[0121] The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas.

[0122] The tobacco field background removal module is used to create a tobacco plant background mask based on the HSV color space. Based on the obtained tobacco field segmentation results, it further removes small background areas, including soil and mulch, to obtain an image that retains only the tobacco plants.

[0123] The tobacco plant count module is used to perform Gaussian filtering, morphological denoising, and opening operation denoising on the image that retains only tobacco plants. Then, the tobacco plant contours are extracted, and the extracted tobacco plant contours are filtered and counted to obtain the number of tobacco plants.

[0124] It should be noted that the implementation process of the functions and roles of each functional module in the tobacco plant segmentation and counting system in this embodiment 2 is detailed in the implementation process of the corresponding steps in the method of embodiment 1 above, and will not be repeated here.

[0125] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for tobacco plant segmentation and counting based on large-scale images from unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Step 1. Build a tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve segmentation between tobacco field areas and non-tobacco planting areas; Step 2. Based on the HSV color space, a tobacco plant background mask is created. Based on the tobacco field segmentation results obtained in Step 1, small background areas, including soil and plastic film, are further removed to obtain an image that retains only the tobacco plants. Step 3. Perform Gaussian filtering, morphological denoising and opening operation denoising on the image of tobacco plants obtained in Step 2. Then extract the tobacco plant contours. After filtering and counting the extracted tobacco plant contours, obtain the number of tobacco plants. The encoder of the tobacco field segmentation framework MED-Net includes a MobileNetV2 module, an ECA-Net module, and an ASPP module; In the encoder section, the Xception network in the DeepLabV3+ network is replaced by the MobileNetV2 module to reduce the number of feature extraction network layers; the high-level semantic features obtained by the MobileNetV2 module are sent to the ECA-Net module for adaptive attention weighting of the channel dimensions of the feature map; deep features are enhanced and extracted through dilated convolutions with different ASPP rates, and then concat stacked; after compressing the features through 1×1 convolution operations, the feature channels are adjusted, and the enhanced features are transmitted to the decoder of MED-Net; In the decoder section, the low-level semantic features acquired by the MobileNetV2 module in the encoding stage are first processed by the ECA-Net module to calculate the attention weight of each feature channel, suppressing minor feature channels and increasing the weight of important feature channels. Then, the weighted features are concatenated with the upsampled enhanced features output from the encoding stage after adjusting the feature channels through a 1×1 convolution. After adjusting the feature channels through a 3×3 convolution, bilinear interpolation upsampling is performed to obtain the output prediction result with the same size as the original input image of the input MED-Net network. The feature extraction network MobileNetV2 has been improved upon the original MobileNetV1 as follows: An inverse residual structure is added. The inverse residual structure first uses a 1×1 convolution to achieve dimensionality increase, then uses a 3×3 depthwise convolution (DW) to extract features, and finally uses a 1×1 convolution to reduce dimensionality. The whole process is "expansion-convolution-compression" and is spindle-shaped. The convolution operation uses a 3×3 DW to replace the standard 3×3 convolution. The first two ReLU activation functions in MobileNetV1 are replaced with ReLU6. A linear bottleneck structure is added, which consists of two convolutional layers. The first 1×1 convolutional layer is used for dimensionality reduction, reducing the number of input channels to 1 / 6 of the original. The second 3×3 deep convolutional layer is used for feature extraction and expansion, expanding the number of channels back to the original size. At the same time, linear activation is used in the last layer to reduce the loss of low-dimensional feature information. The ECA-Net module first processes the input features through global average pooling to obtain all features without dimensionality reduction; then, after all features, it performs feature learning by using a fast one-dimensional convolution of size k that can share weights, and generates the weight ratio of each channel; finally, it combines the original input features with the channel weights to obtain features with channel attention. The ECA-Net module only calculates feature channels. and its Information exchange between adjacent channels, and through function Generate the weight percentage for each channel and the channel attention weight. The calculation formula is as follows: ； In the formula, This represents the weight of the i-th channel. express of A set of adjacent channels, , Represents the learning parameters, Indicates adjacent channels; To further improve the feature extraction performance of ECA-Net, a kernel size of [missing value] was used. 1D convolution To achieve information exchange between channels and share the same learning parameters, the above formula can be rewritten as follows: ； in, Indicates the weight. Indicates kernel size as 1D convolution For the channel Perform convolution operations; Since the ECA module is designed to properly capture local cross-channel interactions, it is necessary to determine the local cross-channel interaction coverage, i.e. kernel size of convolution ,in With channel dimension Proportional; When given channel dimension The size of the kernel is obtained from the dimensions of the kernel. The adaptive function is calculated as follows: ； in, This represents the odd number closest to t. ; Step 2 specifically involves: Step 2.

1. First, convert the RGB format tobacco field segmentation map into the HSV color space to obtain the tobacco field HSV color space map; then, by adjusting the hue component H, saturation component S, and lightness component V, observe the effect of tobacco plant background removal and count the corresponding H, S, and V values, establish the H, S, and V threshold range, and obtain the tobacco field color mask map. Step 2.

2. Overlay the obtained tobacco field color mask image with the RGB format tobacco field segmentation image to mask the background area of ​​tobacco plants in the remote sensing image of the tobacco field using the masking effect. Define the tobacco field partitioning map as the first input array. The color mask of the tobacco field is the second input array. , The output array is the same size as the input array, where all elements have a default value of 0, representing a black mask image; therefore: ； in, Represents an image. Represents a mask image. Indicates the output image; By performing an AND operation between the tobacco field color mask and the tobacco field segmentation image, the mask image... The white area displays the content of the corresponding area in the tobacco field segmentation image, in the mask image. The black area is used to mask the corresponding area in the tobacco field segmentation map, thereby separating the tobacco plants from the background area and obtaining the tobacco field background removal map.

2. The method for tobacco plant segmentation and counting based on large-scale UAV images according to claim 1, characterized in that, In step 1, a dataset is prepared before training the tobacco field segmentation framework MED-Net; Step 1.

1. Data Acquisition: The data collection period is during the early stage of tobacco growth. Data collection is carried out by planning different flight routes according to different flight altitudes to obtain the original image data of the tobacco field. At the same time, in order to ensure that the data acquired by the UAV can be stitched together, there is some overlap in different original image data. The collected original image data of the tobacco field is further stitched to remove the overlapping areas, so as to avoid the same area being counted repeatedly when counting tobacco plants in the experimental data. The spliced ​​data was cropped to obtain the tobacco plant counting experimental data; Step 1.

2. Select data with differences from the raw tobacco field image data collected by the UAV to construct training data for the tobacco field segmentation model; after compressing the selected training data, use the image annotation tool Labelme to annotate the data, and use Mosaic and Augmentor tools to enhance the annotated data; Mosaic data augmentation involves randomly selecting four original images, adjusting their size and position, and then stitching them together to obtain new data. The Augmentor tool uses various methods such as rotation, scaling, cropping, and deformation to achieve data augmentation. Step 1.

3. After data augmentation, divide the training set and test set according to the ratio for MED-Net training.

3. The method for tobacco plant segmentation and counting based on large-scale UAV images according to claim 2, characterized in that, The dataset obtained in step 1 is a large-scale image dataset of UAVs. This dataset is used to train the tobacco field segmentation framework MED-Net, and the trained MED-Net is used to segment tobacco field images.

4. The method for tobacco plant segmentation and counting based on large-scale UAV images according to claim 1, characterized in that, In step 3, contour filtering is divided into two operations: the first is the contour filtering process based on the Euclidean distance between the centroids of the contours, and the second is the contour filtering process based on the pixel area inside the contour.

5. The method for tobacco plant segmentation and counting based on large-scale UAV images according to claim 4, characterized in that, In step 3, the contour selection process based on the Euclidean distance between the centroids of the contours is as follows: The centroid of a contour is the geometric center of a closed contour. The coordinates of the centroid are determined by calculating the rectangular moments of the profile, as shown in the following formula: ； in, and Let x and y represent the x and y coordinates of the centroid, respectively. The zeroth moment of the profile, and These represent the first moments of the profile; Let there be two contours , The Euclidean distance between the centers of mass is Distance threshold is The pixel areas of the outlines are respectively , When the Euclidean distance between the centroids of two contours Greater than or equal to the distance threshold When the distance is less than the threshold, retain the contour information of both contours; when the distance is less than the threshold, retain the contour information of both contours. When comparing two contours , The pixel area size is determined, and the outline information of the larger pixel area is retained to ensure that only the outline information of one tobacco plant is retained within the range of less than the distance threshold. The specific screening strategy is shown in the following formula: ； Set the pixel area threshold for the tobacco plant outline to and To eliminate outlines that are too large or too small; when the pixel area of ​​the tobacco plant outline meets the requirements... At the same time, retain the outline information of the tobacco plants. Otherwise, delete the tobacco plant outline information. .

6. A tobacco plant segmentation and counting system based on large-scale UAV images for implementing the tobacco plant segmentation and counting method based on large-scale UAV images as described in claim 1, characterized in that, The tobacco plant segmentation and counting system based on large-scale UAV images includes the following modules: The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas. The tobacco field background removal module is used to create a tobacco plant background mask based on the HSV color space. Based on the obtained tobacco field segmentation results, it further removes small background areas, including soil and mulch, to obtain an image that retains only the tobacco plants. The tobacco plant count module is used to perform Gaussian filtering, morphological denoising, and opening operation denoising on the image that retains only tobacco plants. Then, the tobacco plant contours are extracted, and the extracted tobacco plant contours are filtered and counted to obtain the number of tobacco plants.

Images