Tobacco plant segmentation and counting method and system based on unmanned aerial vehicle large-scale image

Abstract

The invention belongs to the technical field of tobacco plant number statistics, and discloses a tobacco plant segmentation and counting method and system based on unmanned aerial vehicle large-scale images. According to the method, on the basis of an unmanned aerial vehicle large-scale tobacco field segmentation image data set, a feature extraction complex MobileNetV2 and a channel attention mechanism ECA are integrated on a DeeplabV3 + network, and a tobacco field segmentation framework MED-Net is constructed to realize tobacco field segmentation; a tobacco plant background removing method is further utilized to remove background information; in addition, the invention also establishes a tobacco plant contour screening method based on a pixel level, which is beneficial to screening of small target contours such as weeds and the like, and improves the extraction accuracy of the tobacco plant contour. According to the method, tobacco field segmentation and tobacco plant counting in the large-scale unmanned aerial vehicle remote sensing image can be effectively achieved, the over-segmentation problem caused by the fact that the unmanned aerial vehicle remote sensing image needs to be cut in a large amount due to insufficient equipment performance is solved, and statistics of the tobacco plant number is achieved on the premise that a large amount of data labeling is not needed; and the tobacco plant statistics efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of tobacco plant quantity statistics, and in particular relates to a tobacco plant segmentation and counting method and system based on large-scale unmanned aerial vehicle images. Background Art

[0002] Tobacco plant counting refers to the accurate counting of the number of tobacco seedlings after transplanting, which is the key to ensuring the quality and yield of tobacco. Traditional tobacco plant counting mainly relies on manual counting, which has problems such as large workload, low efficiency, high cost, and difficulty in ensuring statistical accuracy, making the acquisition of tobacco planting monitoring information inaccurate and incomplete.

[0003] In the tobacco plant number statistics based on drone images, the drone flight altitude not only affects the image reconstruction details and accuracy, but also affects the processing efficiency and tobacco plant object detection accuracy. The higher the flight altitude, the more targets are obtained in a single image, the higher the data collection efficiency and algorithm processing efficiency, but the less detailed information will be obtained, which tests the algorithm's ability to obtain target features. The lower the flight altitude, the richer the detailed information obtained, and the lower the difficulty of algorithm processing, but it will reduce the data collection efficiency, increase the number of images collected in the same area, and reduce the processing efficiency.

[0004] At present, some crop quantity statistics based on multispectral and hyperspectral data have some limitations in practical applications due to high collection costs, complex data processing, and difficult data integration. In addition, some tobacco plant detection based on deep learning requires a lot of data annotation and detection model training, and fails to achieve tobacco field segmentation and tobacco plant area extraction.

[0005] At present, in order to detect crops in drone images, some algorithms based on target detection can only lower the flight altitude of the drone to obtain detailed image information, or can only crop the large-scale remote sensing images obtained by the drone into small-scale remote sensing images under limited computing resources before performing tobacco plant detection. This target detection method can easily result in incomplete plant contours, and there are problems such as over-segmentation, repeated statistics, and low statistical efficiency. Summary of the invention

[0006] The purpose of the present invention is to propose a tobacco plant segmentation and counting method based on large-scale UAV images. The method combines UAV remote sensing imaging technology with deep learning and image processing methods, and can effectively segment the tobacco fields in large-scale UAV images, and further remove the tobacco plant background in the segmented tobacco fields. At the same time, it is conducive to the extraction and screening of tobacco plant contours, thereby realizing effective statistics of the number of tobacco plants.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] A tobacco plant segmentation and counting method based on large-scale UAV images includes the following steps:

[0009] Step 1. Build a tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas;

[0010] Step 2. Create a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation result obtained in step 1, further remove small-sized background areas including soil and mulch to obtain an image that only retains tobacco plants.

[0011] Step 3. Perform Gaussian filtering, morphological denoising and opening operation denoising on the image obtained in step 2 that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

[0012] In addition, based on the above-mentioned tobacco plant segmentation and counting method based on large-scale drone images, the present invention also proposes a corresponding tobacco plant segmentation and counting system based on large-scale drone images, which adopts the following technical solutions:

[0013] A tobacco plant segmentation and counting system based on large-scale UAV images, including the following modules:

[0014] The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas;

[0015] The tobacco field background removal module is used to establish a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation results, small-sized background areas including soil and mulch are further removed to obtain an image that only retains tobacco plants.

[0016] The tobacco plant number counting module is used to perform Gaussian filtering, morphological denoising and opening operation denoising on the image that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

[0017] The present invention has the following advantages:

[0018] As described above, the present invention relates to a tobacco plant segmentation and counting method and system based on large-scale drone images. The method of the present invention establishes a tobacco field segmentation image dataset based on drones, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA on the DeeplabV3+ network to construct a tobacco field segmentation framework MED-Net, which is suitable for segmenting tobacco fields in large-format drone images; in addition, the present invention also proposes a method for tobacco plant background removal, which can remove background information such as soil and mulch, avoid incomplete outlines of some tobacco plants caused by over-segmentation, and improve the accuracy and efficiency of tobacco plant statistics. In addition, the present invention also establishes a tobacco plant contour screening method based on pixel level, which can screen the outlines of small targets such as weeds, improve the accuracy of tobacco plant contour extraction, and realize the statistics of the number of tobacco plants without the need for a large amount of data annotation. The present invention can effectively realize the segmentation of tobacco fields and the counting of tobacco plants in large-scale drone remote sensing images, reduce the over-segmentation problem caused by the need to perform a large amount of cropping on drone remote sensing images due to insufficient equipment performance, and improve the efficiency and accuracy of tobacco plant statistics. Compared with the tobacco plant number statistics based on the target detection algorithm, the method of the present invention does not require a large amount of tobacco plant data labeling and tobacco plant detection model training, and is more likely to achieve tobacco plant number statistics over a large area. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flow chart of a method for segmenting and counting tobacco plants based on large-scale drone images in an embodiment of the present invention.

[0020] Figure 2 This is a network structure diagram of the tobacco field segmentation framework MED-Net built in an embodiment of the present invention.

[0021] Figure 3 This is a flowchart of tobacco plant background removal in an embodiment of the present invention.

[0022] Figure 4 This is a flow chart of tobacco plant profile screening and counting in an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the curve of the loss function value of the model of the present invention and the comparison model changing with Epoch.

[0024] Figure 6 Schematic diagram of segmentation of different models. Figure 6 (a) represents the original image, (b) represents the label file of the corresponding image, and the green areas in (c) to (g) represent the segmentation results of different models such as PSPnet, Unet, DeeplabV3+_Xception, DeeplabV3+_MobileNetV2, and MED-Net.

[0025] Figure 7 The following is a schematic diagram of the background removal effect of tobacco plants in different plots. Figure 7 (a) to (e) in the figure represent different plots, respectively. The (i)th row represents the tobacco field segmentation map, the (ii)th row represents the tobacco plant background removal map, and the (iii)th row represents the local area magnification effect map of the tobacco plant background removal map.

[0026] Figure 8 This is a schematic diagram of the statistical results of the number of tobacco plants in different plots. Figure 8 (a) to (e) in the figure respectively represent the statistical results of the number of tobacco plants in different plots. The first row (i) shows the statistical results of the number of tobacco plants in different plots, the first row (ii) shows the statistical results of the number of tobacco plants as a whole, and the first row (iii) shows the enlarged effect diagram of the local area of ​​the statistical results of tobacco plants. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0028] Example 1

[0029] This embodiment 1 describes a tobacco plant segmentation and counting method based on large-scale drone images. The method is based on the acquired large-scale drone images and builds a tobacco field segmentation framework MED-Net to achieve the segmentation of tobacco field areas and non-tobacco planting areas; on this basis, a tobacco plant background mask is further established based on the HSV color space to remove small-size background areas including soil and ground film to obtain an image that only retains the tobacco plants; finally, the image is subjected to Gaussian filtering, morphological denoising and opening operation denoising, and then the tobacco plant contours are extracted. The extracted tobacco plant contours are screened and counted to obtain the number of tobacco plants.

[0030] The large-scale images here specifically refer to images with a large size. Generally speaking, the flying altitude of a drone is about 100 meters, and traditional target detection algorithms have difficulty in detecting such a large number of targets.

[0031] like Figure 1 As shown, the tobacco plant segmentation and counting method based on large-scale drone images in this embodiment includes the following steps:

[0032] Step 1. Build a tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas.

[0033] The segmentation of tobacco fields in drone images is a binary classification problem. The present invention uses the DeepLabV3+ network to achieve the segmentation of tobacco fields, but the recognition and segmentation effect of some complex ridges and boundaries, similar corn fields, etc. is poor and cannot meet the needs of tobacco field segmentation. It is necessary to improve the problems existing in the segmentation model in the application of tobacco field segmentation.

[0034] Based on this, the present invention further integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA on the basis of the structure of the DeepLabV3+ network framework, and constructs the MED-Net framework for tobacco field segmentation. Figure 2 shown.

[0035] The encoder of the tobacco field segmentation framework MED-Net includes the MobileNetV2 module, the ECA-Net module and the ASPP module.

[0036] In the encoder part, the MobileNetV2 module is first used to replace the Xception network in the DeepLabV3+ network to reduce the number of feature extraction network layers, reduce the number of parameters, reduce model complexity, and improve the model running speed; secondly, the high-level semantic features obtained by the MobileNetV2 module are transmitted to the ECA-Net module for adaptive attention weighting of the channel dimension of the feature map, helping the model to better understand and utilize the information of different channels and improve the model's attention to different channels; then, the deep features are enhanced and extracted through the hollow convolution of different rates of the ASPP module, and then Concat stacking is performed; finally, the features are compressed through a 1×1 convolution operation and the feature channels are adjusted, and the enhanced features are transmitted to the decoder of MED-Net.

[0037] In the decoder part, the low-level semantic features obtained by the MobileNetV2 module in the encoding stage are first used to calculate the attention weight of each feature channel through the ECA-Net module, suppressing the secondary feature channels and increasing the weights of the important feature channels; secondly, the weighted features are adjusted through 1×1 convolution and then concatenated with the enhanced features after upsampling output in the encoding stage, and then the feature channels are adjusted through 3×3 convolution and bilinear interpolation upsampling is performed to obtain the output prediction result of the same size as the original input image of the input MED-Net.

[0038] The feature extraction network MobileNetV2 has a 20% reduction in model parameters compared to MobileNetV1, but has better accuracy. Compared with MobileNetV1, the main improvements are as follows:

[0039] 1. Added the inverted residual structure Inverted Residuals.

[0040] The inverted residual structure first uses 1×1 convolution to achieve dimensionality increase, then extracts features through 3×3 depthwise convolution (DW), and finally uses 1×1 convolution to reduce dimensionality. The whole process is "expansion-convolution-compression" and is shuttle-shaped; the convolution operation uses 3×3 DW to replace the 3×3 standard convolution; and the first two ReLU activation functions of MobileNetV1 are replaced by ReLU6. The ReLU6 function is shown in the following formula:

[0041] y=ReLU6(x)=min(max(x,0),6).

[0042] The inverted residual structure in MobileNetV2 uses a method of first increasing the dimension and then reducing the dimension to deepen the network layers, which enhances the gradient propagation and feature expression capabilities, and helps the network better capture the details and features in the image.

[0043] At the same time, compared with ReLU, ReLU6 can provide better nonlinear expression capabilities, and can directly truncate the input value greater than 6 to 6, reducing the computational complexity of the network and improving performance. By replacing the first two ReLUs with ReLU6, MobileNetV2 can further improve computational efficiency while maintaining a certain accuracy.

[0044] 2. Add linear bottleneck structure Linear Bottlenecks.

[0045] The linear bottleneck structure refers to mapping high-dimensional space to low-dimensional space, and using linear activation in the last convolution layer to reduce the loss of low-dimensional feature information. The linear bottleneck structure consists of two convolution layers. The first 1×1 convolution layer is used for dimensionality reduction, reducing the number of input channels to 1 / 6 of the original. The second 3×3 deep convolution layer is used for feature extraction and expansion, expanding the number of channels back to the original size. At the same time, linear activation is used in the last layer to reduce the loss of low-dimensional feature information, reducing the amount of calculation and the number of model parameters, and improving the computational efficiency of the network while maintaining a certain accuracy.

[0046] The ECA-Net module weights the features of different channels by learning a weight vector, enhancing the model's ability to represent features of different channels, and has great potential in improving model performance. The ECA-Net module first processes the input features through global average pooling (GAP) to obtain all features without dimensionality reduction; secondly, a fast one-dimensional convolution with a size of k that can share weights is connected to all features for feature learning, and the weight ratio of each channel is generated; finally, the original input features are combined with the channel weights to obtain features with channel attention.

[0047] On the basis of avoiding dimensionality reduction, ECA-Net proposes a local cross-channel interaction strategy and a method of adaptively selecting the size of one-dimensional convolution kernel, which effectively captures the cross-channel interaction information while significantly reducing the complexity of the model.

[0048] In the ECA-Net module, only the feature channel y is calculated i The information interaction between its k adjacent channels is realized through the Sigmoid function σ Generate the weight ratio of each channel, channel attention weight w i The calculation formula is as follows:

[0049]

[0050] In the formula, w i represents the weight of the i-th channel, Represents y i The set of k adjacent channels, j = 1, 2, ..., k, ω j represents the learning parameters, Indicates adjacent channels.

[0051] In order to further improve the feature extraction performance of ECA-Net, a 1D convolution C1D with a kernel size of k is used to realize information interaction between channels and share the same learning parameters. The above formula is rewritten as follows:

[0052] w=σ(C 1D k (y)).

[0053] Among them, w represents the weight, C1D k (y) represents a 1D convolution C1D with a kernel size of k performing a convolution operation on channel y.

[0054] Since the ECA module aims to properly capture local cross-channel interactions, the local cross-channel interaction coverage, i.e., the kernel size k of the 1D convolution, needs to be determined, where k is proportional to the channel dimension C.

[0055] When the size of the channel dimension C is given, the calculation adaptive function of the kernel size k is:

[0056]

[0057] Where |t| odd Represents the odd number closest to t, r=2, b=1.

[0058] Before training the tobacco field segmentation framework MED-Net, prepare the data set as follows:

[0059] Step 1.1. Obtaining data. The collection time stage is the early stage of tobacco growth. According to different flight altitudes, different routes are planned for data collection to obtain the original image data of the tobacco field. At the same time, in order to ensure that the data obtained by the drone can be spliced ​​during the data collection process (for example, using DJI Zhitu to splice the flight data), different original image data have some overlapping areas. In order to avoid repeated counting of the same area when counting tobacco plants for experimental data, it is necessary to splice the collected original image data to remove the overlapping areas; the spliced ​​data is cropped to obtain the tobacco plant counting experimental data. At the same time, in order to reduce the amount of data processing, the Global_Mapper software is used to preprocess the data to obtain the drone tobacco field data, such as Figure 3 As shown in (a).

[0060] Step 1.2. Select the differentiated data from the original image data of the tobacco field collected by the drone to construct the training data of the tobacco field segmentation model; after compressing the selected training data, use the image annotation tool Labelme to annotate the data, and enhance the annotated data through Mosaic and Augmentor tools.

[0061] The Mosaic data enhancement tool obtains new data by randomly selecting four original images, adjusting their sizes and positions, and then splicing them together. The Augmentor tool achieves data enhancement through rotation, scaling, cropping, and deformation.

[0062] Step 1.3. After completing data enhancement, divide the training set and test set into proportions for MED-Net training.

[0063] The dataset obtained here is a large-scale UAV image dataset, which is used to train the constructed tobacco field segmentation framework MED-Net, and the trained MED-Net is used to perform tobacco field image segmentation.

[0064] After tobacco field image segmentation based on MED-Net, most non-tobacco field areas such as corn fields, grasslands, woods, roads, etc. can be removed to obtain the following Figure 3The tobacco field segmentation diagram shown in (b) to (c).

[0065] Step 2. Create a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation result obtained in step 1, further remove small-sized background areas including soil and ground film to obtain an image that only retains tobacco plants.

[0066] After completing the tobacco field image segmentation based on MED-Net, most non-tobacco field areas such as corn fields, grasslands, woods, roads, etc. can be removed. However, the existence of small-sized background areas such as mulch, soil, and weeds between tobacco plants will interfere with the subsequent statistics of the number of tobacco plants, and the MED-Net image segmentation model is difficult to segment these areas from tobacco plants. Therefore, the present invention constructs a method for removing small-sized background areas to achieve fine segmentation of tobacco fields and background areas.

[0067] Step 2.1. First, convert the tobacco field segmentation map in RGB format into HSV color space to obtain Figure 3 The HSV color space diagram of the tobacco field shown in (d) is shown in the figure. Secondly, by adjusting the hue component (Hue, H), saturation component (Saturation, S), and brightness component (Value, V), the background removal effect of the tobacco plants is observed and the corresponding H, S, and V values ​​are counted to establish the H, S, and V threshold ranges, as shown in the figure below. Figure 3 Color mask image of the tobacco field shown in (e).

[0068] Step 2.2. Overlay the obtained tobacco field color mask image with the tobacco field segmentation image in RGB format, and use the mask shielding effect to block the background area of ​​the tobacco plants in the tobacco field remote sensing image; when the two images have the same size, define the tobacco field segmentation image as the first input array src1, the tobacco field color mask image as the second input array src2, and dst as the output array of the same size as the input array, in which the default value of all elements is 0, that is, a black mask image; then:

[0069]

[0070] Among them, I represents image, mask(I) represents mask image, and dst(I) represents output image.

[0071] The tobacco field color mask image is used to perform an “AND” operation with the tobacco field segmentation image. The content of the corresponding area in the tobacco field segmentation image is displayed in the white area of ​​the mask image mask(I)≠0, and the corresponding area in the tobacco field segmentation image is blocked in the black area of ​​the mask image mask(I)=0, so as to separate the tobacco plants from the background area. Figure 3 The background of the tobacco field is removed as shown in (f).

[0072] Step 3. Perform Gaussian filtering, morphological denoising and opening operation denoising on the image obtained in step 2 that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

[0073] Step 3.1. Image grayscale: Grayscale processing compresses the original image data volume, improves data processing speed, reduces interference and avoids stripe distortion. The grayscale value Y of each component is obtained by the following formula.

[0074] Y = 0.229R + 0.587G + 0.114B;

[0075] In the above formula, R, G, and B represent the three channel components corresponding to the RGB color space.

[0076] Step 3.2. Gaussian filtering to smooth the image: Smoothing is achieved by reducing the variation between pixels in the image. It is used to remove noise in the image before edge detection to improve image quality and accuracy.

[0077] Define the input image as S(u,v), and get G(u,v) after processing it with a two-dimensional Gaussian function. Use the weighted average gray value of the pixels in the neighborhood to replace the value of the central pixel of the template. The two-dimensional Gaussian function processing is as follows:

[0078]

[0079] Step 3.3. Image binarization operation: As shown in the following formula, the grayscale threshold T is set by threshold binarization operation. 1 , when the gray value is less than or equal to the threshold T 1 When the gray value is set to 0, it is greater than the threshold T 1 When , set its grayscale value to 255.

[0080]

[0081] By graying, Gaussian filtering, and binarization, the dimension of the image data is reduced, the variation between pixels in the image is reduced, the tobacco plant is separated from the background, and the outline of the tobacco plant is highlighted. The processing results are as follows: Figure 4 As shown in (a).

[0082] Step 3.4. Image morphology and opening operation denoising operation:

[0083] First, define a 3×3 cross-shaped structure element M as shown in the following formula, perform morphological operations on the structure element M and the image, and return a structure element of a specified shape and size.

[0084]

[0085] Then, the opening operation is used to perform corrosion and expansion operations to separate the cigarette and part of the background area, remove the external white pixel interference area around the cigarette, and keep the other parts unchanged, and get the following Figure 4 The results are shown in (b).

[0086] Step 3.5. Contour extraction:

[0087] By compressing the elements in the horizontal, vertical, and diagonal directions, only the end point coordinates in the direction are retained, and the outer contour information of the tobacco plant is output. The contour of the tobacco plant is drawn through the contour information, such as Figure 4 As shown in (c).

[0088] Step 3.6. Contour screening:

[0089] Contour filtering is divided into two operations, the first is the contour filtering process through the Euclidean distance between contour centroids, and the second is the contour filtering process based on the pixel area inside the contour.

[0090] The centroid of a contour refers to the geometric center of a closed contour.

[0091] Specifically, the contour screening process by the Euclidean distance between contour centroids is as follows:

[0092] The coordinates of the centroid are determined by calculating the rectangular moment of the contour, as shown in the following formula:

[0093]

[0094] Among them, x i and i Respectively represent the horizontal and vertical coordinates of the center of mass, M 00 Represents the 0th-order moment of the contour, M 01 and M 10 They represent the first-order moments of the contour respectively.

[0095] Suppose there are two contours C i , C j The Euclidean distance between the centroids is d i,j , the distance threshold is T d , the pixel areas of the contours are A i ,A j ; When the Euclidean distance d between the two contour centroids i,j Greater than or equal to the distance threshold T d When the distance is less than the threshold T, the contour information of the two contours is retained; when the distance is less than the threshold T d When comparing two contours C i , C jThe pixel area size is determined by retaining the contour information with a larger pixel area, ensuring that only one tobacco plant contour information is retained within a range less than the distance threshold.

[0096] The specific screening strategy is shown in the following formula:

[0097]

[0098] The contour results after filtering by contour centroid distance are as follows: Figure 4 As shown in (d).

[0099] The contour filtering process based on the pixel area inside the contour is as follows:

[0100] Set the pixel area threshold of the tobacco plant outline to T min and T max , to eliminate contours that are too large or too small; when the pixel area of ​​the tobacco plant contour meets T min <A i ≤T max When retaining the tobacco plant contour information C i , otherwise delete the tobacco plant outline information C i .

[0101] The contour results after filtering based on contour area are as follows Figure 4 As shown in (e).

[0102] The number of tobacco plants is obtained by counting the number of contours after screening, and the coordinate information of the upper left corner and lower right corner of the minimum enclosing rectangle of the tobacco plant contour is detected according to the contour information after screening, and the tobacco plant position in the remote sensing image of the tobacco field is circled with a rectangular frame to obtain the following Figure 4 (f) shows the statistical results of the number of tobacco plants.

[0103] In addition, in order to verify the effectiveness of the method proposed in the present invention, the following specific experimental process is also given.

[0104] 1. MED-Net tobacco field segmentation experiment.

[0105] In order to segment tobacco fields at the plot scale and remove non-tobacco planting areas such as roads, woods, and non-tobacco crops in drone remote sensing images, the present invention builds a MED-Net model based on the above experimental software and hardware environment, and allocates training sets and test sets in a 9:1 ratio for model training and testing. The data allocation and model training parameters are shown in Table 1.

[0106] Table 1 Model training parameters

[0107]

[0108] As shown in Table 1, the tobacco field segmentation dataset has a total of 600 images, of which 540 are training sets and 60 are test sets. In order to compare and analyze the performance indicators of MED-Net, PSPnet, Unet, DeeplabV3+_Xception, and DeeplabV3+_MobileNetV2 models are introduced for comparison. The learning rate is uniformly set to 7e-3, the batch size is 4, the momentum parameter is 0.9, and the weight decay is 1e-4. The optimizer uses SGD, and the following is obtained: Figure 5 The loss function values ​​of the five models change with Epoch as shown in the figure. When the loss function performance indicators of the five models no longer improve after about 112 training iterations, a qualitative comparative test analysis of the tobacco field segmentation performance of the five models is performed, and the following results are obtained: Figure 6 The tobacco field segmentation comparison results are shown.

[0109] like Figure 6 As shown, Figure 6 (a) in the figure represents the original image, which has five plots in total, namely plots (i), (ii), (iii), (iv), and (v). Figure 6 (b) in the figure represents the label file of the corresponding plot image. Figure 6 The green areas in (c) to (g) represent the segmentation results of different algorithms. In plot (i), it mainly includes large areas of unplanted arable land, greenhouses and corn planting areas. The tobacco field only occupies a small part of the UAV remote sensing image. MED-Net can completely segment the tobacco field, DeeplabV3+_MobileNetV2 can segment larger tobacco field areas, but smaller areas cannot be segmented. The other three algorithms cannot achieve the segmentation of the tobacco field. In plot (ii), all the comparison algorithms can achieve complete segmentation of the tobacco field, but MED-Net has the best segmentation result in the segmentation of the tobacco field ridge. In plot (iii), the tobacco field area is relatively scattered and small in area, interspersed with different crop planting areas and green backgrounds such as grassland and woods. Compared with other algorithms, MED-Net can segment the tobacco field planting area more completely. In plot (iv), PSPnet, DeeplabV3+_Xception, and DeeplabV3+_MobileNetV2 can segment the tobacco field area, but the segmentation effect of the ridges between the tobacco fields is poor, the boundary segmentation of some areas is incomplete, and Unet has incorrect segmentation. MED-Net can effectively segment the ridges between the tobacco fields while ensuring the integrity of the segmentation boundaries. In plot (v), there are a large number of corn planting areas, which have similar mulch and planting methods as the tobacco field planting areas. The difference between the two types of plots in the drone remote sensing images is small. All test algorithms can correctly segment the tobacco field area, but MED-Net has the best segmentation result.

[0110] 2.HSV background removal experiment.

[0111] In order to remove small-sized background areas such as mulch and soil between tobacco plants, based on the tobacco field segmentation based on MED-Net, the optimal HSV threshold for removing tobacco plant background was obtained through experiments. This threshold can remove background pixels to the maximum extent while retaining the pixel information of the tobacco plant part, thus achieving the removal of small-sized background. Some processing results are shown in the figure. Figure 7 shown.

[0112] Figure 7 (a) to (e) represent different plots of land. Figure 7 The (i)th row in FIG. 4 represents the tobacco field segmentation map, the (ii)th row represents the tobacco plant background removal map, and the (iii)th row represents the local area magnification effect map of the tobacco plant background removal map.

[0113] 3. Experiment on the statistical analysis of the number of tobacco plants.

[0114] like Figure 8 As shown, the background of plot (a) is relatively simple, with little background interference information, but there are areas where tobacco plants grow vigorously, and the leaves of the tobacco plants overlap and occlude each other. It is difficult for the algorithm to separate the areas with too much overlap, resulting in missed detections. There are too many weeds in some areas of plot (b), which interfere with the extraction of tobacco plant contours. At the same time, some tobacco plants grow slowly, and even if the smaller tobacco plants are calculated by manual observation of the image, they need to be confirmed repeatedly, resulting in low prediction accuracy. In plot (c), there are cases where the tobacco plants are obscured by tree shadows. When removing the background based on the HSV color space, the tobacco plants obscured by the shadows are easily removed as background information, resulting in failure to extract the tobacco plant contours in the obscured area, which in turn leads to missed detections in the tobacco plant count. The backgrounds of plots (d) and (e) are relatively simple, the tobacco plants grow more uniformly, and the tobacco plant counting accuracy is relatively high. As shown Figure 2 The statistics of the number of tobacco plants in plots (a) to (e) are shown. In Table 2, U_C (Unfiltered contours) represents the number of contours that have not been filtered, D_T (Distance threshold) represents the number of contours that have been removed by the distance threshold between tobacco plants, and A_T (Area threshold) represents the number of contours that have been removed by the area threshold of tobacco plants.

[0115] Table 2 Statistical results of tobacco plant numbers

[0116]

[0117] It can be seen from Table 2 that the MED-Net framework proposed in the present invention can realize the accurate segmentation of tobacco fields in large-scale drone images with high similarity and complex backgrounds. In addition, the tobacco plant contour extraction and screening strategy proposed in the present invention can effectively obtain the tobacco plant contour, while reducing the large amount of data labeling and model training work brought by crop counting based on deep learning, and realize the accurate statistics of the number of tobacco plants, meeting the needs of tobacco agriculture tobacco plant statistics.

[0118] Example 2

[0119] This embodiment 2 describes a tobacco plant segmentation and counting system based on large-scale drone images. This system is based on the same inventive concept as the tobacco plant segmentation and counting method based on large-scale drone images described in the above embodiment 1.

[0120] A tobacco plant segmentation and counting system based on large-scale UAV images, including the following modules:

[0121] The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas;

[0122] The tobacco field background removal module is used to establish a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation results, small-sized background areas including soil and mulch are further removed to obtain an image that only retains tobacco plants.

[0123] The tobacco plant number counting module is used to perform Gaussian filtering, morphological denoising and opening operation denoising on the image that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

[0124] It should be noted that, in the tobacco plant segmentation and counting system in this embodiment 2, the implementation process of the functions and effects of each functional module is specifically described in the implementation process of the corresponding steps in the method in the above embodiment 1, and will not be repeated here.

[0125] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A tobacco plant segmentation and counting method based on large-scale drone images, characterized in that: The steps include: Step 1. Build a tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrate the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas; Step 2. Create a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation result obtained in step 1, further remove small-sized background areas including soil and mulch to obtain an image that only retains tobacco plants. Step 3. Perform Gaussian filtering, morphological denoising and opening operation denoising on the image obtained in step 2 that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

2. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 1 is characterized in that: The encoder of the tobacco field segmentation framework MED-Net includes a MobileNetV2 module, an ECA-Net module and an ASPP module; In the encoder part, the MobileNetV2 module replaces the Xception network in the DeepLabV3+ network to reduce the number of feature extraction network layers; the high-level semantic features obtained by the MobileNetV2 module are transmitted to the ECA-Net module for adaptive attention weighting of the channel dimension of the feature map; the deep features are enhanced and extracted through the hole convolution of different ASPP rates, and then Concat stacking is performed; the features are compressed through 1×1 convolution operation and the feature channels are adjusted, and the enhanced features are transmitted to the decoder of MED-Net; In the decoder part, the low-level semantic features obtained by the MobileNetV2 module in the encoding stage are first used to calculate the attention weight of each feature channel through the ECA-Net module, suppressing the secondary feature channels and increasing the weights of the important feature channels; secondly, the weighted features are adjusted through 1×1 convolution and then concatenated with the enhanced features after upsampling output in the encoding stage, and then the feature channels are adjusted through 3×3 convolution and bilinear interpolation upsampling is performed to obtain the output prediction result of the same size as the original input image of the input MED-Net network.

3. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 2 is characterized in that: The feature extraction network MobileNetV2 has made the following improvements based on the original MobileNetV1: Add an inverted residual structure. The inverted residual structure first uses 1×1 convolution to achieve dimensionality increase, then extracts features through 3×3 deep convolution DW, and finally uses 1×1 convolution to reduce dimensionality. The whole process is "expansion-convolution-compression" and is shuttle-shaped; the convolution operation uses 3×3 DW to replace 3×3 standard convolution; the first two ReLU activation functions in MobileNetV1 are replaced with ReLU6; Add a linear bottleneck structure, which consists of two convolutional layers. The first 1×1 convolutional layer is used for dimensionality reduction, reducing the number of input channels to 1 / 6 of the original. The second 3×3 deep convolutional layer is used for feature extraction and expansion, expanding the number of channels back to the original size. At the same time, linear activation is used in the last layer to reduce the loss of low-dimensional feature information.

4. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 2 is characterized in that: The ECA-Net module first processes the input features through global average pooling to obtain all features without dimensionality reduction; secondly, a fast one-dimensional convolution with size k that can share weights is connected after all features for feature learning, and the weight proportion of each channel is generated; finally, the original input features are combined with the channel weights to obtain features with channel attention; In the ECA-Net module, only the feature channel y is calculated i The information interaction between its k adjacent channels is realized through the Sigmoid function σ Generate the weight ratio of each channel, channel attention weight w i The calculation formula is as follows: In the formula, w i represents the weight of the i-th channel, Represents y i The set of k adjacent channels, j = 1, 2, ..., k, ω j represents the learning parameter, y i j Indicates adjacent channels; In order to further improve the feature extraction performance of ECA-Net, a 1D convolution C1D with a kernel size of k is used to realize information interaction between channels and share the same learning parameters. The above formula is rewritten as follows: in=σ(C1D k (y)); Among them, w represents the weight, C1D k (y) represents the convolution operation of channel y by a 1D convolution C1D with a kernel size of k; Since the ECA module aims to properly capture local cross-channel interactions, it is necessary to determine the local cross-channel interaction coverage, i.e., the kernel size k of the 1D convolution, where k is proportional to the channel dimension C; When the size of the channel dimension C is given, the calculation adaptive function of the kernel size k is: Where |t| odd Represents the odd number closest to t, r=2, b=1.

5. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 1 is characterized in that: In the step 1, before training the tobacco field segmentation framework MED-Net, a data set is prepared; Step 1.

1. Acquisition of data The data collection period is the early stage of tobacco growth. Different routes are planned according to different flight altitudes to collect data and obtain the original image data of the tobacco field. At the same time, in order to ensure that the data obtained by the drone can be spliced ​​during the data collection process, different original image data have some overlapping areas. The collected original image data of the tobacco field are further spliced ​​to remove the overlapping areas to avoid repeated counting of the same area when counting tobacco plants for experimental data. The spliced ​​data are trimmed to obtain the experimental data of tobacco plant counting; Step 1.

2. Select the differentiated data from the original image data of the tobacco field collected by the drone to construct the training data of the tobacco field segmentation model; after compressing the selected training data, use the image annotation tool Labelme to annotate the data, and enhance the annotated data through Mosaic and Augmentor tools; Mosaic data enhancement randomly selects four original images, adjusts their sizes and positions, and then splices them to obtain new data. The Augmentor tool achieves data enhancement through rotation, scaling, cropping, and deformation. Step 1.

3. After completing data enhancement, divide the training set and test set into proportions for MED-Net training.

6. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 5 is characterized in that: The data set obtained in step 1 is a large-scale UAV image data set, which is used to train the constructed tobacco field segmentation framework MED-Net, and the trained MED-Net is used to perform tobacco field image segmentation.

7. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 1 is characterized in that: The step 2 is specifically as follows: Step 2.

1. First, convert the tobacco field segmentation map in RGB format into HSV color space to obtain the tobacco field HSV color space map; secondly, by adjusting the three components of hue component H, saturation component S, and brightness component V, observe the tobacco plant background removal effect and count the corresponding H, S, and V values, establish the H, S, and V threshold range, and obtain the tobacco field color mask map; Step 2.

2. Overlay the obtained tobacco field color mask image with the tobacco field segmentation image in RGB format, and use the mask shielding effect to block the background area of ​​the tobacco plants in the tobacco field remote sensing image; Define the tobacco field segmentation map as the first input array src1, the tobacco field color mask map as the second input array src2, and dst as the output array of the same size as the input array, where the default value of all elements is 0, that is, a black mask image; then: Where I represents the image, mask(I) represents the mask image, and dst(I) represents the output image; The tobacco field color mask image and the tobacco field segmentation map are used to perform an "AND" operation. The content of the corresponding area in the tobacco field segmentation map is displayed in the white area of ​​the mask image mask(I)≠0, and the corresponding area in the tobacco field segmentation map is blocked in the black area of ​​the mask image mask(I)=0, so as to separate the tobacco plants from the background area and obtain the tobacco field background removal map.

8. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 1 is characterized in that: In step 3, contour screening is divided into two operation processes, the first is a contour screening process based on the Euclidean distance between contour centroids, and the second is a contour screening process based on the pixel area inside the contour.

9. The tobacco plant segmentation and counting method based on large-scale drone images according to claim 8 is characterized in that: In step 3, the contour screening process using the Euclidean distance between contour centroids is as follows: The centroid of a contour is the geometric center of the closed contour; The coordinates of the centroid are determined by calculating the rectangular moment of the contour, as shown in the following formula: Among them, x i and i Respectively represent the horizontal and vertical coordinates of the center of mass, M 00 Represents the 0th-order moment of the contour, M 01 and M 10 They represent the first-order moment of the contour respectively; Suppose there are two contours C i , C j The Euclidean distance between the centroids is d i,j , the distance threshold is T d , the pixel areas of the contours are A i ,A j ; When the Euclidean distance d between the two contour centroids i,j Greater than or equal to the distance threshold T d When the distance is less than the threshold T, the contour information of the two contours is retained; when the distance is less than the threshold T d When comparing two contours C i , C j The pixel area size is determined by the distance threshold, and the contour information with a larger pixel area is retained to ensure that only one tobacco plant contour information is retained within the distance threshold range; The specific screening strategy is shown in the following formula: Set the pixel area threshold of the tobacco plant outline to T min and T max , to eliminate contours that are too large or too small; when the pixel area of ​​the tobacco plant contour meets T min <A i ≤T max When retaining the tobacco plant contour information C i , otherwise delete the tobacco plant outline information C i .

10. A tobacco plant segmentation and counting system based on large-scale drone images, characterized in that: Includes the following modules: The tobacco field segmentation module is used to build the tobacco field segmentation framework MED-Net based on the DeepLabV3+ network, and integrates the feature extraction network MobileNetV2 and the channel attention mechanism ECA to achieve the segmentation of tobacco field areas and non-tobacco planting areas; The tobacco field background removal module is used to establish a tobacco plant background mask based on the HSV color space. Based on the tobacco field segmentation results, small-sized background areas including soil and mulch are further removed to obtain an image that only retains tobacco plants. The tobacco plant number counting module is used to perform Gaussian filtering, morphological denoising and opening operation denoising on the image that only retains the tobacco plants, and then extract the outlines of the tobacco plants. After screening and counting the extracted outlines of the tobacco plants, the number of tobacco plants is obtained.

Images