Unmanned aerial vehicle wheat image lodging area extraction method based on PSPNet network

Abstract

The present application relates to a UAV wheat image lodging area extraction method based on a PSPNet network, which solves the defect that it is difficult to extract the image of the lodging area of different growth periods of wheat compared with the prior art. The present application comprises the following steps: obtaining and preprocessing training data; constructing a PSPNet semantic segmentation network; training the PSPNet semantic segmentation network; obtaining an image to be extracted of a lodging area; and extracting a UAV wheat image lodging area. The present application overcomes the problem that the lodging extraction of a single growth period of wheat image cannot estimate the lodging area of multiple growth periods, and ideal lodging extraction results are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle image processing, in particular to a wheat image lodging area extraction method based on a PSPNet network. BACKGROUND

[0002] Wheat lodging refers to the whole plant or sheet lodging on the ground due to the influence of various adverse conditions after the jointing stage. Wheat lodging can be divided into early lodging and late lodging from the time, and root lodging and stem lodging from the form. Wheat lodging monitoring technology can timely find wheat lodging and guide agricultural technicians to take remedial measures in time.

[0003] Traditional lodging monitoring methods require investigators to use rulers, GPS and other tools to go to the field to investigate to obtain the location and area of the lodging crops and other information, which is low in efficiency, especially for irregular lodging areas, which cannot be accurately measured. Remote sensing technology can quickly obtain image information and spatial information of a large range of farmland and has been widely used in crop lodging monitoring in recent years. Satellite remote sensing images cover a wide area and are suitable for large-area lodging disaster monitoring. However, satellite remote sensing images also have low spatial resolution, long revisit period, and are greatly affected by weather factors. For specific time and specific area rice lodging monitoring, satellite remote sensing images often cannot meet the demand. At the same time, wheat lodging disasters are generally caused by extreme weather conditions, and poor weather conditions also seriously affect the quality of satellite images. Restricted by the transit period, the satellite revisits after a period of time, which is not accurate enough for damage estimation and cannot meet the requirements of post-disaster rapid assessment.

[0004] In recent years, with the rapid development of unmanned aerial vehicle systems carrying remote sensing platforms, the data splicing method and technology for unmanned aerial vehicle remote sensing images have also matured, and unmanned aerial vehicle remote sensing has carried out certain research in natural disaster information rapid acquisition and physical damage estimation. Unmanned aerial vehicle remote sensing has the characteristics of low cost, high timeliness, small weather influence, etc., and is very suitable for rapid acquisition of crop disaster information.

[0005] In the prior art, although there are some researches on unmanned aerial vehicles in crop growth monitoring, there are few researches on unmanned aerial vehicles in monitoring crop lodging, especially wheat lodging. Some scholars use unmanned aerial vehicle images as data sources to monitor lodging of corn, rice and other crops, but there are few researches on unmanned aerial vehicles in monitoring wheat lodging. Belton et al. demonstrated the application of unmanned aerial vehicle technology in crop height monitoring and modeling to provide quantitative crop growth data. Based on the DSM data obtained by the unmanned aerial vehicle, the crop height was estimated by the CSM derived from photogrammetry, and was compared with the reference height obtained by GNSS to verify the CSM. The evaluation results show that, compared with the wind condition of GNSS height, the average deviation is 2-10 cm. This technology has great potential in quantitative and qualitative monitoring of forest canopy, crop height and growth. Zhao Licheng et al. proposed a wheat lodging identification method based on unmanned aerial vehicle DSM in 2019. Paulo FLORES proposed different machine learning and deep learning algorithms for wheat lodging rate detection in 2021. SUMMARY

[0006] The purpose of the present application is to solve the problem that it is difficult to extract images of different growth period wheat lodging areas in the prior art, and to provide a method for extracting unmanned aerial vehicle wheat image lodging areas based on a PSPNet network to solve the above problems.

[0007] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0008] A method for extracting unmanned aerial vehicle wheat image lodging areas based on a PSPNet network, comprising the following steps:

[0009] Obtaining and preprocessing training data: obtaining unmanned aerial vehicle wheat lodging images that have been sample labeled, dividing them into 20m, 40m RGB images of two growth periods of wheat, and taking them as a training data set;

[0010] Construction of PSPNet semantic segmentation network: using Mobilenetv2 as the main feature extraction network to construct the PSPNet semantic segmentation network;

[0011] Training of PSPNet semantic segmentation network: inputting the training data set into the PSPNet semantic segmentation network for training;

[0012] Obtaining images of lodging areas to be extracted: obtaining unmanned aerial vehicle wheat images to be extracted and preprocessing them;

[0013] Extraction of unmanned aerial vehicle wheat image lodging areas: inputting the preprocessed unmanned aerial vehicle wheat images to be extracted into the trained PSPNet semantic segmentation network to extract the wheat lodging area images.

[0014] The construction of the PSPNet semantic segmentation network comprises the following steps:

[0015] Feature extraction is performed by using a feature extraction network MobileNet2.

[0016] MobileNet2 is set to use 1x1 convolution to increase dimension before 3x3 network structure, and use 1x1 convolution to reduce dimension after 3x3 network structure, and no longer perform Relu6 layer, but directly perform addition of residual network.

[0017] The enhanced feature extraction PPM module is set to divide the obtained feature layer into regions of different sizes, and each region is individually subjected to average pooling.

[0018] The picture features obtained by using the feature extraction network MobileNet2 and the enhanced feature extraction PPM module are used to obtain a prediction result.

[0019] A 3x3 convolution is used to integrate the features, and then a 1x1 convolution is used to adjust the channels to the number of semantic segmentation categories, and finally, resize is used for up-sampling to obtain an output layer with the same width and height as the input picture, thereby obtaining a prediction result.

[0020] The training of the PSPNet semantic segmentation network comprises the following steps:

[0021] The Adam optimizer function is selected as the parameter optimizer, the Dice loss is used as the loss function, the initial learning rate is 0.0001, the batch processing size is set to 32, and the iteration number epoch is set to 100.

[0022] The VOC format data is prepared for training.

[0023] The obtained unmanned aerial vehicle wheat lodging image and label are cropped, each sliding window has a size of 256x256, and finally, the training image has a size of 256x256.

[0024] The training data set is subjected to feature extraction by using the feature extraction network MobileNet2 and the enhanced feature extraction PPM module, and then a 3x3 convolution is used to integrate the features, and then a 1x1 convolution is used to adjust the channels to the number of semantic segmentation categories, thereby obtaining a prediction result.

[0025] Beneficial effects

[0026] Compared with the prior art, the unmanned aerial vehicle wheat image lodging area extraction method based on the PSPNet network overcomes the problem that the single growth period wheat image lodging extraction cannot estimate multiple growth period lodging areas, and ideal lodging extraction results are obtained.

[0027] The application establishes a pyramid scene parsing network (PSPNet) based on a MobileNet2 feature extraction network, and through a pyramid pooling module (PPM) in the MobileNet2 network and the PSPNet network, i.e., a multi-scale feature fusion module, the acquired feature layer is divided into grids of different sizes, each grid is respectively subjected to average pooling, and double feature extraction is performed on the input image.

[0028] In addition, for the problem of a single growth period, the application collects data of two growth periods and two heights of wheat, and simultaneously achieves the purpose of expanding the data set, thereby improving the segmentation performance and generalization ability of the model to a certain extent.

[0029] The MobileNet2 feature network is used as the feature extraction of the PSPNet, and experiments are performed on a wheat double growth period double height data set, and the experimental results show that the PSPNet segmentation method proposed in the application is superior to the comparative segmentation model, and even in the case of wheat data in different growth periods, an ideal unmanned aerial vehicle wheat segmentation result can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The method sequence diagram of the application is shown in the figure;

[0031] Figure 2 The data research area profile diagram of the application is shown in the figure;

[0032] Figure 3 The data annotation diagram of the application using Labelme is shown in the figure;

[0033] Figure 4a The unmanned aerial vehicle wheat original diagram related to the application is shown in the figure;

[0034] Figure 4b The wheat lodging label diagram is shown in the figure; Figure 4a

[0035] The prediction result diagram extracted by using the prior art Unet network is shown in the figure; Figure 4c Figure 4a The prediction result diagram extracted by using the method of the application is shown in the figure.

[0036] Figure 4d Figure 4a DETAILED DESCRIPTION

[0037] ​​​In order to make the structural features and the effects achieved by the present application more further understood and recognized, the preferred embodiments are described in detail below with the accompanying drawings.

[0038] As shown in Figure 1 The unmanned aerial vehicle wheat image lodging extraction method provided by the present application comprises the following steps:

[0039] Firstly, the training data is obtained and preprocessed: the unmanned aerial vehicle wheat lodging image with sample annotation is obtained, which is divided into 20m and 40m RGB pictures of two growth periods of wheat, and is made into 20m and 40m VOC data sets of two growth periods of wheat as a training data set.

[0040] Secondly, the unmanned aerial vehicle wheat image lodging area extraction method based on the PSPNet network, the construction of the PSPNet semantic segmentation network comprises the following steps:

[0041] (1) Feature extraction is performed by using the feature extraction network MobileNet2. MobileNet2 uses 1x1 convolution to increase the dimension before the 3x3 network structure, and uses 1x1 convolution to reduce the dimension after the 3x3 network structure, which is expanded first and then compressed. In order to avoid the damage of Relu to the features, 1x1 convolution is used to increase the dimension before the 3x3 network structure, and 1x1 convolution is used to reduce the dimension after the 3x3 network structure, and then the Relu6 layer is not used, but the addition of the residual network is directly performed.

[0042] (2) The PPM module for strengthening feature extraction divides the obtained feature layer into different size regions, and each region is averaged and pooled inside. The context information of different regions is aggregated to improve the ability to obtain global information. In the typical case of PPM structure, the input feature layer is divided into 6x6, 3x3, 2x2 and 1x1 grids, and each grid is averaged and pooled inside.

[0043] (3) The picture features obtained by using the feature extraction network MobileNet2 and the PPM module for strengthening feature extraction are used to obtain the prediction results. A 3x3 convolution is used to integrate the features, and then a 1x1 convolution is used to adjust the channels, adjust the number of semantic segmentation categories, and finally use resize to upsample the final output layer, so that the width, height and input picture are the same, and the prediction results are obtained.

[0044] Thirdly, the unmanned aerial vehicle wheat image lodging area extraction method based on the PSPNet network, wherein the training of the PSPNet semantic segmentation network comprises the following steps:

[0045] (1) The VOC format data is made for training.

[0046] In order to obtain the data set required for training, the obtained unmanned aerial vehicle wheat lodging image and label are cropped, and the size of each sliding window is set to 256x256. Finally, the size of the training image is 256x256. At the same time, the label also needs to be processed in the same way to obtain a label data set of the same size. Because it is to make a VOC format data set, the cropped original data and label data are respectively placed in the ImageSets and JPEGImages folders. After completing the placement of the data set, the data set needs to be randomly divided into two parts to obtain the training set and the test set. Random selection is to select the wheat image to be uniform in different wheat lodging conditions and different light intensity conditions.

[0047] (2) The training data set is subjected to feature extraction network MobileNet2 and enhanced feature extraction PPM module for feature extraction, and then a 3x3 convolution is used to integrate the features, and then a 1x1 convolution is used to adjust the channel to adjust the number of semantic segmentation categories, and finally the prediction result is obtained. In the model training, the Adam optimizer function is selected as the parameter optimizer, the Dice loss is used as the loss function, the initial learning rate is 0.0001, the batch processing size is set to 32, and the iteration number epoch is set to 100.

[0048] In addition, the evaluation indicators that can be used are the average intersection over union (MIoU) and the average pixel accuracy (MPA). In the field of semantic segmentation, the accuracy of feature extraction is usually determined by the confusion matrix. The definition of the confusion matrix is shown in Table 1. The MPA (average pixel accuracy) and MIoU (average intersection over union) are calculated by the confusion matrix.

[0049] Table 1 Confusion matrix comparison table

[0050]

[0051] TP is the correct detection of the lodging feature, TN is the correct detection of the non-lodging feature, FP is the false detection of the non-lodging feature as the lodging feature, and FN is the false detection of the wheat lodging feature as the non-lodging feature.

[0052] The average pixel accuracy (MPA) is the average value of the ratio of the number of pixels classified correctly to the total number of pixels in each class, and is defined as follows:

[0053]

[0054] where p ii represents the number of pixels classified correctly, and p ijThe number of pixels that belong to class i but are predicted as class j.

[0055] MIoU is a standard evaluation index for segmentation problems, which calculates the intersection of two sets and the coincidence ratio of their union, and the intersection and union ratio between the label marked real lodging and the system predicted lodging is calculated in this paper. This ratio can be redefined as the number of real lodging (intersection) divided by the total number (including real lodging, false lodging, and false non-lodging (union)). MIoU is also calculated by class and averaged, defined as:

[0056]

[0057] Where k represents the number of classes. The average intersection and union ratio reflects the coincidence degree between the predicted graph and the real graph, and the closer the ratio is to 1, the higher the coincidence degree and the higher the quality of semantic segmentation.

[0058] The effect of the present application is further illustrated in combination with a simulation experiment as follows:

[0059] 1. Simulation experiment conditions: Model training is based on a Windows environment, using the Pytorch deep learning framework and Python 3.9 programming language, with tensorflow as the backend. In model training, the Adam optimizer function is selected as the parameter optimizer, the Dice loss is the loss function, the initial learning rate is 0.0001, and the model training is run in an Intel Xeon Gold6248R processor, 192 Gb memory, NVIDIA Quadro P4000 graphics card, GPU (CUDA 10.0) environment. All subsequent training and testing experiments are based on this platform.

[0060] In order to verify the effectiveness of the PSPNet method proposed for wheat lodging extraction, two growth periods and two heights of wheat unmanned aerial vehicle RGB images are used. The research area profile of the data set, the Labelme annotation of the data, the wheat RGB image and the label graph are as shown in Figure 2 、 Figure 3 and Fig. 4.

[0061] In this paper, the average intersection and union ratio (Mean Inetersection over Union, MIoU) and the average pixel accuracy (Mean Pixel Accuracy, MPA) are used as evaluation indexes.

[0062] 2. Simulation experiment content and result analysis:

[0063] In order to verify the influence of different wheat growth periods on the segmentation result in the method, several factors affecting the model segmentation effect of the PSPNet method are analyzed, mainly two growth period data and two height data. And the average value of the segmentation accuracy of 10 experiments is used as the experimental result.

[0064] First, the network training result in single height and single growth period is shown in Table 2:

[0065] Table 2 Comparison table of different network segmentation results of 20m height mature wheat

[0066] Network name MPA / % MIoU / % Unet 84.19 72.91 Deeplabv3+ 91.44 84.07 Segnet 81.88 71.25 PSPNet 92.15 85.33

[0067] Then the network is trained in double growth period and double height, and the training result is shown in Table 3:

[0068] Table 3 Comparison table of different height and different growth period results

[0069]

[0070] As shown in Figure 4a , the standard lodging area extraction diagram is shown in Figure 4b , as shown in Figure 4d , the accuracy of lodging area extraction is obviously better than the effect of Unet method as shown in Figure 4c .

[0071] The above shows and describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.

Claims

1. A method for extracting lodged areas in wheat images from UAVs based on PSPNet network, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of training data: Acquire drone images of wheat lodging that have been labeled with samples, and divide them into 20m and 40m RGB images of wheat at two growth stages as training datasets; 12) Construction of PSPNet semantic segmentation network: PSPNet semantic segmentation network is constructed using Mobilenetv2 as the backbone feature extraction network; The construction of the PSPNet semantic segmentation network includes the following steps: 121) Feature extraction is performed using the MobileNet2 feature extraction network; MobileNet2 is configured to use 1x1 convolutions to increase dimensionality before the 3x3 network structure and 1x1 convolutions to reduce dimensionality after the 3x3 network structure. Instead of using 6 ReLU layers, the addition of the residual network is performed directly. 122) The enhanced feature extraction PPM module is configured to divide the acquired feature layer into regions of different sizes, and average pooling is performed within each region. 123) Obtain prediction results using image features acquired through the MobileNet2 feature extraction network and the enhanced feature extraction PPM module: The features are integrated using a 3x3 convolution, then channel adjustment is performed using a 1x1 convolution to adjust the number of semantic segmentation categories, and finally resizing is used to upsample so that the width and height of the final output layer are the same as the input image, thus obtaining the prediction result. 13) Training the PSPNet semantic segmentation network: Input the training dataset into the PSPNet semantic segmentation network for training; 14) Acquisition of images to be extracted from lodged areas: Acquire drone images of wheat to be extracted and perform preprocessing; 15) Extraction of lodged wheat areas from UAV images: Input the preprocessed UAV wheat images to be extracted into the trained PSPNet semantic segmentation network to extract the lodged wheat areas.

2. The method for extracting lodged areas from UAV wheat images based on PSPNet network according to claim 1, characterized in that, The training of the PSPNet semantic segmentation network includes the following steps: 21) The Adam optimizer function is selected as the parameter optimizer, Dice loss is used as the loss function, the initial learning rate is 0.0001, the batch size is set to 32, and the number of iterations (epochs) is set to 100. 22) Create VOC format data for training: The obtained drone images of wheat lodging and the tags were cropped, and the size of each sliding window was set to 256x256. The final training image size was 256x256. 23) The training dataset is processed by the MobileNet2 feature extraction network and the enhanced feature extraction PPM module for feature extraction. Then, a 3x3 convolution is used to integrate the features, and a 1x1 convolution is used to adjust the channels to the number of semantic segmentation categories, and finally the prediction result is obtained.

Images