Wheat lodging segmentation method based on the Lstm_PSPNet deep learning network

Abstract

The present invention particularly relates to a wheat lodging segmentation method based on the Lstm_PSPNet deep learning network, which comprises the following steps: collecting images of the wheat field to be detected by using an unmanned aerial vehicle, and performing geometric correction and stitching processing to obtain a large RGB image to be detected; inputting the large RGB image to be detected into the trained Lstm_PSPNet network to obtain a lodging area segmentation map; the Lstm_PSPNet network includes a feature map calculation module, an improved spatial pyramid pooling module, and a segmentation prediction module. The present invention improves the PSPNet model by introducing a ConvLSTM neural network with the ability to remember long-term and short-term information, inserting a convolutional attention module CBAM, and a Tversky loss function, and transmitting the temporal relationship between different growth stages through the network one by one, thereby improving the segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of crop lodging detection, and particularly relates to a wheat lodging segmentation method based on an Lstm_PSPNet deep learning network. Background Technique

[0002] As one of the most important food crops in the world, the increase in wheat yield is of great significance for ensuring global food security. All along, affected by pests and diseases, typhoons and heavy rainfall, there are lodging phenomena of different severity levels in the wheat planting process, which seriously threatens the healthy production of wheat and causes serious economic losses. The rapid assessment of the crop lodging area is of great significance for understanding the causes of lodging phenomena, yield assessment, loss estimation and agricultural research. In lodging monitoring, traditional methods mainly rely on manual labor, which is time-consuming and laborious. While based on the latest information processing technology, on the one hand, it can accurately identify wheat lodging, providing a powerful technical means for scientific damage assessment and post-disaster compensation; on the other hand, accurately obtaining wheat lodging information provides strong support for quickly counting wheat yield. Therefore, researching methods and technologies for non-destructive monitoring of wheat lodging has important value and significance for world food stability.

[0003] In recent years, with the rapid development of unmanned aerial vehicle (UAV) platforms and information processing technologies, UAVs have played an increasingly important role in agricultural monitoring. With the advantages of low cost, high efficiency, and high flexibility, UAVs have received extensive attention and been rapidly promoted and applied in precision agriculture. New information processing technologies have played an increasingly important role in image classification, object detection, and image segmentation with their efficient deep learning algorithms. Many scholars at home and abroad have done a lot of work on crop lodging monitoring based on UAV remote sensing technology and deep learning methods. Zhao Xin and others used the Unet network to extract the lodging area of late-maturing rice in the paper "Use of Unmanned Aerial Vehicle Imagery and Deep Learning UNet to Extract Rice Lodging", and the dice coefficient could reach 0.9442. Yang and others used the FCN-AlexNet network to construct an extraction model suitable for rice lodging in the paper "Semantic Segmentation Using Deep Learning with Vegetation Indices for Rice Lodging Identification in Multi-date UAV Visible Images", and the extraction accuracy reached 94.43%, which was better than traditional methods such as the maximum likelihood method. For the extraction of maize lodging areas, Zheng Ergong and others achieved an extraction accuracy of 88.65% for maize lodging areas based on the FCN8s network of VGG16 in the paper "Extraction of Maize Lodging Areas from UAV Images Based on Deep Learning". In addition, in addition to applying existing deep learning models, improving existing models and constructing new models for high-precision estimation of specific crop lodging are also important research directions. For sunflower lodging monitoring, Song and others improved the semantic segmentation network SegNet in the paper "Identifying sunflower lodging based on image fusion and deep semantic segmentation with UAV remote sensing imaging", adding skip connections, separable convolutions, and conditional random fields to the original network, making the improved network achieve an accuracy of 89.8% when extracting lodging, which was better than support vector machines, fully convolutional networks (FCNs), and the original SegNet. Deep learning has achieved certain research results in the extraction of lodging of various crops. However, it has just started and urgently needs more research to serve practical applications.Regarding wheat lodging, due to the cumbersome data collection, most researchers extract wheat lodging based on a single growth stage, and few scholars conduct research on multiple growth stages. Based on unmanned aerial vehicle (UAV) remote sensing data, Zhao et al. constructed extraction models for two growth stages, namely the early filling stage and the maturity stage of wheat, using a convolutional neural network in the paper "Automatic Wheat Lodging Detection and Mapping in Aerial Imagery to Support High-Throughput Phenotyping and In-Season Crop Management", and the average accuracy can reach 89.23%. Zhang et al. automatically extracted the wheat lodging areas in five growth stages, namely the early flowering stage, the late flowering stage, the filling stage, the early maturity stage, and the late maturity stage, based on transfer learning and the DeepLab v3+ network in the paper "Automatic extraction of wheat lodging area based on transfer learning method and deeplabv3+ network", and the extraction accuracies are 90.7%, 87.6%, 88.8%, 92.3%, and 91.3% respectively. However, these studies only simply use the characteristics of each growth stage for segmentation, without considering that lodging information can be transmitted between different growth stages of crops during the segmentation of time-series images. Summary of the Invention

[0004] The purpose of the present invention is to provide a wheat lodging segmentation method based on the Lstm_PSPNet deep learning network, which can more accurately segment the lodging areas in wheat fields.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows: A wheat lodging segmentation method based on the Lstm_PSPNet deep learning network, comprising the following steps: Using a UAV to collect images of the wheat field to be detected, and performing geometric correction and stitching on the images of the wheat field to be detected to obtain a large RGB image to be detected; Inputting the large RGB image to be detected into the trained Lstm_PSPNet network to obtain a lodging area segmentation map; wherein, the Lstm_PSPNet network includes a feature map calculation module, an improved spatial pyramid pooling module, and a segmentation prediction module. The feature map calculation module processes the input five-dimensional data into a feature map of size H×W×C by obtaining the relationship between time-series images through ConvLstm; the improved spatial pyramid pooling module further extracts the channel and spatial information of the images at each pooling size using the CBAM module before convolution and upsampling; the segmentation prediction module performs a convolution operation on the stitched feature map to obtain a lodging area segmentation map.

[0006] Compared with the prior art, the present invention has the following technical effects: In the prior art, only conventional machine learning and deep learning network models are used to segment the characteristics of each growth period, without considering that lodging information can be transmitted between different growth periods of crops during the segmentation of time-series images. By introducing a ConvLSTM neural network with the ability to remember long-term and short-term information, which is a time-series model applied to two-dimensional images and uses the output of the previous layer as the input of the next layer, after adding convolutional operations, not only can the time-series relationship be obtained, but also features can be extracted like a convolutional layer to extract spatial features. Inserting a convolutional attention module CBAM and a Tversky loss function to improve the PSPNet model, and transmitting the time-series relationship between different growth periods layer by layer through the network, thereby improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is the structural diagram of the Lstm_PSPNet model;

[0008] Figure 2 is the structural diagram of the CBAM model;

[0009] Figure 3 are the visible light images and label results of five growth periods;

[0010] Figure 4 are the prediction results of images of different sizes;

[0011] Figure 5 is the comparison chart of segmentation accuracy for each growth period. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following will Figures 1 to 5 be described in further detail in conjunction with

[0013] Refer to Figure 1, the present invention discloses a wheat lodging segmentation method based on the Lstm_PSPNet deep learning network, which includes the following steps: using a drone to collect images of the wheat field to be detected, and performing geometric correction and stitching on the images of the wheat field to be detected to obtain a large RGB image to be detected; inputting the large RGB image to be detected into the trained Lstm_PSPNet network to obtain a lodging area segmentation map; wherein, the Lstm_PSPNet network includes a feature map calculation module, an improved spatial pyramid pooling module, and a segmentation prediction module. The feature map calculation module processes the input five-dimensional data into a feature map of size H×W×C by obtaining the relationship between sequential images through ConvLstm. Here, the five dimensions refer to the number of samples, the number of time slices, and the height, width, and number of channels of the image. One slice is a time series of length 5. The improved spatial pyramid pooling module further extracts the channel and spatial information of the image at each pooling size using the CBAM module before convolution and upsampling; the segmentation prediction module performs convolution operations on the stitched feature map to obtain a lodging area segmentation map. In the prior art, only the original machine learning and deep learning network models are used to segment the features of each growth stage, without considering that during the segmentation of time series images, lodging information can be transmitted between different growth stages of crops. The present invention improves the PSPNet model by introducing a ConvLSTM neural network with the ability to remember long-term and short-term information, inserting a convolutional attention module CBAM, and a Tversky loss function, and transmitting the sequential relationship between different growth stages through the network layer by layer, thereby improving the segmentation accuracy.

[0014] The PSPNet network is a deep learning network proposed to address segmentation problems at different scales. The feature of this network is the introduction of a pyramid pooling module (PPM), which deeply analyzes the high-dimensional feature layer rich in semantic information, enabling the global and local connections of this feature layer to jointly act on the final prediction, thereby improving the segmentation quality of images. Due to many practical problems making it difficult to obtain sufficient high-quality data, PSPNet has good applications in many fields. For example, bridge crack monitoring, mural segmentation, remote sensing image segmentation, etc. However, in the field of crop lodging extraction, few scholars have extracted crop lodging information based on PSPNet. In the field of lodging monitoring, the PSPNet network is applicable to scenarios where the lodging dataset has complex lodging conditions and different sizes of lodging areas. Although the PSPNet network model can fuse sufficient deep features, the shallow features are not fused enough. For drone images rich in detail features, only analyzing high-dimensional features during the parsing process will inevitably lose the details in the image, resulting in blurred segmentation edges.

[0015] The RGB image data obtained by the drone has the characteristics of high pixels and many detailed features. To obtain a better segmentation effect, the segmentation model needs to take into account both the in-depth analysis of a single feature layer and the utilization of features in different dimensions. To obtain a model more suitable for wheat lodging information extraction, the present invention improves the PSPNet model by combining the ConvLSTM time series model, inserting the convolutional attention module CBAM, and the Tversky loss function. Among them, LSTM (Long Short Term Memory) is a neural network with the ability to remember long-term and short-term information, which solves the problem of feature coverage caused by too long a network. When extracting the feature information of an image, it extracts the context information of the image in a similar way to extracting sequence features. ConvLSTM is a time series model applied to two-dimensional images. Its core essence is the same as that of LSTM, taking the output of the previous layer as the input of the next layer. The difference is that after adding the convolutional operation, not only can the time series relationship be obtained, but also features can be extracted like a convolutional layer, extracting spatial features. And the switching between states is also replaced by convolutional calculations.

[0016] Further, the improved spatial pyramid pooling module processes the input feature map according to the following steps: The feature map with a size of H×W×C is pooled to obtain four sub-feature maps with different widths and heights; the channel and spatial information of the sub-feature maps are further extracted through the CBAM module; the number of channels of the feature map output by the CBAM module is adjusted to C / 4 through a 1×1 convolution, so as to splice the features of all levels with the original feature map. The input feature map and the four feature maps after Unpool upsampling are spliced and output, and Unpooling is used instead of Unsampling to further retain the original feature information.

[0017] Further, referring to Figure 2 , the CBAM module includes a channel attention module (Channel Attention Module) and a spatial attention module (Spatial Attention Module), which saves parameters and computing power. Specifically, the CBAM module processes the feature map according to the following steps: The input feature map is processed by the channel attention module to obtain a feature Figure 1 ; the input feature map and the feature Figure 1 are fused to obtain a feature Figure 2 ; the feature Figure 2 is processed by the spatial attention module to obtain a feature Figure 3 ; the feature Figure 2 and the feature Figure 3 are fused and the resulting feature map is output.

[0018] Further, the pooling in obtaining four sub-feature maps with different widths and heights from a feature map with an input size of H×W×C is to divide the input feature map into different sub-regions according to the ratios of 1×1, 2×2, 3×3, and 6×6 respectively to form information expressions of different regions.

[0019] Refer to Figure 3 , similar to other neural networks, the Lstm_PSPNet network here also needs to be trained first. Specifically, the Lstm_PSPNet network is trained through the following steps: Use a drone to capture wheat field images at five growth stages to obtain multiple RGB original pictures; Correct and splice the RGB original pictures to obtain 5 RGB large pictures, with each RGB large picture corresponding to a growth stage; With the help of expert experience, manually mark the lodging areas in the RGB large pictures to obtain the label corresponding to the RGB large pictures. Figure 3 A1 - A5 in are the visible light images at the early flowering stage, late flowering stage, filling stage, early maturity stage, and late maturity stage respectively. Figure 3 B1 - B5 in are the labels corresponding to the visible light images. In the label data, the black part is the non-lodging area. Obtain sample data based on the RGB large pictures and their labels, and divide the sample data into a training data set and a validation data set according to a certain ratio; After training the Lstm_PSPNet network with the training set data, a trained Lstm_PSPNet network is obtained. Since in the obtained drone images, the area gap between the wheat lodging area and the normal growth area is large, resulting in an extremely uneven ratio of normal and lodging samples in the final obtained samples, the tversky function is used as the loss function during training in the present invention. The loss function is constructed based on the Tversky index, which well balances precision and recall. The calculation formula of the tversky function is as follows:

[0020]

[0021] In the formula, α = 0.3, β = 0.7, ε = 10 -7 , TP is the area correctly classified as lodging wheat, FP is the area misclassified as lodging wheat, and FN is the area misclassified as normal wheat.

[0022] Specifically, in the present invention, the Lstm_PSPNet network is built in the Keras framework, with TensorFlow as the backend. The optimizer used is Adadelta, and the original parameters are used by default. The TimeDistributed wrapper is used to apply a layer to each time slice of the input, that is, to perform convolution operations on each sequence of the time dimension separately to extract features. The ConvLstm kernel is set to 5*5, the convolution kernel is set to 3*3, and leaky ReLU and Batch Normalization are followed after ConvLstm. The Batch Size is set to 2 sequences, the training epoch is set to 80, and the learning rate is halved every ten cycles. In the same programming environment, the PSPNet network is built, and the optimizer used is Adadelta, with the original parameters used by default. Based on the dataset constructed in this study, the network is trained 80 times, and the Batch Size is set to 10. Based on the Nvidia 3070 hardware environment, relevant programs are written using Python 3.8 software to implement the above process.

[0023] Further, the five growth stages are respectively the early flowering stage (EF) of wheat, the late flowering stage (LF), the filling stage (F), the early maturity stage (EM), and the late maturity stage (LM). When shooting, the flight altitude of the drone is 20m, and the spatial resolution at this altitude is 0.5cm / pixel. The drone uses software for flight path planning, and the forward overlap is set to 80%, the side overlap is set to 80%, the flight speed is 3m / s, and the camera shooting interval is 2s; Trimble R2 is used to set up ground GPS control points for geometric correction of the drone images.

[0024] Further, the steps of obtaining sample data from the RGB large image and its label and dividing the sample data into a training dataset and a validation dataset according to a certain ratio are as follows: The RGB large image is cropped into a specified size by means of a sliding window. The size of the sliding window can be selected according to actual needs and can be 156*156, 312*312, 468*468, or 624*624. In the present invention, the image size is 468*468, and a total of 2400 RGB images are obtained. After data augmentation by means of flipping, rotating, adding noise, and mean filtering, 7200 images are obtained. The augmented data is grouped according to every five images as a time slice, and a total of 1440 groups are divided. In order to avoid the imbalance of the training set and the validation set data caused by differences in illumination, growth, etc. of the drone image data under a single growth stage, the data of each time slice is randomly shuffled and randomly selected to form a dataset, and the dataset is divided into a training dataset and a validation dataset according to a certain ratio. Specifically, in the present invention, the division is carried out according to a ratio of 4:1.

[0025] To evaluate the classification quality of the model, three evaluation metrics were adopted: the extraction precision Pr (precision), the recall rate Rc (recall), and their combined result F1_score. The extraction precision and recall rate can well reflect the accuracy of the model in extracting the lodging area. These three evaluation metrics are calculated through the following formulas:

[0026]

[0027] In the formula, TP, FP, and FN represent the areas correctly, wrongly classified as lodging wheat, and wrongly classified as normal wheat, respectively.

[0028] Taking the results of the late maturity stage output by the PSPNet and Lstm_PSPNet network models as an example, that is, the numerical results of the three precision metrics Pr (precision), Rc (recall), and F1-score are illustrated. As shown in Table 1, the precision, recall, and F1-score of Lstm_PSPNet are 5.1 percentage points, 6.6 percentage points, and 6.3 percentage points higher than those of PSPNet, respectively. The F1-Score of Lstm_PSPNet reaches 0.950, obtaining the highest precision, fully balancing the prediction errors of precision and recall, indicating that the improved Lstm_PSPNet network model is more effective in lodging prediction.

[0029] Table 1: Precision comparison of deep learning networks before and after improvement

[0030]

[0031] Refer to Figure 4 , here taking the lodging monitoring effect in the late maturity stage as an example, to illustrate the influence of 4 image sizes on the lodging segmentation effect. Among the 4 image sizes, 468*468 performs the best, and the three classification metrics are all better than other sizes. 156*156 has the worst effect, with Precision being 0.943, Recall being 0.925, and F1-Score being 0.934, which are 0.009, 0.015, and 0.016 less than the highest precision respectively. 624*624 is second only to 468*468, and its performance is better than 312*312. 312*312 is better than 156*156. From this, it can be concluded that when using UAV images for wheat lodging segmentation, the larger the image size is not necessarily better, and choosing an appropriate size can better improve the segmentation accuracy. The optimal image size determined in the present invention is 468*468. In actual applications, multiple sizes can be selected and the best image size can be picked out.

[0032] Refer to Figure 5, the lodging monitoring effects at different growth stages were compared. Here, an image size of 468*468 was used as the experimental sample, and the results of wheat lodging extraction at 5 growth stages were obtained based on the Lstm_PSPNet deep learning network. From the early flowering stage to the late maturity stage, the three evaluation indicators increased successively. Precision increased from 0.932 to 0.952, Recall increased from 0.912 to 0.940, and F1-Score increased from 0.922 to 0.950. However, there was little difference between the early flowering stage and the late flowering stage, and their F1-Scores differed only by 0.001. From the late flowering stage to the late maturity stage, the increase in precision was relatively large. Overall, the Lstm_PSPNet model in the present invention had good extraction effects in the middle and late growth stages.

[0033] The present invention also discloses a computer-readable storage medium and an electronic device. Among them, a computer-readable storage medium has a computer program stored thereon. When the computer program is executed by a processor, it implements the wheat lodging segmentation method based on the Lstm_PSPNet deep learning network as described above. An electronic device includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, it implements the wheat lodging segmentation method based on the Lstm_PSPNet deep learning network as described above.

[0034] In previous crop lodging monitoring studies, traditional machine learning methods were mostly used, such as nearest neighbor, linear discriminant analysis, random forest (RF), neural network (NN), and support vector machine (SVM). These studies only simply used the original machine learning and deep learning network models to segment the characteristics of each growth stage, without considering that during the segmentation of time-series images, lodging information can be transmitted between different growth stages of crops. The present invention improves the PSPNet model by introducing a ConvLSTM neural network with the ability to remember long-term and short-term information, inserting a convolutional attention module (CBAM), and a Tversky loss function, etc., and transmitting the time-series relationship between different growth stages through the network one by one, thereby improving the segmentation accuracy. Compared with the original PSPNet model, the improved Lstm_PSPNet network can improve the accuracy of wheat lodging information monitoring; it is clear that choosing an appropriate image size can ensure the maximization of the efficiency of lodging image information; as the growth stage progresses, the segmentation accuracy of the model gradually increases, fully indicating that the model can improve the segmentation accuracy by using time-series characteristics. Compared with previous lodging prediction models designed only for one growth stage, this model has higher application potential in the monitoring of wheat lodging information at multiple growth stages.

Claims

1. A wheat lodging segmentation method based on the Lstm_PSPNet deep learning network, comprising the following features: It includes the following steps: Use a drone to collect images of the wheat field to be detected, and perform geometric correction and stitching on the images of the wheat field to be detected to obtain a large RGB image to be detected; Input the large RGB image to be detected into the trained Lstm_PSPNet network to obtain a lodging area segmentation map; Among them, the Lstm_PSPNet network includes a feature map calculation module, an improved spatial pyramid pooling module, and a segmentation prediction module. The feature map calculation module processes the input five-dimensional data into a feature map of size H×W×C by obtaining the relationship between sequential images through ConvLstm; The improved spatial pyramid pooling module further extracts the channel and spatial information of the image at each pooling size using the CBAM module before convolution and upsampling; The improved spatial pyramid pooling module processes the input feature map according to the following steps: Pool the input feature map of size H×W×C to obtain four sub-feature maps with different widths and heights; Further extract the channel and spatial information of the sub-feature maps through the CBAM module; Adjust the number of channels of the feature map output by the CBAM module to C / 4 through 1×1 convolution; Concatenate the input feature map and the four feature maps after Unpool upsampling and then output; The segmentation prediction module performs a convolution operation on the concatenated feature map to obtain a lodging area segmentation map.

2. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 1, characterized in that: The CBAM module includes a channel attention module and a spatial attention module. The CBAM module processes the feature map according to the following steps: The input feature map is processed by the channel attention module to obtain Feature Map 1; Fuse the input feature map and Feature Map 1 to obtain Feature Map 2; Feature Map 2 is processed by the spatial attention module to obtain Feature Map 3; Output the feature map obtained by fusing Feature Map 2 and Feature Map 3.

3. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 1, wherein: The pooling in the step of pooling the input feature map of size H×W×C to obtain four sub-feature maps with different widths and heights divides the input feature map into different sub-regions according to the ratios of 1×1, 2×2, 3×3, and 6×6 respectively to form information expressions of different regions.

4. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 1, characterized in that: The Lstm_PSPNet network is trained according to the following steps: Use a drone to take images of the wheat field at five growth stages to obtain multiple RGB original images; Perform correction and stitching on the RGB original images to obtain 5 large RGB images, and each large RGB image corresponds to a growth stage; With the help of expert experience, manually mark the lodging areas in the large RGB images to obtain the label corresponding to the large RGB images; Obtain sample data according to the large RGB images and their labels, and divide the sample data into a training data set and a validation data set according to a certain ratio; Use the training set data to train the Lstm_PSPNet network to obtain a trained Lstm_PSPNet network. The tversky function is used as the loss function during training.

5. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 4, characterized in that: The calculation formula of the tversky function is as follows: Where α = 0.3, β = 0.7, and ε = 10 -7 , TP is the area correctly classified as lodged wheat, FP is the area misclassified as lodged wheat, and FN is the area misclassified as normal wheat.

6. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 4, wherein: The five growth stages are early wheat flowering stage, late flowering stage, filling stage, early maturity stage, and late maturity stage. When shooting, the flight altitude of the drone is 20 m, and the spatial resolution at this altitude is 0.5 cm / pixel. The drone uses software for flight path planning, with a forward overlap of 80% and a side overlap of 80%. The flight speed is 3 m / s, and the camera shooting interval is 2 s. Trimble R2 is used to set up ground GPS control points for geometric correction of drone images.

7. The wheat lodging segmentation method based on the Lstm_PSPNet deep learning network according to claim 4, characterized in that: The steps of obtaining sample data from the RGB large image and its label and dividing the sample data into a training data set and a validation data set in a certain proportion are as follows: The RGB large image is cropped into a specified size by means of a sliding window. Data augmentation is performed by means of flipping, rotation, adding noise, and mean filtering. The augmented data is grouped according to every five images as a time slice. The data of each time slice is randomly shuffled and randomly selected to form a data set, and the training data set and the validation data set are obtained by dividing according to a certain proportion.

8. A computer-readable storage medium, characterized in that: It stores a computer program, and when the computer program is executed by a processor, it implements the wheat lodging segmentation method based on the Lstm_PSPNet deep learning network as described in any one of claims 1-7.

9. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, it implements the wheat lodging segmentation method based on the Lstm_PSPNet deep learning network as described in any one of claims 1-7.

Images