Crop lodging area grading identification method and system

Abstract

The invention discloses a crop lodging region grading identification method and system, and the method comprises the steps: obtaining a remote sensing image of crop lodging in a research region, carrying out the preprocessing of the remote sensing image, and carrying out the marking of the preprocessed remote sensing image, and making a data set; establishing a two-stage crop lodging grading monitoring model LHMNet, wherein the two-stage crop lodging grading monitoring model LHMNet comprises an LAR-ResUNet model used for first-stage lodging region identification and an LSC-ResUNet model used for second-stage lodging severity grading of a lodging region identified by the LAR-ResUNet model; the method comprises the following steps: respectively training an LAR-ResUNet model and an LSC-ResUNet model in a crop lodging grading monitoring model LHMNet by adopting data set training to obtain the trained LAR-ResUNet model and the trained LSC-ResUNet model, and then predicting the grading degree of a crop lodging region by adopting the trained crop lodging grading monitoring model LHMNet. According to the method, the lodging region can be clearly and completely extracted from the large region, and meanwhile, the difference of segmentation accuracy of rice with different lodging severity degrees can be reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a disaster classification method based on remote sensing images. Background Art

[0002] Rice lodging is the result of the combined effects of internal and external factors such as meteorological and environmental factors, agricultural management measures, and its own physiological characteristics. Lodging will destroy the original spatial structure of the rice canopy, reduce the photosynthetic efficiency of rice leaves, affect the accumulation of grain dry matter under the shade of leaves, and ultimately lead to a reduction in the number of panicles and panicle weight; at the same time, bent or broken stems will hinder the transmission of water and photosynthetic products inside the rice and form exposed wounds, increasing the risk of infection with pests and diseases and affecting the normal growth of rice. While lodging leads to a decrease in rice yield and quality, it also increases the difficulty of mechanized harvesting and post-harvest grain drying, and increases the manpower and material costs of crop field management. Therefore, strengthening the monitoring and identification of rice lodging disasters, timely and accurately obtaining information such as the location, area, and severity of rice lodging, and estimating the post-disaster crop yield loss are of great significance for scientifically assessing disaster risks, rationally formulating post-disaster field management measures, and accurately implementing agricultural insurance claims and subsidy policies.

[0003] Due to the differences in crop growth conditions and the diversity of factors affecting lodging, lodging disasters at the field scale often have certain temporal and spatial heterogeneity. However, most regions lack systematic and reliable ground crop data collection technology, and it is difficult to implement high-precision lodging monitoring in the region. Traditional crop lodging monitoring mainly relies on manual on-site measurement using handheld GPS, tape measures and other tools. The information collection efficiency is low, the cost is high, and the subjectivity is strong. In addition, the survey process will cause damage to crops [20,21]. It can no longer meet the current requirements of my country's agricultural development for the timeliness, stability and accuracy of large-scale crop lodging monitoring.

[0004] On the other hand, the advantages of artificial intelligence algorithms such as machine learning and deep learning in processing large-scale data have emerged. They make up for the shortcomings of traditional mathematical models in solving complex nonlinear problems and play an important role in improving the efficiency and accuracy of agricultural remote sensing information extraction and analysis

[33] . Traditional machine learning methods are based on feature engineering. Through manual editing processes such as feature selection, feature extraction, and feature evaluation, the original remote sensing data is converted into features that can better reflect the essence of the problem. A large number of iterative calculations are then performed to extract the implicit rules in high-dimensional data and provide repeated reliable decisions. They have good applicability in tasks such as image classification, regression, and clustering. Deep learning goes a step further on the basis of traditional machine learning. Without manually extracting image features, it can rely on the construction of a reasonable deep neural network structure and a large number of parameters to perform more complex feature vector space calculations, and automatically learn more advanced deep semantic features from the original image. Therefore, it can realize crop yield prediction, crop classification, disaster monitoring, and other functions based on the large-scale, high-dimensional training samples provided by remote sensing images, and achieve results with higher accuracy than traditional machine learning algorithms. However, most of the current studies on crop lodging monitoring based on remote sensing images using deep learning can only qualitatively determine whether crops have lodged, and cannot fully utilize remote sensing data to build a targeted crop lodging monitoring model to conduct in-depth analysis of the degree of crop lodging; secondly, most studies are limited to using field-scale drone remote sensing images to monitor crop lodging conditions, which is difficult to provide a reference for macroscopic analysis of crop lodging conditions at a larger spatial scale. Therefore, it is necessary to develop a method that can monitor crop lodging at a large spatial scale. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the prior art and provide a method for grading and identifying crop lodging areas. The present invention can clearly and completely extract the lodging area from a large area, and can reduce the difference in rice segmentation accuracy with different lodging severity levels, thereby solving the problems of unbalanced lodging rice samples and high scale diversity of the target area when grading the severity of rice lodging in the prior art.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for identifying crop lodging areas by classification comprises the following steps:

[0008] Step 1: obtain remote sensing images of crop lodging in the study area, preprocess them, and then annotate the preprocessed remote sensing images to produce a data set;

[0009] Step 2, establishing a two-stage crop lodging classification monitoring model LHMNet, which includes a LAR-ResUNet model for identifying the lodging area in the first stage and an LSC-ResUNet model for grading the lodging severity of the lodging area identified by the LAR-ResUNet model in the second stage, wherein the LAR-ResUNet model and the LSC-ResUNet model are coupled through a weight fusion structure;

[0010] Step 3, use the data set training to train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet respectively, to obtain the trained LAR-ResUNet model and the LSC-ResUNet model, and then use the trained crop lodging grading monitoring model LHMNet to predict the grading degree of crop lodging areas.

[0011] Furthermore, in step 1, a multispectral image and a panchromatic image are obtained, and the multispectral image is subjected to radiation calibration, atmospheric correction and orthorectification to obtain a preprocessed multispectral image. The panchromatic image is subjected to radiation calibration and orthorectification to obtain a preprocessed panchromatic image. A spatial mapping relationship is then established based on the preprocessed panchromatic image for geometric registration. Finally, the two images are fused using the Gram-Schmidt method, and the images are mosaicked and cropped according to the scope of the study area.

[0012] Furthermore, when the degree of lodging of crops is marked in step 1, the severity of lodging is quantitatively divided into no lodging NL, mild lodging LL, moderate lodging ML and severe lodging SL according to the lodging inclination angle; wherein, the angle θ between the crop stem of NL and the vertical direction is between 0 and 5°; the angle θ between the crop stem of LL and the vertical direction is between 5 and 30°; the angle θ between the crop stem of ML and the vertical direction is between 30 and 60°, and the angle θ between the crop stem of SL and the vertical direction is between 60 and 90°.

[0013] Furthermore, when constructing the rice lodging dataset, polygons are drawn to select the target area to make segmentation labels; when making the lodging area labels, non-crop land is regarded as the background area and marked as 255, the pixels of the non-lodging rice area are marked as 0, and the pixels of the lodging rice area are marked as 1; then the lodging situation of the lodging area is graded and labeled, and the pixel values ​​of the LL, ML and SL areas are labeled according to the severity of the lodging. After the labeling is completed, it is cropped to generate a sub-dataset, and then the dataset is randomly divided into a training set, a validation set and a test set.

[0014] Furthermore, the construction method of the LAR-ResUNet model is:

[0015] Based on the Backbone framework, the weighted feature fusion input module WFFI and the edge perception module EPB are added to obtain the LAR-ResUNet model, where WFFI is used in the feature input stage of the network and EPB is used in the feature recovery stage of the network.

[0016] Furthermore, the WFFI module includes an artificial feature fusion module and a channel attention weighting module;

[0017] The original remote sensing image is input into the artificial feature fusion module, which automatically calculates two types of artificial features, namely vegetation index and texture features, and fuses them with the original remote sensing image as the initial input features. After that, a channel attention weighting module is added to the input feature fusion calculation end and the encoder. The channel attention weighting module automatically assigns corresponding weights to different channels according to the importance of channel information to strengthen the attention to effective channel features.

[0018] The EPB module acts in the encoding stage, including an edge feature extraction module and an edge supervision module. The edge feature extraction module extracts edge features based on the convolution structure, and then learns the residual between edge features and edge labels through the supervision of the edge supervision module, providing detailed information about the edge for feature map reconstruction.

[0019] Furthermore, the construction method of the LSC-ResUNet model is:

[0020] Add the multi-attention mechanism module MAM and the adaptive loss function module AdaptiveLoss on the basis of ResUNet;

[0021] The MAM module includes multiple multi-scale self-attention layers MSSA and reverse attention layers RA. The multi-scale self-attention layers MSSA are located at the end of the encoder and decoder respectively to guide the network to pay attention to the spatial importance features at different scales. The reverse attention layer RA is located at the end of each residual operation unit in the decoder to improve the recognition ability of blurred areas.

[0022] Adaptive Loss is an improvement on Focal Loss. It combines the improved loss function with Dice Loss to replace the commonly used cross entropy. By changing the penalty intensity of different types of samples, it dynamically adjusts the optimization direction of the model prediction results to allow the network to develop in the direction of improving the classification accuracy of difficult samples.

[0023] Furthermore, in step 2, the output structure of the LAR-ResUNet model and the encoder structure of the LSCResUNet model are improved, and then the lodging area information output by the LAR-ResUNet model in the first stage is coupled with the LSCResUNet model in the second stage in a layer-by-layer weighted form to obtain the crop lodging grading monitoring model LHMNet.

[0024] Furthermore, methods for improving the output structure of the LAR-ResUNet model and the encoder structure of the LSCResUNet model include:

[0025] The output structure of the LAR-ResUNet model is improved by channel-superimposing the edge features and the encoder upsampled features, restoring the feature map size, and using 1×1 convolution to generate a feature map X with the same size as the input remote sensing image. S1 ; Use Sigmoid function for activation, and convert X S1 The output value on is mapped to the range of 0 to 1, and the probability of each point pixel belonging to the lodging and non-lodging area is represented in the channel dimension, and the pixel category probability map X is obtained. S1p ; Use continuously stacked 3×3 convolutions, batch normalization, and ReLU activation to extract probability features step by step, and obtain a probability feature map with the same size as the corresponding layer of LSC-ResUNet as the attention weight W of the lodging area L , and perform the above-mentioned probability feature extraction operation multiple times;

[0026] The input structure of the LSC-ResUNet model is improved by replacing W L The feature X extracted from the same layer as the encoder of the LSC-ResUNet model S2 Multiply the elements at the corresponding positions to make the model pay more attention to the high probability areas given in the first stage and generate the weighted feature X of the lodging area S2p ; Then X S2p With the original feature X S2 Add the elements at the corresponding positions to obtain a new feature map X that integrates the probability features of the lodging area S2new , the calculation process is as follows:

[0027] X S2new,ij =W L,ij X S2,ij +X S2,ij

[0028] After the probability feature fusion is completed, X S2new The residual operation and feature downsampling are performed again in the encoder calculation process of the LSC-ResUNet model; the above-mentioned improvement operations are performed in each layer of the encoder of the LSC-ResUNet model.

[0029] Another object of the present invention is to provide a system for implementing the above-mentioned crop lodging area classification identification method, characterized in that it includes:

[0030] The data acquisition module is used to obtain remote sensing images of crop lodging in the study area, pre-process them, and then annotate the pre-processed remote sensing images to produce a data set;

[0031] A model building module is used to build a LAR-ResUNet model for identifying lodging areas in the first stage and a LSC-ResUNet model for grading the severity of lodging in the second stage for the lodging areas identified by the LAR-ResUNet model. The LAR-ResUNet model and the LSC-ResUNet model are then coupled through a weight fusion module to obtain a two-stage crop lodging grading monitoring model LHMNet;

[0032] The model training and prediction module is used to respectively train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet using the data set training to obtain the trained LAR-ResUNet model and the LSC-ResUNet model, and then use the trained crop lodging grading monitoring model LHMNet to predict the grading degree of crop lodging areas.

[0033] Compared with the prior art, the present invention has the following beneficial effects: the present invention constructs a LAR-ResUNet model for first-stage lodging area identification and an LSC-ResUNet model for second-stage lodging severity classification of the lodging area identified by the LAR-ResUNet model, and then couples the LAR-ResUNet model and the LSC-ResUNet model through a weight fusion module to obtain a two-stage crop lodging classification monitoring model LHMNet. Compared with other advanced convolutional semantic segmentation models, LHMNet has significant advantages in crop lodging classification identification and monitoring tasks; LHMNet can clearly and completely extract the lodging area from the field while reducing the difference in segmentation accuracy of rice with different lodging severities, solving the problems of imbalanced samples of lodging rice and high scale diversity of the target area when grading the severity of rice lodging in the prior art, improving the feature extraction, interaction structure and loss function of the network, and realizing the graded monitoring of rice lodging severity on a large regional scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method for grading and identifying crop lodging areas according to an embodiment of the present invention;

[0035] Figure 2This is a flowchart of remote sensing image preprocessing according to an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the structure of the LAR-ResUNet model according to an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of the structure of the LSC-ResUNet model according to an embodiment of the present invention;

[0038] Figure 5 It is the weight fusion structure of the LHMNet model in an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of the framework of the LHMNet model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in combination with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0042] The present invention will be further described below in conjunction with specific embodiments, but the present invention is not limited thereto.

[0043] like Figure 1 As shown, the embodiment of the present invention discloses a method for grading and identifying crop lodging areas, comprising the following steps:

[0044] Step 1: obtain remote sensing images of crop lodging in the study area, preprocess them, and then annotate the preprocessed remote sensing images to produce a data set;

[0045] This paper takes Friendship Farm in Shuangyashan City, Heilongjiang Province as the research area. The cultivated land area of ​​Friendship Farm is about 12.6×10 4 ha, with concentrated and contiguous land, large-scale and high degree of mechanization, it is an important dry grain planting base in northern China. The main grain crops include rice, corn, soybeans, etc. Among them, the rice planting area is about 6.4×10 4 ha, accounting for more than 50% of the total area of ​​food crops, so this example uses rice as the research crop.

[0046] This embodiment uses the remote sensing images of Gaofen-2 and Gaofen-7 satellites as data sources. Gaofen-2 satellite is equipped with two high-resolution 0.8m panchromatic and 3.2m multispectral cameras, which have the characteristics of high positioning accuracy, high radiation accuracy and rapid attitude maneuvering. Gaofen-7 satellite operates in a sun-synchronous orbit with an orbital altitude of about 500km and a regression period of 60 days. The satellite is equipped with two linear array stereo cameras with a sensor width of 20km, which can capture panchromatic images and multispectral images. The spatial resolution of panchromatic images is 0.65m, and the spatial resolution of multispectral images is 2.6m. The specific parameters are shown in Tables 2.1 and 2.2. Download the Gaofen-2 and Gaofen-7 satellite remote sensing images from the Land Observation Satellite Data Service Platform of the China Resources Satellite Application Center (https: / / data.cresda.cn / ), which correspond to the two main growth stages of rice lodging disasters in the study area, so that they can cover the rice planting area of ​​Youyi Farm as much as possible while ensuring the imaging quality, including four scenes of Gaofen-2 satellite remote sensing images, the imaging date is August 31, 2022. Two scenes of Gaofen-7 satellite remote sensing images, the imaging date is September 27, 2022. The downloaded files include the original image files in GeoTIFF format, metadata files storing image information, and raster data processing files recording RPC model parameters.

[0047] During the imaging process, the satellite will be affected by its own conditions and environmental factors such as flight attitude, orbital altitude, atmospheric radiation, and solar altitude angle, resulting in varying degrees of radiation distortion and geometric distortion, making it difficult for the remote sensing images directly captured to accurately reflect the true characteristics of the ground objects. Therefore, the original remote sensing images need to be preprocessed.

[0048] The remote sensing image preprocessing process of this embodiment mainly includes multispectral and panchromatic image radiation correction and geometric correction, two-image registration fusion and mosaic cropping. The specific preprocessing process is as follows: Figure 2As shown. First, the multispectral image was radiometrically calibrated using ENVI5.3 remote sensing image processing software (NV5 Geospatial Solutions, Inc., VA, USA), and the output grayscale value was converted into a radiation brightness value with actual physical meaning to eliminate the system error caused by the sensor itself. Since the rice lodging monitoring process involves quantitative calculation of each pixel and extraction of biophysical quantities, the accuracy of remote sensing data is required to be high. Therefore, the multispectral image after radiometric calibration was atmospherically corrected to minimize the radiation error caused by atmospheric scattering and absorption. The multispectral image was orthorectified based on the digital elevation model DEM to correct the image geometric distortion caused by terrain changes. After that, the panchromatic image was radiometrically calibrated and orthorectified using a method similar to that of the multispectral image. In particular, since the panchromatic image is not used for quantitative calculation, atmospheric correction is not required. In order to obtain a multispectral image with a higher spatial resolution, the corrected panchromatic image and the multispectral image were fused. Before the image fusion, it was necessary to establish a spatial mapping relationship based on the panchromatic image with a higher spatial resolution for geometric registration to eliminate the spatial position error between the two images. Finally, the Gram-Schmidt method was used to fuse the two images, and the images were mosaicked and cropped according to the scope of the study area to obtain the preprocessed image.

[0049] After obtaining the pre-processed images, they are manually annotated. In this embodiment, combined with the research objectives and referring to the existing research methods, the severity of rice lodging is quantitatively divided into four levels according to the lodging inclination angle, namely, none-lodging (NL), light lodging (LL), moderate lodging (ML) and severe lodging (SL). Among them, NL rice basically remains upright, and the angle θ between the stem and the vertical direction is NL Between 0 and 5°; LL The rice stems are slightly tilted, with an angle θ with the vertical direction LL Between 5 and 30°; the stems of ML rice are obviously tilted, with an angle θ with the vertical direction ML 30-60°, SL rice is close to or completely lying on the ground, and the angle θ with the vertical direction SL It is 60~90°.

[0050] When constructing the rice lodging dataset, ENVI 5.3 remote sensing image processing software was selected as the platform for annotating segmentation labels, and the ROI tool was used to draw polygons to select the target area to make segmentation labels. When making rice lodging area labels, roads, water bodies, trees and other non-rice objects were regarded as background areas and labeled as 255, the pixels of non-lodging rice areas were labeled as 0, and the pixels of lodging rice areas were labeled as 1. After that, the lodging conditions inside the rice fields were further labeled, and the pixel values ​​of mild lodging, moderate lodging and severe lodging rice areas were labeled as 2, 3 and 4 respectively according to the severity of rice lodging. Since the size of the entire remote sensing image is too large to be directly input as a data set into the deep learning model for training, this embodiment uses a sliding window with a size of 256×256 pixels to sequentially crop the preprocessed satellite remote sensing image and the corresponding label image to generate an original data set containing 1,200 remote sensing images and lodging labels respectively. After that, this embodiment randomly divides the data set into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0051] In addition, in order to further improve the robustness and generalization ability of the rice lodging monitoring model, enhance the adaptability of the model to different scenes, and reduce model overfitting, this embodiment uses two main methods, geometric transformation and color transformation, to perform data enhancement on the images of the training set and the verification set. Among them, the geometric transformation is to rotate, flip, horizontally transform and randomly scale the original image and label in sequence with a certain probability to simulate the differences in the scale and angle of the target object; the color transformation is to simulate the differences in the lighting environment and external interference by adding Gaussian noise and changing the image brightness and contrast. The degree of image transformation is changed by randomly selecting a given range of data enhancement parameters and repeating the above operations, and finally the number of samples in the training set and the verification set is expanded to 2880 and 960.

[0052] Step 2, establishing a two-stage crop lodging classification monitoring model LHMNet, which includes a LAR-ResUNet model for identifying the lodging area in the first stage and an LSC-ResUNet model for grading the lodging severity of the lodging area identified by the LAR-ResUNet model in the second stage, wherein the LAR-ResUNet model and the LSC-ResUNet model are coupled through a weight fusion structure;

[0053] In order to fully explore and utilize the effective information in satellite remote sensing images and further improve the accuracy of quantitative monitoring of rice lodging, this embodiment first constructs a LAR-ResUNet model for lodging area identification, then constructs a LSC-ResUNet model for lodging severity identification, and then couples the two to obtain a two-stage lodging classification monitoring model LHMNet, which realizes end-to-end lodging area classification monitoring.

[0054] The LAR-ResUNet model is based on the backbone network framework, and adds a weighted feature fusion input module (Weighted Feature Fusion Input, WFFI) and an edge perception block (Edge Perception Block, EPB), which are applied to the feature input and feature recovery stages of the network respectively.

[0055] Compared with ordinary images, multispectral remote sensing images contain more intensive and complex information. Therefore, when deep learning networks extract features with specific goals, they are more easily disturbed by noise, resulting in insufficient and irrelevant feature extraction and falling into local optimal solutions. It usually requires the production of large-scale data sets and long-term learning and training. Integrating specific prior knowledge into the model can provide targeted semantic information for the model, provide important features that are difficult for neural networks to learn directly from multispectral reflectance images, enhance the model input's ability to describe segmented targets, and ensure the depth of feature extraction while saving redundant calculations. Among them, WFFI mainly includes an artificial feature fusion module and a channel attention weighting module. The original remote sensing image is input into WFFI, and the artificial feature fusion module automatically calculates two types of artificial features, vegetation index and texture features, and fuses them with the original remote sensing image as the initial input features. Afterwards, the ECA channel attention weighting module is added to the input feature fusion calculation end and the encoder. The channel attention weighting module automatically assigns corresponding weights to different channels according to the importance of channel information, strengthens the attention to effective channel features, and improves the utilization efficiency of the input multi-channel image.

[0056] Edge features are one of the basic information of an image. They refer to the set of pixels whose grayscale values ​​suddenly change between adjacent areas in an image. They have strong discontinuity characteristics in mathematics. There are relatively obvious edge features between the images of lodging rice and non-lodging rice areas, which can provide effective spatial position relationship information for the extraction of rice lodging areas. However, in the process of feature extraction, edge features as local information will be partially discarded by the neural network, resulting in the problem that the lodging rice area finally segmented by the model often has inaccurate boundaries and is confused with the background. In order to make full use of the edge information in the image and improve the accuracy of lodging rice edge segmentation, the EPB edge perception module is designed. The EPB module mainly acts on the encoding stage, including edge feature extraction and edge supervision. The EPB module is based on the edge features extracted by the convolution structure. Through the supervision of the edge loss function, it learns the residual between the edge features and the edge labels, and provides detailed information about the edges for feature map reconstruction.

[0057] The LAR-ResUNet model structure is as follows Figure 3 As shown, the size of C 0×H×W remote sensing image X is input to the network, where C 0 is the number of channels, H is the height of the feature map, and W is the width of the image.

[0058] (1) First, based on the artificial feature fusion calculation in the WFFI module, the initial fusion input feature X combining vegetation index and texture features is obtained. f , size C 1 ×H×W; then, ECA assigns different weights to channel features of different importance to obtain the weighted fusion input feature X wf .

[0059] (2) X wf In the input encoder, first, it is converted into the initialization feature X by the initial residual operation unit 0 , with a size of 32×H×W. Then, three consecutive residual operation units and ECA are used to extract image features step by step, increase feature channels and compress feature sizes, and obtain three feature extraction levels with gradually deepening features X 1 , X 2 and X 3 , the sizes are 64×H / 2×W / 2, 128×H / 4×W / 4, and 256×H / 8×W / 8 respectively, then X 3 Input the residual operation unit that acts as a bridge to obtain the deepest feature X 4 , the size is 512×H / 8×W / 8.

[0060] (3) The feature X calculated by the encoder 4 In the input decoder, after three consecutive transposed convolutions and residual operation units, the feature size is restored step by step and the feature channels are compressed. At the same time, X 1 , X 2 and X 3 Fuse with the corresponding features generated in the decoder to obtain three features X with gradually richer information 5 , X 6 and X 7 , the sizes are 256×H / 4×W / 4, 128×H / 2×W / 2 and 64×H×W respectively.

[0061] (4) X 5 , X 6 and X 7 They are input into the EPB module respectively and converted into multi-scale edge features X using the edge feature extraction structure (EFE) e , using the edge supervision (ES) structure to generate edge labels for X e After supervision and guidance,7 Fusion, then transposed convolution upsampling to obtain the feature X before network output 8 , the size is 32×H×W.

[0062] (5) Finally, an output layer consisting of a transposed convolution, a 1×1 convolution kernel, and a Sigmoid activation function is used to compress the feature map channels and obtain the segmentation result Y with a size of 1×H×W.

[0063] In order to solve the problem that the existing technology is difficult to identify the lodging severity classification task, this embodiment also constructs a lodging severity classification model LSC-ResUNet. Compared with the rice lodging area identification task, grading the severity of the lodging rice faces more uncertainty problems: First, with the increase of image segmentation types, the sample imbalance problem is aggravated, and the probability of misclassification of some discretely distributed or small-area rice lodging samples at the edge increases. Secondly, the inter-class similarity between rice with different lodging severities is small, while the intra-class differences in different regions and times are large, which makes the parts between the lodging severity areas more easily confused. In addition, the scale diversity problem of the same type of lodging rice areas also puts higher requirements on the network's ability to take into account both coarse-grained and fine-grained segmentation.

[0064] In this embodiment, LSC-ResUNet designs two new modules, Multi-Attention Mechanism (MAM) and Adaptive Loss, based on the original ResUNet. The MAM module guides the network to focus on important areas or features in the image through a combination of different types of attention mechanisms, thereby improving the semantic segmentation performance of the model, including two multi-scale self-attention layers (MSSA) and three reverse attention layers (RA). Among them, the multi-scale self-attention layer MSSA is located at the end of the encoder and decoder respectively, which is used to guide the network to focus on the spatial importance features at different scales and improve the network's ability to comprehensively consider the global features and local details of the image; the reverse attention layer RA is located at the end of each residual operation unit in the decoder, and the importance of uncertain pixels such as the boundaries of different segmentation types is highlighted to improve the recognition ability of fuzzy areas.

[0065] Adaptive Loss improves Focal Loss and combines the improved loss function with Dice Loss to propose a new adaptive loss function to replace the commonly used cross-entropy (CE Loss) loss function. By changing the penalty intensity of different types of samples, the optimization direction of the model prediction results is dynamically adjusted, allowing the network to develop in the direction of improving the classification accuracy of difficult samples, so as to solve the more serious problem of multi-classification sample imbalance in the task of rice lodging severity grading.

[0066] The LSC-ResUNet model structure is as follows Figure 4 As shown, it retains the WFFI module in LAR-ResUNet to improve the efficiency of feature extraction.

[0067] (1) When simulating the model, first set the size of C 0 ×H×W remote sensing image X is input into the WFFI module for weighted feature fusion to obtain the input feature X wf , size C 1 ×H×W. Then X wf In the input encoder, three consecutive residual operation units and ECA are used to extract image features step by step to obtain feature X 1 , X 2 and X 3 , the sizes are 64×H / 2×W / 2, 128×H / 4×W / 4 and 256×H / 8×W / 8 respectively.

[0068] (2) Next, X 3 In the input connection layer, the residual operation unit is first used to further extract image features and increase the channel depth of the feature map. It is then connected with a multi-scale self-attention layer MSSA to extract the deepest feature X that integrates the spatial attention of different scales. 4 , the size is 512×H / 8×W / 8.

[0069] (3) Then, the feature X 4 In the input decoder, three consecutive decoding structures are used to restore the feature size and compress the feature channel step by step. Each decoding structure consists of a transposed convolution, a residual operation unit and an RA. The high-level semantic features obtained by upsampling the transposed convolution are respectively compared with the features X of the corresponding size in the encoder. 1 , X 2 and X 3 Fusion, and use RA to give them details and structural features about non-salient areas, to obtain feature X 5 , X 6 and X 7 , the sizes are 256×H / 8×W / 8, 128×H / 4×W / 4 and 64×H / 2×W / 2 respectively.

[0070] (4) Finally, a transposed convolution is used to restore the feature map size to obtain a feature X of size 32×H×W 8 , then X 8 It is input into an MSSA again, and the context information of different scales is fused before the feature output to ensure that the model can capture global information when making predictions. Finally, a 1×1 convolution kernel and a Sigmoid activation function are used to compress the feature channel and perform nonlinear mapping, and the segmentation result Y is output with a size of 1×H×W.

[0071] The effective connection of the above two different models is the key to realize the two-stage crop classification recognition monitoring. For this reason, this embodiment designs a weight fusion structure to fuse the two models, such as Figure 5 As shown in the figure, by improving the output structure of the LAR-ResUNe model and the encoder structure of the LSC-ResUNet model, the rice lodging area information extracted by the first-stage network can be effectively combined with the second-stage network in the form of layer-by-layer weighting, so as to improve the second-stage network's attention to the inside of the rice lodging area during the rice lodging severity classification process, and reduce the interference of pixels with similar inter-class characteristics in the non-lodging rice area on further classification. The specific method is as follows:

[0072] First, the output structure of LAR-ResUNet is improved. After superimposing the edge features with the encoder upsampled features and restoring the feature map size to 32×H×W, a 1×1 convolution is used to generate a feature map X with the same size as the input remote sensing image and a channel number of 2. S1 . Then use the Sigmoid function to activate X S1 The output value on the map is mapped to the range of 0 to 1, and the probability of each point pixel belonging to the lodging rice and non-lodging rice area is represented in the channel dimension, and the pixel category probability map X is obtained. S1p After that, the probability features are extracted step by step using continuously stacked 3×3 convolutions, batch normalization, and ReLU activation, and the probability feature map with the same size as the corresponding layer of LSC-ResUNet is obtained as the attention weight W of the rice lodging area. L ,As the network deepens, a total of four probabilistic feature extraction operations are performed.

[0073] Then, the input structure of LSC-ResUNet is improved by converting W L Feature X extracted from the same layer as the LSC-ResUNet encoder S2 Multiply the elements at the corresponding positions to make the network pay more attention to the high probability areas given in the first stage and generate the weighted feature X of the rice lodging area S2p . Then X S2p With the original feature XS2 Add the elements at the corresponding positions to obtain a new feature map X that integrates the probability features of the rice lodging area S2new , the calculation process is shown in formula 5.1:

[0074] X S2new,ij =W L,ij X S2,ij +X S2,ij

[0075] After the probability feature fusion is completed, X S2new The residual operation and feature downsampling were performed again in the encoder calculation process of LSC-ResUNet. In order to improve the depth of weight fusion of rice lodging area, the above improvement operation was performed in each layer of the encoder of LSC-ResUNet for a total of four times.

[0076] Through the above coupling, the overall operation framework of the two-stage crop lodging classification monitoring model LHMNet is obtained as follows: Figure 6 shown.

[0077] Step 3, using the data set training to respectively train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet to obtain the trained LAR-ResUNet model and the LSC-ResUNet model, and then using the trained crop lodging grading monitoring model LHMNet to predict the grading degree of crop lodging areas;

[0078] During training, in the first stage of LHMNet, the data set is input into the LAR-ResUNet model for training, with the goal of extracting the rice lodging area and outputting the probability distribution of each pixel to characterize the probability of it belonging to the rice lodging / non-lodging area. In the second stage of LHMNet, the pixel category probability map is first converted into the attention weight of the rice lodging area, and superimposed with the remote sensing image as the network input. After that, the LSC-ResUNet model is used to classify the severity of rice lodging, and the final result of quantitative monitoring of rice lodging is obtained.

[0079] Specifically, first, the first-stage LAR-ResUNet model is trained using the training set and its lodging area labels in the data set and verified using the validation set and its lodging area labels, and then the prediction results of lodging area identification are output. Then, the rice lodging area identification prediction results of the first-stage LAR-ResUNet model are tested using the test set and its lodging area labels. After that, the second-stage LSC-ResUNet model is trained using the training set and its lodging severity labels in the data set and verified using the validation set and its lodging area labels. During training and verification, the lodging area probability prediction results output by the first stage are used as attention weights and fused with the input of the second stage to obtain the rice lodging severity classification results. The test set and its lodging severity labels are then used to test the rice lodging severity classification results of the second-stage network. Finally, the evaluation index is used to evaluate the trained crop lodging classification monitoring model LHMNet, and the crop lodging classification monitoring model LHMNet with the accuracy symbol requirement is selected as the optimized crop lodging classification monitoring model LHMNet. Among them, this embodiment uses precision (Precision), recall (Recall), F1 score (F1), intersection over Union (IoU), mean pixel accuracy (MPA) and mean intersection over Union (MIoU) as evaluation indicators of the crop lodging classification monitoring model LHMNet to evaluate the accuracy of the model from different angles.

[0080] In order to verify the superiority of the proposed two-stage LHMNet in the task of crop lodging grading monitoring, this example selected 6 other advanced convolutional semantic segmentation models, namely FPN, PAN, MANet, UNet, UNet++ and DeepLabV3+, and trained them under the same experimental environment to obtain their respective rice lodging grading monitoring models. The test results of the trained different models on the test set were compared with LHMNet, and the results in Table 1 were obtained.

[0081] Table 1 gives the detailed comparison results of the extraction accuracy of rice areas with different lodging severity of each network. It can be seen from Table 1 that LHMNet has higher extraction accuracy than other networks for rice areas with different lodging severity on the test set, and has good accuracy and generalization ability. Especially in the LL and ML rice areas, the F1 and IoU of LHMNet are improved most significantly, increasing by 16.82% and 25.53%, and 12.45% and 20.17% respectively compared with FPN. This embodiment believes that this is due to the splitting of complex tasks and the attention to difficult samples and multi-scale information in the network structure design of LHMNet. On the one hand, the use of a two-stage network framework allows different networks to focus on solving a single problem, avoiding the interference of background pixels with similar characteristics between classes on the difficult-to-classify lodging rice types. On the other hand, the introduction of multiple modules such as WFFI, EPB, MAM and the improvement of the network loss function improve the model's ability to obtain more advanced and rich semantic information, and improve the classification accuracy of difficult samples.

[0082] Further analysis of the rice lodging classification monitoring results of LHMNet shows that the network has the highest extraction accuracy for NL rice area, with IoU and F1 of 98.86% and 97.75% respectively, followed by SL and ML rice, with IoU and F1 of 97.66% and 95.43% and 95.80% and 91.93% respectively, and the extraction effect for LL rice is relatively the worst, with IoU and F1 scores of 92.18% and 87.01% respectively. Compared with other semantic segmentation networks, LHMNet has achieved relatively good segmentation accuracy for different types of lodging rice, and the differences between types are small. The IoU and F1 of NL and LL rice differ by 6.68% and 10.74% respectively. In contrast, in UNet++, which has the second highest overall segmentation accuracy, the IoU and F1 of NL rice and LL rice differ by 10.92% and 18.56% respectively. This shows that LHMNet effectively reduces pixel confusion between different types of lodging rice areas and alleviates the sample imbalance problem in the rice lodging classification monitoring task.

[0083] Table 5.3 Comparison of rice lodging extraction results of different networks with different lodging severity on the test set

[0084]

[0085]

[0086] It can be seen that LHMNet has significant advantages over other advanced convolutional semantic segmentation models in the task of rice lodging classification extraction. LHMNet can clearly and completely extract the lodging rice area from the field, while narrowing the difference in segmentation accuracy of rice with different lodging severity, thus improving the accuracy of monitoring.

[0087] The embodiment of the present invention further provides a system for implementing the above-mentioned crop lodging area classification identification method, including:

[0088] The data acquisition module is used to obtain remote sensing images of crop lodging in the study area, pre-process them, and then annotate the pre-processed remote sensing images to produce a data set;

[0089] A model building module is used to build a LAR-ResUNet model for identifying lodging areas in the first stage and a LSC-ResUNet model for grading the severity of lodging in the second stage for the lodging areas identified by the LAR-ResUNet model. The LAR-ResUNet model and the LSC-ResUNet model are then coupled through a weight fusion module to obtain a two-stage crop lodging grading monitoring model LHMNet;

[0090] The model training and prediction module is used to respectively train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet using the data set training to obtain the trained LAR-ResUNet model and the LSC-ResUNet model, and then use the trained crop lodging grading monitoring model LHMNet to predict the grading degree of crop lodging areas.

[0091] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the specification of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying crop lodging areas by classification, characterized in that: The following steps are involved: Step 1: obtain remote sensing images of crop lodging in the study area, preprocess them, and annotate the preprocessed remote sensing images to produce a data set; Step 2, establishing a two-stage crop lodging classification monitoring model LHMNet, which includes a LAR-ResUNet model for identifying the lodging area in the first stage and an LSC-ResUNet model for grading the lodging severity of the lodging area identified by the LAR-ResUNet model in the second stage, wherein the LAR-ResUNet model and the LSC-ResUNet model are coupled through a weight fusion structure; Step 3, using the data set training to respectively train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet, obtain the trained LAR-ResUNet model and the LSC-ResUNet model, obtain the trained crop lodging grading monitoring model LHMNet, and use it to predict the grading degree of crop lodging areas.

2. The crop lodging area classification and identification method according to claim 1 is characterized in that: The remote sensing images obtained in step 1 include multispectral images and panchromatic images. The multispectral images are subjected to radiation calibration, atmospheric correction and orthorectification to obtain the preprocessed multispectral images. The panchromatic images are subjected to radiation calibration and orthorectification to obtain the preprocessed panchromatic images. The spatial mapping relationship is then established based on the preprocessed panchromatic images for geometric registration. Finally, the Gram-Schmidt method is used to fuse the two images, and the images are mosaicked and cropped according to the scope of the study area.

3. The method for identifying crop lodging areas by classification according to claim 1, characterized in that: When the degree of lodging of crops is marked in step 1, the severity of lodging is quantitatively divided into non-lodging NL, mild lodging LL, moderate lodging ML and severe lodging SL according to the lodging inclination angle; wherein, the angle θ between the crop stem of NL and the vertical direction is between 0 and 5°; the angle θ between the crop stem of LL and the vertical direction is between 5 and 30°; the angle θ between the crop stem of ML and the vertical direction is between 30 and 60°, and the angle θ between the crop stem of SL and the vertical direction is between 60 and 90°.

4. The method for identifying crop lodging areas by classification according to claim 3, characterized in that: When constructing the rice lodging dataset, polygons are drawn to select the target area to make segmentation labels; when making the lodging area labels, non-crop land is regarded as the background area and marked as 255, the pixels of the non-lodging rice area are marked as 0, and the pixels of the lodging rice area are marked as 1; then the lodging situation of the lodging area is graded and labeled, and the pixel values ​​of the LL, ML and SL areas are labeled according to the severity of the lodging. After the labeling is completed, it is cropped to generate a sub-dataset, and then the dataset is randomly divided into a training set, a validation set and a test set.

5. The method for identifying crop lodging areas by classification according to claim 1, characterized in that: The construction method of the LAR-ResUNet model is: Based on the Backbone framework, the weighted feature fusion input module WFFI and the edge perception module EPB are added to obtain the LAR-ResUNet model, where WFFI is used in the feature input stage of the network and EPB is used in the feature recovery stage of the network.

6. The method for identifying crop lodging areas by classification according to claim 5, characterized in that: The WFFI module includes an artificial feature fusion module and a channel attention weighting module; The original remote sensing image is input into the artificial feature fusion module, which automatically calculates two types of artificial features, namely vegetation index and texture features, and fuses them with the original remote sensing image as the initial input features. After that, a channel attention weighting module is added to the input feature fusion calculation end and the encoder. The channel attention weighting module automatically assigns corresponding weights to different channels according to the importance of channel information to strengthen the attention to effective channel features. The EPB module acts in the encoding stage, including an edge feature extraction module and an edge supervision module. The edge feature extraction module extracts edge features based on the convolution structure, and then learns the residual between edge features and edge labels through the supervision of the edge supervision module, providing detailed information about the edge for feature map reconstruction.

7. The method for identifying crop lodging areas by classification according to claim 1, characterized in that: The construction method of the LSC-ResUNet model is: Add the multi-attention mechanism module MAM and the adaptive loss function module AdaptiveLoss on the basis of ResUNet; The MAM module includes multiple multi-scale self-attention layers MSSA and reverse attention layers RA. The multi-scale self-attention layers MSSA are located at the end of the encoder and decoder respectively to guide the network to pay attention to the spatial importance features at different scales. The reverse attention layer RA is located at the end of each residual operation unit in the decoder to improve the recognition ability of blurred areas. Adaptive Loss is an improvement on Focal Loss. It combines the improved loss function with Dice Loss to replace the commonly used cross entropy. By changing the penalty intensity of different types of samples, it dynamically adjusts the optimization direction of the model prediction results to allow the network to develop in the direction of improving the classification accuracy of difficult samples.

8. The method for identifying crop lodging areas by classification according to claim 1, characterized in that: In step 2, the output structure of the LAR-ResUNet model and the encoder structure of the LSCResUNet model are improved, and then the lodging area information output by the LAR-ResUNet model in the first stage is coupled with the LSCResUNet model in the second stage in a layer-by-layer weighted form to obtain the crop lodging grading monitoring model LHMNet.

9. The method for identifying crop lodging areas by classification according to claim 8, characterized in that: Methods for improving the output structure of the LAR-ResUNet model and the encoder structure of the LSCResUNet model include: The output structure of the LAR-ResUNet model is improved by channel-superimposing the edge features and the encoder upsampled features, restoring the feature map size, and using 1×1 convolution to generate a feature map X with the same size as the input remote sensing image. S1 ; Use Sigmoid function for activation, and convert X S1 The output value on is mapped to the range of 0 to 1, and the probability of each point pixel belonging to the lodging and non-lodging area is represented in the channel dimension, and the pixel category probability map X is obtained. S1p ; Use continuously stacked 3×3 convolutions, batch normalization, and ReLU activation to extract probability features step by step, and obtain a probability feature map with the same size as the corresponding layer of LSC-ResUNet as the attention weight W of the lodging area L , and perform the above-mentioned probability feature extraction operation multiple times; The input structure of the LSC-ResUNet model is improved by replacing W L The feature X extracted from the same layer as the encoder of the LSC-ResUNet model S2 Multiply the elements at the corresponding positions to make the model pay more attention to the high probability areas given in the first stage and generate the weighted feature X of the lodging area S2p ; Then X S2p With the original feature X S2 Add the elements at the corresponding positions to obtain a new feature map X that integrates the probability features of the lodging area S2new , the calculation process is as follows: X S2new,ij =W L,ij X S2,ij +X S2,ij After the probability feature fusion is completed, X S2new The residual operation and feature downsampling are performed again in the encoder calculation process of the LSC-ResUNet model; the above-mentioned improvement operations are performed in each layer of the encoder of the LSC-ResUNet model.

10. A system for implementing the crop lodging area classification and identification method according to any one of claims 1 to 9, characterized in that: include: The data acquisition module is used to obtain remote sensing images of crop lodging in the study area, pre-process them, and then annotate the pre-processed remote sensing images to produce a data set; The model building module is used to build a LAR-ResUNet model for identifying the lodging area in the first stage and a LSC-ResUNet model for grading the severity of lodging in the second stage for the lodging area identified by the LAR-ResUNet model. The LAR-ResUNet model and the LSC-ResUNet model are then coupled through a weight fusion structure to obtain a two-stage crop lodging grading monitoring model LHMNet. The model training and prediction module is used to respectively train the LAR-ResUNet model and the LSC-ResUNet model in the crop lodging grading monitoring model LHMNet using the data set training to obtain the trained LAR-ResUNet model and the LSC-ResUNet model, and then use the trained crop lodging grading monitoring model LHMNet to predict the grading degree of crop lodging areas.

Images