Cultivated land extraction method based on efficient local attention mechanism and adversarial learning

Abstract

The invention discloses a cultivated land extraction method based on an efficient local attention mechanism and adversarial learning. The method comprises the following steps: S1, preparing a sample; carrying out preprocessing and blocking processing on the cultivated land image and the label data, and constructing a cultivated land sample library; s2, data enhancement; performing data enhancement on the cultivated land sample library, and dividing the enhanced cultivated land sample library into a training data set, a verification data set and a test data set in proportion; s3, an ELA (Extreme Laboratory Antigen)-DeepLabv < 3 + > network is constructed; inputting the training data set and the verification data set into the constructed ELA-DeepLabv3 + network for training, and inputting the test data set into the trained model to obtain a cultivated land extraction result predicted by the model; s4, constructing a GAN (Generic Area Network); and inputting a cultivated land extraction result and real cultivated land data into the constructed GAN network for adversarial learning training, and inputting a model prediction result into the trained GAN generative adversarial network to obtain an optimized cultivated land extraction result. The method is applied to the technical field of remote sensing image processing.

Description

Technical Field

[0001] The present invention pertains to the technical field of remote sensing image processing, and particularly relates to a cultivated land extraction method based on an efficient local attention mechanism and adversarial learning. Background Art

[0002] Cultivated land extraction is an important part of agricultural remote sensing. With the development of the aerospace industry and computer technology, the spatial resolution of remote sensing images has been continuously improved, providing data support for the transformation of the cultivated land extraction object in agricultural remote sensing from large-scale cultivated land in the past to more accurate cultivated land plots. Accurately extracting cultivated land can provide information such as the types, planting areas, and growth status of crops for agricultural production, and support decision-making in related fields such as land resource management, food production estimation, and non-grainization monitoring.

[0003] At present, the cultivated land extraction methods for high-resolution remote sensing images are mainly divided into two types. One is to rely on manual interpretation of remote sensing images for cultivated land extraction. This method has relatively accurate data, but it is costly and time-consuming, and is not suitable for large-scale extraction. The other is to use machine learning methods for cultivated land extraction. Traditional machine learning methods such as decision trees, support vector products, and random forests are used to classify based on pixel or object manual feature selection and classifier design. When the feature selection is appropriate, for simple classification tasks, the effect is good and it is easy to understand and implement. However, it is often limited by predefined rules and assumptions. In the face of the characteristics of high-resolution remote sensing images with high resolution and complex backgrounds, the extraction accuracy is often difficult to meet the actual application requirements.

[0004] While traditional unsupervised classification, supervised classification, and object-oriented classification machine learning methods have been continuously developing, using deep learning algorithms for cultivated land extraction shows application advantages and potential. Deep learning algorithms can automatically learn feature representations, capture local spatial correlations, and process complex image structures. The current cutting-edge technology is to introduce an attention mechanism on the basis of a neural network, such as the SE attention mechanism, the CA attention mechanism, and the CBAM attention mechanism. By using the attention mechanism to assign weights to features, the neural network focuses on certain feature channels, thereby further obtaining local features of the image. However, these methods also have problems such as failing to effectively utilize spatial information or sacrificing channel dimension information while using it, destroying the direct correspondence between channels and weights, resulting in the loss of dimension information, and being prone to boundary misclassification and edge blurring problems in complex and changeable agricultural environments. Summary of the Invention

[0005] To address the above deficiencies, the present invention proposes a cultivated land extraction method based on an efficient local attention mechanism and adversarial learning. By introducing the efficient local attention mechanism (ELA) to construct a network, accurate position predictions can be obtained in the spatial dimension without compromising the channel dimension of the input feature map. A generative adversarial network (GAN) is added for optimization to refine the results, generate more precise and realistic cultivated land boundaries, enhance the robustness of the model and the authenticity of the extraction results, and effectively solve the problems of boundary misclassification and edge blurring in the extraction results of classical deep learning models.

[0006] The technical solution adopted by the present invention is as follows: The present invention includes the following steps:

[0007] Step S1: Sample preparation; preprocess and block the cultivated land images and label data to construct a cultivated land sample library;

[0008] Step S2: Data augmentation; perform data augmentation on the cultivated land sample library, and divide the augmented cultivated land sample library into a training data set, a validation data set, and a test data set according to a certain proportion;

[0009] Step S3: Construction of the ELA-DeepLabv3+ network; based on the DeepLabv3+ model structure, insert the efficient local attention mechanism (ELA) module on the basis of the original encoder to construct the ELA-DeepLabv3+ network. Input the training data set and the validation data set into the constructed ELA-DeepLabv3+ network for training, and input the test data set into the trained model to obtain the cultivated land extraction result predicted by the model;

[0010] Step S4: Construction of the GAN network; input the cultivated land extraction result and the real cultivated land data into the constructed GAN network for adversarial learning training, and input the model prediction result into the trained GAN generative adversarial network to obtain the optimized cultivated land extraction result.

[0011] Further, in the step S1, preprocess the original cultivated land images. For the classification task of cultivated land extraction, obtain the spectral information of the red, green, blue, and near-infrared bands of the original cultivated land images, calculate the normalized difference vegetation index (NDVI), and generate 5-band images to enhance the model's ability to identify the spectral characteristics of cultivated land; the formula for the normalized difference vegetation index (NDVI) is:

[0012]

[0013] where NIR is the reflectance value of the near-infrared band and Red is the reflectance value of the red band.

[0014] Further, in the step S1, the preprocessed cultivated land image and the labeled data image are segmented by the sliding window method, and the predicted cultivated land image and the corresponding labeled data image are segmented into images of H×W×C, where H, W, and C are the number of rows, columns, and channels of the image, respectively.

[0015] Further, in the step S2, the data enhancement methods for the cultivated land sample library are image rotation, flipping, random scaling, and generative adversarial network enhancement to enhance the robustness of the model.

[0016] Further, in the efficient local attention mechanism ELA module, first, the input feature x c Performs strip-shaped average pooling operations on each channel in the horizontal and vertical directions, and performs average pooling in two spatial ranges, (H, 1) in the horizontal direction and (1, W) in the vertical direction, respectively, to obtain the output representations of the feature vector at height H and at width W. This process can be expressed as

[0017]

[0018] where H and W are the height and width of the image. x c (h, i) and x c (j, ω) represent the pixel values of the input feature x c at the h-th row and i-th column and at the j-th row and ω-th column. z h c (h) represents the average value calculated for the h-th row of the feature vector representation. z ω c (ω) represents the average value calculated for the ω-th column of the feature vector representation.

[0019] Further, the feature vectors in both directions are processed using convolution, group normalization, and non-linear activation functions to obtain the position attention features y h and y w . The two position attention features are fused with the input feature through a product operation to generate the final position attention feature Y. This process is expressed as

[0020] Y = x c × y h × y w = σ(GN(F h (z h ))) × σ(GN(F ω (z ω ))) (4)

[0021] where σ is the Sigmoid function, GN is the group normalization operation, and F his a 1D horizontal convolution operation, F ω is a 1D vertical operation, Z h and Z w are the feature vectors in the horizontal and vertical directions extracted in Equations 2 and 3.

[0022] Furthermore, the GAN network is used to optimize the cultivated land extraction results. The GAN network is constructed with the cultivated land prediction result map of ELA-DeepLabv3+ and the real cultivated land label as samples.

[0023] Furthermore, the GAN network includes a discriminator D and two generators G. Through the adversarial learning between the generator G and the discriminator D, the cultivated land extracted by the neural network is made to conform more to the texture characteristics of the real cultivated land through the adversarial learning mechanism, which is beneficial to improving the boundary clarity of the model prediction results and reducing the occurrence of edge blurring.

[0024] Furthermore, the two generators G are respectively generator G AB and generator G BA . The training process is as follows: the cultivated land result predicted by the input model is used, and the generator G AB generates a prediction result approximating the real cultivated land. The generator G BA performs reconstruction to ensure cycle consistency. Then, the real cultivated land sample and the prediction result approximating the real cultivated land are input into the discriminator D for judgment. If it fails, it is returned to the generator G, and the generator G AB , G BA are trained until it passes the judgment of the discriminator D. Then, the parameters of the generator G AB are fixed, and the discriminator D is trained, repeating until the adversarial learning effect is achieved.

[0025] Furthermore, the training process of the discriminator D can be expressed as

[0026] L GAN (G AB , D, A, B) = E b～B [log(D(b))] + E a～A [log(1 - D(G AB (a)))]

[0027] where A and B are two datasets input to the model, and G AB is the generator from dataset A to dataset B. L GAN (G AB , D, A, B) represents the GAN loss function for the generator G AB from dataset A to dataset B and the discriminator D. During the training process, the goal of the discriminator is to maximize the loss function L GAN (G AB, the value of D, A, B) to enhance the performance of the discriminator. E b～B [logD(b)] represents the confidence of judging the sample as true obtained by calculation after the sample b sampled from the dataset B passes through the discriminator D. E a～A log(1 - D(G AB (a))) represents the confidence of judging the input sample as false obtained by calculation after the discriminator processes the generated sample obtained by converting the sample a sampled from the dataset A through the generator G AB and then passing it through the discriminator D.

[0028] Furthermore, the training process of the generator G can be expressed as

[0029] L GAN (G AB , G BA , A, B) = E a～A [||G BA G AB (a) - a||]

[0030] where A and B are two datasets input to the model, G AB is the generator from dataset A to dataset B, and G BA is the generator from dataset B to dataset A. L GAN (G AB , G BA , A, B) is the cycle consistency loss function based on the generators G AB and G BA . During the training process, the goal of the generator G is to continuously minimize the loss function L GAN (G AB , D, A, B) to reduce the difference between the output after two conversions and the original input. E a～A [||(G BA G AB (a) - a||] represents the difference between the output after conversion through the generators G AB , G BA and the original sample a calculated through the norm.

[0031] The beneficial effects of the present invention are as follows: The present invention proposes a cultivated land extraction method based on an efficient local attention mechanism and adversarial learning. By constructing an ELA-DeepLabv3+ convolutional neural network and adding an efficient local attention mechanism ELA module, the representation ability of the convolutional neural network is enhanced, the spatial features and channel features of the extracted ground objects are fully obtained, the recognition ability of the model for cultivated land is improved, and the occurrence of boundary misclassification is reduced. Then, the GAN generative adversarial network is used for optimization. By obtaining the texture features of real cultivated land samples, the prediction result is made more consistent with real cultivated land through adversarial learning, and the boundary is clearer. Finally, the effects of reducing boundary misclassification and edge blurring and improving the prediction accuracy are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart of the present invention;

[0033] Figure 2 is a structural diagram of the ELA-DeepLabv3+ network framework proposed by the present invention;

[0034] Figure 3 is a structural diagram of the ELA efficient local attention mechanism provided by an embodiment of the present invention;

[0035] Figure 4 is a structural diagram of the GAN network provided by an embodiment of the present invention;

[0036] Figure 5 is a prediction result diagram of the GAN network provided by an embodiment of the present invention, where the left image represents the model extraction result and the right image represents the GAN network prediction result;

[0037] Figure 6 is a remote sensing image to be predicted provided by an embodiment of the present invention;

[0038] Figure 7 is a cultivated land extraction result diagram provided by an embodiment of the present invention, where the white part represents cultivated land and the black part represents the background. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] Combined with Figure 1 The specific steps of cultivated land extraction. The present invention can implement an automatic operation process using computer software. Based on the environment of python3.8, the present invention uses the Pytorch framework to implement the construction and training of the model. The technical solution of the present invention will be described in detail below with reference to the drawings and embodiments.

[0040] Figure 1 is a flowchart of a cultivated land extraction method based on an efficient local attention mechanism and adversarial learning proposed by the present invention. As Figure 1 shown, the cultivated land extraction method based on an efficient local attention mechanism and adversarial learning proposed by the present invention may include:

[0041] Step S1: Sample production. Extract the spectral information of the red, green, blue, and near-infrared bands from the original cultivated land image, calculate the normalized difference vegetation index (NDVI), synthesize the bands into a 5-band image, and the formula for calculating the normalized difference vegetation index (NDVI) is as follows:

[0042]

[0043] where NIR is the reflection value of the near-infrared band and Red is the reflection value of the red band.

[0044] Through the sliding window method, the cultivated land image after stitching the 5-band image and the label data image are segmented. The cultivated land image and the corresponding label data image are divided into multiple sample images, and the size of each cultivated land sample image is 512×512×5;

[0045] Step S2: To improve the generalization ability of the model and increase the number of samples, perform data augmentation operations such as image rotation, flipping, and random scaling on the segmented cultivated land sample images to expand the number of cultivated land samples. The enhanced cultivated land sample images are divided into a training data set and a validation data set at a ratio of 4:1;

[0046] Step S3: Based on the DeepLabv3+ model structure, DeepLabv3+ is an advanced deep convolutional neural network architecture mainly used for semantic segmentation tasks in computer vision. On the basis of the original encoder, an efficient local attention mechanism (ELA) module is inserted to construct the ELA-DeepLabv3+ network. The training data set and the validation data set are input into the backbone network of the constructed ELA-DeepLabv3+ network to extract deep semantic features; the extracted deep semantic features are input into the efficient local attention mechanism (ELA) module to enhance feature extraction and update the feature weights of each channel; the updated deep semantic features are input into the ASPP module to obtain multi-scale features for encoding, and finally, through upsampling and feature fusion, the classification result after attention enhancement is obtained. Among them, the ELA-DeepLabv3+ network is as Figure 2 shown, and the ELA efficient local attention mechanism is as Figure 3 shown;

[0047] Furthermore, the backbone network of the ELA-DeepLabv3+ network described in step S3 is ResNet101. For the input 5-band image, the number of input layer channels is adjusted to 5, and the normalization index is 0.5;

[0048] Furthermore, in the efficient local attention mechanism (ELA) module part of step S3, first, the input feature x cPerform strip-shaped average pooling operations to obtain feature vectors z in the horizontal and vertical directions respectively h 、z w , and this process can be expressed as

[0049]

[0050] Furthermore, use convolution, group normalization, and non-linear activation functions to process the feature vectors in both directions to obtain position attention features y in both directions h 、y w , and perform feature fusion on the two position attention features and the input features through a multiplication operation to generate the final position attention feature Y. This process can be expressed as

[0051] Y = x c ×y h ×y w =σ(GN(F h (z h ))×σ(GN(F ω (z ω ))) (4)

[0052] where σ is the Sigmoid function, also known as the logistic function, which is a mathematical function widely used in machine learning and deep learning; GN is the group normalization operation, F h , F ω are 1D horizontal and vertical convolution operations;

[0053] Step S4: Input the classification results extracted by the ELA-DeepLabv3+ network into a pre-trained GAN generative adversarial network to obtain the optimized cultivated land extraction results of the GAN network, as Figure 5 shown;

[0054] Among them, after obtaining the cultivated land results predicted by the ELA-DeepLabv3+ network, use the cultivated land prediction results and the real cultivated land label data as samples and input them into the GAN network for training to obtain a pre-trained GAN network cultivated land model; in addition, the GAN network includes a discriminator D and two generators G, and adversarial learning is carried out through the generator G and the discriminator D. Through the adversarial learning mechanism among them, the cultivated land extracted by the neural network is more in line with the texture characteristics of the real cultivated land features, which is beneficial to improving the boundary clarity of the model prediction results and reducing the generation of edge blurring phenomena; the two generators G are respectively generator G AB and generator G BA , the generator G AB is a one-way GAN from image A to image B, and the generator G BAIt is a one-way GAN from image B to image A. The training process is to input the cultivated land results predicted by the model, and through the generator G AB generate predicted results that approximate real cultivated land. The generator G BA performs reconstruction to ensure cycle consistency, and then inputs real cultivated land samples and predicted results of approximate real cultivated land into the discriminator D for judgment. If it fails, it returns to the generator G AB In it, for the generator G AB is trained until it passes the judgment of the discriminator D, and then the parameters of the generator G are fixed AB , and the discriminator D is trained, repeating to achieve the effect of adversarial learning; the training process of the discriminator D can be expressed as

[0055] L GAN (G AB ,D,A,B)=E bb～B [log(D(b))]+E a～A [log(1-D(G AB (a)))] (5)

[0056] Among them, A and B are two data sets input by the model, and G AB is the generator from data set A to data set B. L GAN (G AB ,D,A,B) represents the GAN loss function for the generator G AB from data set A to data set B and the discriminator D. During the training process, the goal of the discriminator is to continuously iterate to maximize the value of the loss function L GAN (G AB ,D,A,B) to enhance the performance of the discriminator. E b～B [logD(b)] represents the confidence of judging the sample as true obtained by calculating after the sample b sampled from data set B passes through the discriminator D. E a～A log(1-D(G AB (a))) represents the confidence of judging the input sample as false obtained by the discriminator after calculating the generated sample obtained by converting the sample a sampled from data set A after passing through the generator G AB .

[0057] The training process of the generator G can be expressed as

[0058] L GAN (G AB ,G BA ,A,B)=E a～A [||G BA G AB (a)-a||] (6)

[0059] Among them, A and B are two data sets input to the model, and G AB is a generator from data set A to data set B, and G BA is a generator from data set B to data set A. L GAN (G AB , G BA , A, B) is a cycle consistency loss function based on generators G AB and G BA . During the training process, the goal of generator G is to continuously iterate to minimize the loss function L GAN (G AB , D, A, B), and reduce the difference between the output after two conversions and the original input. E a～A [||(G BA G AB (a)-a||] represents the difference between the output after conversion by generators G AB , G BA and the original sample a calculated through the norm.

[0060] Finally, the cultivated land image to be predicted is input into the ELA-DeepLabv3+ network to obtain the cultivated land prediction result, and the cultivated land extraction result of the image to be predicted is obtained by inputting it into the GAN network. The image to be predicted is as shown in Figure 6 , and the cultivated land extraction result is as shown in Figure 7 ;

[0061] Although the embodiments of the present invention are described with actual solutions, they do not constitute a limitation to the meaning of the present invention. For those skilled in the art, modifications to its implementation solutions according to this specification and combinations with other solutions are obvious.

Claims

1. A method for farmland extraction based on efficient local attention mechanism and adversarial learning, characterized in that: It includes the following steps: Step S1: Sample preparation: pre-processing and block processing of cultivated land images and label data to build a cultivated land sample library; Step S2: data enhancement; Perform data enhancement on the cultivated land sample library, and divide the enhanced cultivated land sample library into training data set, verification data set and test data set in proportion; Step S3: ELA-DeepLabv3+ network construction; Based on the DeepLabv3+ model structure, on the basis of the original encoder, the efficient local attention mechanism ELA module is inserted to construct the ELA-DeepLabv3+ network. The training data set and the verification data set are input into the constructed ELA-DeepLabv3+ network for training. The test data set is input into the trained model to obtain the cultivated land extraction results predicted by the model. Step S4: GAN network construction; The cultivated land extraction results and the real cultivated land data are input into the constructed GAN network for adversarial learning training, and the model prediction results are input into the trained GAN generative adversarial network to obtain the optimized cultivated land extraction results.

2. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 1, characterized in that: In the step S1, the original cultivated land image is preprocessed, and for the classification task of cultivated land extraction, the spectral information of the red, green, blue and near-infrared bands of the original cultivated land image is obtained, the normalized vegetation index NDVI is calculated, and a five-band image is generated to enhance the model's ability to recognize the spectral characteristics of cultivated land; The normalized difference vegetation index NDVI formula is: Among them, NIR is the reflection value of the near-infrared band, and Red is the reflection value of the red light band.

3. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 1, characterized in that: In the step S1, the preprocessed cultivated land image and the label data image are block processed by a sliding window method, and the predicted cultivated land image and the corresponding label data image are divided into H×W×C images, wherein H, W, and C are the number of rows, columns, and channels of the image, respectively.

4. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 1, characterized in that: In step S2, the data enhancement method for the cultivated land sample library is image rotation, flipping, and scaling to enhance the robustness of the model.

5. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 3, characterized in that: The ELA module of the efficient local attention mechanism first converts the input feature x c Perform strip-like average pooling operations on each channel in the horizontal and vertical directions, and perform average pooling in two spatial ranges, (H, 1) in the horizontal direction and (1, W) in the vertical direction, respectively obtaining the output representation of the feature vector at the height H and the output representation at the width W. The process can be expressed as Among them, H and W are the height and width of the image; x c (h,i) and x c (j,ω) represents the input feature x c The pixel value at the hth row, ith column and the jth row, ωth column; z h c (h) represents the average value of the feature vector calculated for the hth row; z ω c (ω) represents the average value of the eigenvector calculated for the ωth column; Use convolution, group normalization and nonlinear activation function to process the feature vectors in the horizontal and vertical directions to obtain the position attention features y in both directions h ,y w , the two position attention features are multiplied with the input feature x c Perform feature fusion to generate the final position attention feature Y. The process is expressed as Y=x c ×y h ×y w =σ(GN(F h (With h ))×σ(GN(F ω (With ω ))) (4) Among them, σ is the Sigmoid function, GN is the group normalization operation, and F h is a 1-dimensional horizontal convolution operation, F ω is a 1-dimensional vertical operation, Z h and Z w are the horizontal and vertical feature vectors extracted from formulas (2) and (3).

6. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 1, characterized in that: The GAN network is used to optimize the cultivated land extraction results, and the GAN network is constructed using the cultivated land prediction result map of ELA-DeepLabv3+ and the real cultivated land labels as samples.

7. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 1, characterized in that: The GAN network consists of a discriminator D and two generators G. Adversarial learning is performed through the generator G and the discriminator D. Through the adversarial learning mechanism, the texture features of the cultivated land extracted by the neural network are more consistent with the real cultivated land characteristics, which is beneficial to improve the boundary clarity of the model prediction results and reduce the occurrence of edge blur.

8. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 7, characterized in that: The two generators G are respectively generator G AB and the generator G BA The training process is to input the farmland results predicted by the model through the generator G AB Generate prediction results that are close to the real farmland, by the generator G BA Reconstruction is performed to ensure cycle consistency, and then the real farmland samples and the prediction results of approximate real farmland are input into the discriminator D for judgment. If it fails, it is returned to the generator G. AB , G BA Train until it passes the judgment of the discriminator D, and then fix the generator G AB , G BA Parameters, train the discriminator D, and repeat to achieve the effect of adversarial learning.

9. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 7, characterized in that: The training process of the discriminator D can be expressed as L GAN (G AB ,D,A,B)(E b～B [log(D(b))]+E a～A [log(1-D(GAB(a)))] (5) Among them, A and B are two data sets input to the model, G AB is the generator from dataset A to dataset B; L GAN (G AB ,D,A,B) represents the generator G from data set A to data set B AB And the GAN loss function of the discriminator D. During the training process, the goal of the discriminator is to maximize the loss function L through continuous iteration GAN (G AB ,D,A,B) values, enhancing the performance of the discriminator; E b～B [logD(b)] represents the confidence that the sample b sampled from the data set B is true after passing through the discriminator D; E a～A log(1-D(G AB (a))) indicates that sample a sampled from dataset A passes through generator G AB After that, the generated sample obtained after conversion passes through the discriminator D, and the discriminator calculates the confidence that the input sample is false.

10. The method for extracting cultivated land based on efficient local attention mechanism and adversarial learning according to claim 7, characterized in that: The training process of the generator G can be expressed as L GAN (G AB ,G BA ,A,B)=E a～A [||G BA G AB (a)-a||] (6) Among them, A and B are two data sets input to the model, G AB is the generator from dataset A to dataset B, G BA is the generator from dataset B to dataset A; L GAN (G AB ,G BA ,A,B) is based on the generator G AB and G BA The cycle consistency loss function; during the training process, the goal of the generator G is to minimize the loss function L through continuous iteration GAN (G AB ,D,A,B), reducing the difference between the output after two conversions and the original input; E a～A [||(G BA G AB (a)-a||] means after the generator G AB , G BA The transformed output is normed to calculate the difference between it and the original sample a.

Images