A method, system, device and medium for farmland extraction from high-resolution SAR images based on Transformer multi-scale feature fusion

Abstract

A method, system, device and medium for extracting farmland from high-resolution SAR images based on Transformer multi-scale feature fusion, the method comprising: data acquisition, data enhancement, network training, result prediction and performance evaluation; the system, device and medium are used to implement a method for extracting farmland from high-resolution SAR images based on Transformer multi-scale feature fusion; the present invention solves problems such as a small number of training samples and low annotation quality by using a multi-scale feature extraction method and a data enhancement strategy suitable for semantic segmentation of farmland in SAR images, thereby improving the performance of the model in supervised farmland extraction and its generalization performance.

Description

Technical Field

[0001] The present invention relates to the technical field of SAR image interpretation, and in particular to a method, system, equipment and medium for extracting farmland from high-resolution SAR images based on Transformer multi-scale feature fusion. Background Art

[0002] Agriculture is one of the most important areas for remote sensing applications and research. Remote sensing provides timely access to extensive information on crop growth dynamics. Currently, crop area estimation, growth monitoring, and yield forecasting are still primarily based on optical remote sensing, as optical imagery is easily interpretable, especially for dryland crops. However, in practical agricultural remote sensing applications, obtaining high-quality optical remote sensing data is often difficult, despite the many challenging weather conditions.

[0003] Synthetic Aperture Radar (SAR) is a high-resolution imaging radar. Due to the cloud-penetrating properties of microwaves, SAR can acquire cloud-free imagery in all weather conditions. It is also capable of nighttime operation, enabling all-day, all-weather Earth observation regardless of lighting or weather conditions. It can even penetrate the ground surface or vegetation to obtain information obscured by it. SAR is sensitive to the dielectric and geometric characteristics of plants and has high penetrating power. Depending on the frequency, it can obtain information below the vegetation canopy cover. Due to these characteristics and advantages, SAR has considerable potential in agricultural remote sensing, where crop location and distribution are essential in many applications.

[0004] Deep learning-based semantic segmentation technology has achieved remarkable results in the field of SAR imagery object classification. However, obtaining accurate predictions remains a significant challenge, facing challenges such as the time-consuming and labor-intensive manual annotation of SAR images, low annotation quality, and feature variations between satellite images. Feature extraction is a crucial step in semantic segmentation, and extracting key and effective farmland features is a key factor influencing model performance. Traditional deep learning-based feature extraction requires a large number of training samples, from which the network extracts the required features for classification.

[0005] Most existing methods for extracting farmland from SAR images are based on convolutional neural networks (CNNs), which have achieved significant results. Offshore farmland segmentation from SAR images based on UNet++ (YU C, LIU Y, XIA X. Precise segmentation of offshore farms in high-resolution SAR images based on improved UNet++ [C / OL] / / International Conference on Computer Application and Information Security (ICCAIS 2021), Wuhan, China. 2022. http: / / dx.doi.org / 10.1117 / 12.2637380. DOI: 10.1117 / 12.2637380.) improves the accuracy of farmland extraction by improving the network structure and adding auxiliary strategies. However, this method requires a large number of SAR image labels, has weak generalization capabilities, and is susceptible to interference from SAR image noise, which reduces the accuracy of farmland extraction. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, equipment and medium for farmland extraction from high-resolution SAR images based on Transformer multi-scale feature fusion. By using a multi-scale feature extraction method and a data enhancement strategy suitable for SAR image farmland semantic segmentation, the problems of insufficient training samples and low annotation quality are solved, and the performance of the model in supervised farmland extraction and its generalization performance are improved.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A farmland extraction method based on high-resolution SAR imagery using Transformer multi-scale feature fusion includes the following steps:

[0009] Step 1, data acquisition: SAR images of similar resolutions with different imaging parameters and different imaging equipment under different scenes are collected, wherein the SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas; the farmland coverage area images of each scene are cropped to obtain image blocks, and the same number of image blocks are randomly selected as samples to form a training data set;

[0010] Step 2, data enhancement: set a label for each image block in the training data set in step 1, calculate the proportion of farmland in each image block, and perform data enhancement on image blocks where the proportion of farmland exceeds the set threshold;

[0011] Step 3, network training: Input the image block after data enhancement in step 2 into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; input the extracted multi-scale features of the image block into the decoder network D for training to obtain the prediction result graph, and update the encoder network and decoder network parameters at the same time until the training termination condition is met to obtain a trained network model;

[0012] Step 4, result prediction: Collect the SAR image to be predicted, perform cropping, label setting, and data enhancement on the SAR image to be predicted in sequence to obtain the image block to be predicted, input the image block to be predicted into the trained network model for prediction, and obtain the farmland prediction result map;

[0013] Step 5, performance evaluation: Use the farmland prediction result map obtained in step 4 and the label map of the SAR image to be predicted to calculate the category evaluation index and overall evaluation index to evaluate the network performance.

[0014] The specific process of step 2 is:

[0015] Step 2.1: set a label for each image block in the training dataset and calculate the number of farmland pixels in each image block label;

[0016] Step 2.2: Divide the number of farmland pixels in each image block label calculated in step 2.1 by the total number of pixels in the corresponding image block to obtain the farmland proportion of each image block.

[0017] Step 2.3, set the farmland ratio threshold;

[0018] In step 2.4, the farmland ratio of each image block in step 2.2 is compared with the farmland ratio threshold in step 2.3, and data enhancement is performed on the image blocks whose farmland ratio exceeds the threshold, wherein the data enhancement includes random contrast and random hue.

[0019] The multi-scale feature extraction encoder network E in step 3 is composed of four Transformer modules;

[0020] The first two Transformer modules are used to extract high-resolution coarse-grained features of image patches;

[0021] The last two Transformer modules are used to extract high-resolution fine-grained features of image patches.

[0022] The Transformer module is composed of an overlapping patch merging module, a self-attention module, and a hybrid FFN module connected in sequence;

[0023] Overlapping patch merging module: used to divide the input image blocks and obtain feature maps of multiple patches. It can merge overlapping patches by setting the patch size, stride size between adjacent patches, and padding size.

[0024] Self-attention module: used to perform deeper feature extraction on the feature maps of multiple patches to obtain high-level feature maps;

[0025] Hybrid FFN module: used to provide position encoding information for the Transformer module, consisting of a 3*3 deep convolutional layer and an MLP layer.

[0026] The decoder network D is composed of an MLP module and an ASPP module. The training process of the decoder network D is as follows:

[0027] Step 3.11: Input the multi-scale features extracted by the multi-scale feature extraction encoder network E into different MLP layers to unify the channel dimension;

[0028] Step 3.12: Input the features after unifying the channel dimensions in step 3.11 into four deep ASPP modules with different expansion rates for decoding;

[0029] In step 3.13, the features decoded in step 3.12 are input into the convolutional layer for feature fusion, segmentation mask prediction, and prediction result graph.

[0030] The category evaluation indicators in step 5 include:

[0031] Precision:

[0032]

[0033] In the formula, i represents a positive example; j represents a negative example; P represents the precision rate, that is, the proportion of correct positive predictions to all positive predictions; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP;

[0034] Intersection over Union (IoU):

[0035]

[0036] Where i represents a positive example; j represents a negative example; IoU represents the intersection over union ratio, which is the ratio of the intersection and union of each type of prediction result and the true value; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij represents the total number of pixels whose true category is j and identified as category i, i.e. false positive FP; pji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j;

[0037] Recall:

[0038]

[0039] In the formula, i represents a positive example; j represents a negative example; R represents the recall rate, that is, the proportion of correctly predicted positive samples to all positive samples; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j;

[0040] F1score value:

[0041]

[0042] Where F1 represents the F1score value, which is the harmonic mean of recall and precision; R represents the recall, which is the proportion of correctly predicted positive samples to all positive samples; P represents the precision, which is the proportion of correctly predicted positive samples to all positive samples.

[0043] The overall evaluation indicators in step 5 include:

[0044] Overall accuracy OA:

[0045]

[0046] Where i represents a positive example; j represents a negative example; OA represents the overall accuracy, that is, the proportion of correctly labeled pixels to the total pixels; p ii The total number of pixels whose true category is i and identified as category i is the true positive case TP;

[0047] Kappa coefficient:

[0048]

[0049] In the formula, kappa represents the kappa coefficient, which means that the "bias" of the penalty model is used to obtain a more fair model; p o represents the proportion of the number of correctly classified samples of each category to the total number of samples, equivalent to OA; a1, a2, ..., aC represent the number of real samples of each category; b1, b2, ..., bC represent the predicted number of samples of each category; the number of categories is C, the total number of samples is n; p e Represents the sum of the products of the actual and predicted numbers of all categories divided by the square of the total number of samples;

[0050] Average Intersection-over-Union (MIoU):

[0051]

[0052] Where i represents a positive example; j represents a negative example; MIoU represents the average intersection-over-union ratio, which is the average sum of the ratios of the intersection and union of each type of prediction result and the true value; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP;

[0053] Frequency Weighted Intersection over Union (FWIoU):

[0054]

[0055] Where i represents a positive example; j represents a negative example; FWIoU represents the frequency-weighted intersection-over-union ratio, which is an improvement of MIoU, setting weights according to the frequency of occurrence of the category; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij It represents the total number of pixels whose true category is j and identified as category i, that is, false positive FP.

[0056] A farmland extraction system for high-resolution SAR images based on Transformer multi-scale feature fusion, including:

[0057] Data acquisition module: collects SAR images of similar resolution with different imaging parameters and different imaging equipment in different scenes. The SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas. The farmland coverage area images of each scene are cropped to obtain image blocks, and the same number of image blocks are randomly selected as samples to form a training data set.

[0058] Data augmentation module: sets a label for each image block in the training dataset, calculates the proportion of farmland in each image block, and performs data augmentation on image blocks where the proportion of farmland exceeds a set threshold;

[0059] Network training module: The image block after data augmentation is input into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; the extracted multi-scale features of the image block are input into the decoder network D for training to obtain the prediction result graph, and the encoder network and decoder network parameters are updated at the same time until the training termination condition is met to obtain a trained network model;

[0060] Result prediction module: collects the SAR image to be predicted, performs cropping, label setting, and data enhancement on the SAR image to be predicted, obtains the image block to be predicted, inputs the image block to be predicted into the trained network model for prediction, and obtains the farmland prediction result map;

[0061] Performance evaluation module: Use the farmland prediction result map and the label map of the SAR image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance.

[0062] A farmland extraction device based on high-resolution SAR imagery using Transformer multi-scale feature fusion, including:

[0063] Memory: used to store a computer program for implementing the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion;

[0064] Processor: used to implement the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion when executing the computer program.

[0065] A computer-readable storage medium:

[0066] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a farmland extraction method for high-resolution SAR images based on Transformer multi-scale feature fusion.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention designs a data enhancement strategy suitable for farmland extraction. By calculating the proportion of farmland in the training samples and setting a threshold, data enhancement is performed on training samples with a large proportion of farmland. Compared with the existing technology, the present invention can increase the supervised farmland semantic segmentation result indicators and improve the generalization performance of the model.

[0069] 2. The present invention designs a multi-scale feature extraction encoder based on Transformer to extract high-resolution coarse-grained features and low-resolution fine-grained features. It can extract more useful information from limited training samples and reduce the number of layers of the coarse-grained feature extraction module to reduce the impact of noise information unique to SAR images on model performance.

[0070] 3. The present invention designs and uses a decoder suitable for farmland semantic segmentation to better utilize the extracted features and alleviate the defect of model performance degradation caused by sample differences. Compared with the existing technology, the generalization ability of the model is improved.

[0071] In summary, the present invention solves the problems of insufficient training samples and low annotation quality by using a multi-scale feature extraction method and a data enhancement strategy suitable for semantic segmentation of farmland in SAR images, thereby improving the performance of the model in supervised farmland extraction and its generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a flow chart of the method of the present invention.

[0073] Figure 2 The present invention is a flowchart of a method for extracting farmland from SAR images.

[0074] Figure 3 This is the SAR image of the Bavaria region in the test set of this invention.

[0075] Figure 4 It is the SAR image of the PoDelta area in the training set of the present invention.

[0076] Figure 5 It is the SAR image of the Rosenheim area in the training set of the present invention.

[0077] Figure 6 It is the SAR image of Jiujiang area in the training set of the present invention.

[0078] Figure 7 It is the prediction map of the SAR image of the PoDelta area in the present invention.

[0079] Figure 8 This is the predicted image of the SAR image of the Rosenheim area in the present invention.

[0080] Figure 9 It is the prediction map of the SAR image of the Jiujiang area in the present invention.

[0081] Figure 10 This is the prediction map of the present invention using the SAR image of the Bavaria region.

[0082] Figure 11 This is the prediction graph of the SAR image of the Bavaria region using a convolutional neural network in the present invention. DETAILED DESCRIPTION

[0083] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and simulations.

[0084] See also Figure 1 and Figure 2 A farmland extraction method based on high-resolution SAR imagery and Transformer multi-scale feature fusion is proposed, comprising the following steps:

[0085] Step 1, data acquisition: Collect SAR images of similar resolution with different imaging parameters and different imaging equipment in different scenes. The SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas. The farmland coverage area images of each scene are cropped to obtain image blocks. The same number of image blocks are randomly selected as samples to form a training data set, specifically:

[0086] Input the SAR images required for the experiment. Select three SAR images of similar resolution from different regions, imaging parameters, and imaging equipment: PoDelta, Rosenheim2, and JiuJiang. Randomly select the same number of samples from the three images. Each sample is a 512×512 image block to form the entire task training dataset.

[0087] Step 2, data enhancement: set a label for each image block in the training data set in step 1, calculate the proportion of farmland in each image block, and perform data enhancement on image blocks where the proportion of farmland exceeds the set threshold;

[0088] Step 3, network training: Input the image block after data enhancement in step 2 into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; input the extracted multi-scale features of the image block into the decoder network D for training to obtain the prediction result graph, and update the encoder network and decoder network parameters at the same time until the training termination condition is met to obtain a trained network model;

[0089] Step 4, result prediction: Collect the SAR image to be predicted, perform cropping, label setting, and data enhancement on the SAR image to be predicted in sequence to obtain the image block to be predicted, input the image block to be predicted into the trained network model for prediction, and obtain the farmland prediction result map;

[0090] Step 5, performance evaluation: Use the farmland prediction result map obtained in step 4 and the label map of the SAR image to be predicted to calculate the category evaluation index and overall evaluation index to evaluate the network performance.

[0091] The specific process of step 2 is:

[0092] Step 2.1: set a label for each image block in the training dataset and calculate the number of farmland pixels in each image block label;

[0093] Step 2.2: Divide the number of farmland pixels in each image block label calculated in step 2.1 by the total number of pixels in the corresponding image block to obtain the farmland proportion of each image block.

[0094] Step 2.3, set the farmland ratio threshold;

[0095] In step 2.4, the farmland ratio of each image block in step 2.2 is compared with the farmland ratio threshold in step 2.3, and data enhancement is performed on the image blocks whose farmland ratio exceeds the threshold, wherein the data enhancement includes random contrast and random hue.

[0096] This method can increase the supervised farmland semantic segmentation result indicators and improve the generalization of the model for farmland.

[0097] The multi-scale feature extraction encoder network E in step 3 is composed of four Transformer modules;

[0098] The first two Transformer modules are used to extract high-resolution coarse-grained features of image blocks. The number of layers of this module is set to 1. Reducing the number of layers of this module can effectively reduce the impact of the original noise of the SAR image on feature extraction.

[0099] The last two Transformer modules are used to extract high-resolution fine-grained features of image blocks, increase the receptive field, and improve the model's feature extraction capabilities.

[0100] The Transformer module is composed of an overlapping patch merging module, a self-attention module, and a hybrid FFN module connected in sequence:

[0101] Overlapping patch merging module: used to divide the input image blocks and obtain feature maps of multiple patches. It can merge overlapping patches by setting the patch size, stride size between adjacent patches, and padding size.

[0102] Self-attention module: used to perform deeper feature extraction on the feature maps of multiple patches to obtain high-level feature maps;

[0103] Traditional self-attention modules have high complexity. However, by reducing their sequence length, the complexity can be effectively reduced and the training time can be shortened without affecting feature extraction.

[0104] Hybrid FFN module: used to provide position encoding information for the Transformer module. It consists of a 3*3 deep convolution layer and an MLP layer. This module provides position encoding information for the Transformer module and does not affect the accuracy.

[0105] The decoder network D is composed of an MLP module and an ASPP module. The training process of the decoder network D is as follows:

[0106] Step 3.11: Input the multi-scale features extracted by the multi-scale feature extraction encoder network E into different MLP layers to unify the channel dimension;

[0107] Step 3.12: Input the features after unifying the channel dimensions in step 3.11 into four deep ASPP modules with different expansion rates for decoding;

[0108] In step 3.13, the features decoded in step 3.12 are input into the convolutional layer for feature fusion, segmentation mask prediction, and prediction result graph.

[0109] The step 4 is specifically as follows:

[0110] Because SAR imagery is not evenly divisible by 512*512, the right and bottom borders of the entire image are cropped with overlapping patches. The rest of the image is cropped with non-overlapping patches of 512*512. During testing, the cropped test patches were fed into the trained model to obtain all prediction results. When stitching together the full image, the overlapping prediction results for the right and bottom borders were discarded, and the final image was stitched together to create a prediction result image of the same size as the original SAR image.

[0111] The category evaluation indicators in step 5 include:

[0112] Precision:

[0113]

[0114] In the formula, i represents a positive example; j represents a negative example; P represents the precision rate, that is, the proportion of correct positive predictions to all positive predictions; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP;

[0115] Intersection over Union (IoU):

[0116]

[0117] Where i represents a positive example; j represents a negative example; IoU represents the intersection over union ratio, which is the ratio of the intersection and union of each type of prediction result and the true value; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij represents the total number of pixels whose true category is j and identified as category i, i.e. false positive FP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j;

[0118] Recall:

[0119]

[0120] In the formula, i represents a positive example; j represents a negative example; R represents the recall rate, that is, the proportion of correctly predicted positive samples to all positive samples; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j;

[0121] F1score value: based on the harmonic mean of recall and precision, defined as follows:

[0122]

[0123] Where F1 represents the F1score value, which is the harmonic mean of recall and precision; R represents the recall, which is the proportion of correctly predicted positive samples to all positive samples; P represents the precision, which is the proportion of correctly predicted positive samples to all positive samples.

[0124] The overall evaluation indicators in step 5 include:

[0125] Overall accuracy OA:

[0126]

[0127] Where i represents a positive example; j represents a negative example; OA represents the overall accuracy, that is, the proportion of correctly labeled pixels to the total pixels; p ii The total number of pixels whose true category is i and identified as category i is the true positive case TP;

[0128] Kappa coefficient:

[0129]

[0130] In the formula, kappa represents the kappa coefficient, which means that the "bias" of the penalty model is used to obtain a more fair model; p o represents the proportion of the number of correctly classified samples of each category to the total number of samples, equivalent to OA; a1, a2, ..., aC represent the number of real samples of each category; b1, b2, ..., bC represent the predicted number of samples of each category; the number of categories is C, the total number of samples is n; p e Represents the sum of the products of the actual and predicted numbers of all categories divided by the square of the total number of samples;

[0131] Average Intersection-over-Union (MIoU):

[0132]

[0133] Where i represents a positive example; j represents a negative example; MIoU represents the average intersection-over-union ratio, which is the average sum of the ratios of the intersection and union of each type of prediction result and the true value; p iirepresents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP;

[0134] Frequency Weighted Intersection over Union (FWIoU):

[0135]

[0136] Where i represents a positive example; j represents a negative example; FWIoU represents the frequency-weighted intersection-over-union ratio, which is an improvement of MIoU, setting weights according to the frequency of occurrence of the category; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij It represents the total number of pixels whose true category is j and identified as category i, that is, false positive FP.

[0137] A farmland extraction system for high-resolution SAR images based on Transformer multi-scale feature fusion, including:

[0138] Data acquisition module: Collect SAR images of similar resolution with different imaging parameters and different imaging equipment in different scenes. The SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas. The farmland coverage area images of each scene are cropped to obtain image blocks, and the same number of image blocks are randomly selected as samples to form a training data set. This module corresponds to step 1.

[0139] Data augmentation module: Set a label for each image block in the training dataset, calculate the proportion of farmland in each image block, and perform data augmentation on image blocks where the proportion of farmland exceeds a set threshold. This module corresponds to step 2.

[0140] Network training module: Input the data-enhanced image block into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; input the extracted multi-scale features of the image block into the decoder network D for training to obtain the prediction result graph, and update the encoder network and decoder network parameters at the same time until the training termination condition is met to obtain a trained network model. This module corresponds to step 3;

[0141] Result prediction module: collects the SAR image to be predicted, performs cropping, label setting, and data enhancement on the SAR image to be predicted, obtains the image block to be predicted, inputs the image block to be predicted into the trained network model for prediction, and obtains the farmland prediction result map. This module corresponds to step 4;

[0142] Performance evaluation module: Use the farmland prediction result map and the label map of the SAR image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance. This module corresponds to step 5.

[0143] A farmland extraction device based on high-resolution SAR imagery using Transformer multi-scale feature fusion, including:

[0144] Memory: used to store a computer program for implementing the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion;

[0145] Processor: used to implement the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion when executing the computer program.

[0146] A computer-readable storage medium:

[0147] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a farmland extraction method for high-resolution SAR images based on Transformer multi-scale feature fusion.

[0148] The application effect of the present invention is described in detail below with reference to simulation.

[0149] 1. Experimental conditions and methods

[0150] The hardware platform is: GTX 2080Ti 11G, 256GB RAM;

[0151] The software platform is: Ubuntu 18.04.6LTS, PyTorch 1.7.1;

[0152] Experimental methods: supervised CNN method and the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion of the present invention.

[0153] 2. Simulation content and results

[0154] Figure 8 This is the prediction map of the SAR image of the Rosenheim area;

[0155] Figure 9 It is the prediction map of the SAR image of the Jiujiang area;

[0156] Figure 10 This is a prediction map of the Bavaria region using the SAR image of the present invention.

[0157] The detailed data parameters used in the simulation experiment of the present invention are shown in Table 1 below.

[0158] Table 1 Detailed parameters of the data used in the present invention

[0159]

[0160]

[0161] Table 2 Regional category legend

[0162]

[0163] Randomly select the same number of samples from the first three scenes (PoDelta, Rosenheim, JiuJiang) in Table 1, each sample is a 512×512 image block, and form the entire task training dataset. Select the SAR image of the four scenes as the test image, such as Figure 4 、 Figure 5 、 Figure 6 As shown in the figure, the large image is cropped into several 512*512 size image blocks for testing.

[0164] The whole SAR images of three scenes from the same source as the training set are used as the test set to test the effectiveness of the supervised model of the invention; the SAR images of the Bavaria region that are not in the training set are used as the test set to detect the improvement effect of the invention on the generalization ability of the model. Figure 3 shown.

[0165] The semantic segmentation results of farmland in SAR images of three scenes, PoDelta, Rosenheim, and JiuJiang, are shown in the figure below. Figure 7 、 Figure 8 、 Figure 9 The evaluation results are shown in Tables 3, 4, and 5, which show the evaluation indicators of the semantic segmentation results of farmland in the entire SAR image of the three scenes of PoDelta, Rosenheim, and JiuJiang.

[0166] The semantic segmentation results of farmland in SAR images of Bavaria are as follows: Figure 7 The evaluation results are shown in Table 6, which shows the evaluation indicators of the semantic segmentation results of the entire SAR image of farmland in the Bavaria region.

[0167] Model_1 is the method proposed by the present invention, and Model_2 is the supervised convolutional neural network (CNN) method. Both models use the same SAR image blocks as training data to demonstrate the effectiveness of the invention.

[0168] Table 3 Comparison of semantic segmentation results of farmland images in the PoDelta region

[0169] Evaluation indicators OA Kappa MIoU FWU Model_1 0.9677 0.8989 0.9054 0.9386 Model_2 0.9669 0.8987 0.851 0.9126

[0170] Table 4 Comparison of semantic segmentation results of farmland images in the Rosenheim area

[0171] Evaluation indicators OA Kappa MIoU FWU Model_1 0.9364 0.8694 0.8776 0.8805 Model_2 0.9026 0.7938 0.7735 0.7862

[0172] Table 5 Comparison table of semantic segmentation results of farmland images in Jiujiang area

[0173] Evaluation indicators OA Kappa MIoU FWU Model_1 0.9072 0.7743 0.8005 0.8319 Model_2 0.8653 0.6694 0.6157 0.6734

[0174] Table 6 Comparison of semantic segmentation results of farmland images in Bavaria region

[0175] Evaluation indicators OA Kappa MIoU FWU Model_1 0.9456 0.8483 0.8615 0.9002 Model_2 0.9145 0.7657 0.6962 0.7952

[0176] 3. Analysis of experimental results

[0177] According to the indicators shown in Tables 3, 4, 5 and 6, the four evaluation indicators of the model, OA, kappa coefficient, MIoU and FWIoU, are improved compared with the supervised model. The method of the present invention has a certain improvement in each evaluation indicator compared with the comparative experiment. The overall accuracy (OA) is improved by about 3%, the kappa coefficient for consistency test is improved by about 8%, the average intersection over union (MIoU) is improved by 5%-19%, and the frequency-weighted intersection over union (FWIoU) is also improved by 2%-15%.

[0178] Judging from the indicators in Tables 3, 4, and 5, the classification performance of the present invention is greatly improved compared with the supervised method of convolutional neural network (CNN); judging from the indicators in Table 6, the generalization performance of the present invention is greatly improved compared with the supervised method of convolutional neural network (CNN).

[0179] from Figure 7 、 Figure 8 、 Figure 9 From the result graph, the farmland classification of the present invention is more complete and has fewer miscellaneous points; Figure 10 and Figure 11 For comparison: Figure 11 There are many misclassifications, many noise points, and poor local details of the classification results; Figure 10 This is the experimental result of the present invention, from which it can be seen that there are fewer misclassifications, the edge details are clearer, and there are fewer noises.

[0180] In summary, the present invention proposes a farmland extraction method for high-resolution SAR images based on Transformer multi-scale feature fusion. The Transformer multi-scale feature extraction method can better extract useful information in the training set, reduce noise information in SAR images, and achieve effective feature extraction. The use of a decoder suitable for farmland feature segmentation can better utilize the extracted features, compensate for the defect of model performance degradation caused by sample differences, and improve the classification performance and generalization ability of the model.

Claims

1. A farmland extraction method based on high-resolution SAR imagery using Transformer multi-scale feature fusion, characterized in that: The steps include: Step 1, data acquisition: SAR images of similar resolutions with different imaging parameters and different imaging equipment under different scenes are collected, wherein the SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas; the farmland coverage area images of each scene are cropped to obtain image blocks, and the same number of image blocks are randomly selected as samples to form a training data set; Step 2, data enhancement: set a label for each image block in the training data set in step 1, calculate the proportion of farmland in each image block, and perform data enhancement on image blocks where the proportion of farmland exceeds the set threshold; Step 3, network training: Input the image block after data enhancement in step 2 into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; input the extracted multi-scale features of the image block into the decoder network D for training to obtain the prediction result graph, and update the encoder network and decoder network parameters at the same time until the training termination condition is met to obtain a trained network model; The multi-scale feature extraction encoder network E in step 3 is composed of four Transformer modules; The first two Transformer modules are used to extract high-resolution coarse-grained features of image patches; The last two Transformer modules are used to extract high-resolution fine-grained features of image patches; The Transformer module is composed of an overlapping patch merging module, a self-attention module, and a hybrid FFN module connected in sequence; Overlapping patch merging module: used to divide the input image blocks and obtain feature maps of multiple patches. It can merge overlapping patches by setting the patch size, stride size between adjacent patches, and padding size. Self-attention module: used to perform deeper feature extraction on the feature maps of multiple patches to obtain high-level feature maps; Hybrid FFN module: used to provide position encoding information for the Transformer module, consisting of a 3*3 deep convolutional layer and an MLP layer; Step 4, result prediction: Collect the SAR image to be predicted, perform cropping, label setting, and data enhancement on the SAR image to be predicted in sequence to obtain the image block to be predicted, input the image block to be predicted into the trained network model for prediction, and obtain the farmland prediction result map; Step 5, performance evaluation: Use the farmland prediction result map obtained in step 4 and the label map of the SAR image to be predicted to calculate the category evaluation index and overall evaluation index to evaluate the network performance.

2. The farmland extraction method based on high-resolution SAR imagery based on Transformer multi-scale feature fusion according to claim 1 is characterized in that: The specific process of step 2 is: Step 2.1: set a label for each image block in the training dataset and calculate the number of farmland pixels in each image block label; Step 2.2: Divide the number of farmland pixels in each image block label calculated in step 2.1 by the total number of pixels in the corresponding image block to obtain the farmland proportion of each image block. Step 2.3, set the farmland ratio threshold; In step 2.4, the farmland ratio of each image block in step 2.2 is compared with the farmland ratio threshold in step 2.3, and data enhancement is performed on the image blocks whose farmland ratio exceeds the threshold, wherein the data enhancement includes random contrast and random hue.

3. The farmland extraction method based on high-resolution SAR imagery using Transformer multi-scale feature fusion according to claim 1, characterized in that: The decoder network D is composed of an MLP module and an ASPP module. The training process of the decoder network D is as follows: Step 3.11: Input the multi-scale features extracted by the multi-scale feature extraction encoder network E into different MLP layers to unify the channel dimension; Step 3.12: Input the features after unifying the channel dimensions in step 3.11 into four deep ASPP modules with different expansion rates for decoding; In step 3.13, the features decoded in step 3.12 are input into the convolutional layer for feature fusion, segmentation mask prediction, and prediction result graph.

4. The farmland extraction method based on high-resolution SAR imagery using Transformer multi-scale feature fusion according to claim 1, characterized in that: The category evaluation indicators in step 5 include: Precision: In the formula, i represents a positive example; j represents a negative example; P represents the precision rate, that is, the proportion of correct positive predictions to all positive predictions; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP; Intersection over Union (IoU): Where i represents a positive example; j represents a negative example; IoU represents the intersection over union ratio, which is the ratio of the intersection and union of each type of prediction result and the true value; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ij represents the total number of pixels whose true category is j and identified as category i, i.e. false positive FP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j; Recall: In the formula, i represents a positive example; j represents a negative example; R represents the recall rate, that is, the proportion of correctly predicted positive samples to all positive samples; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and identified as category j; F1score value: Where F1 represents the F1score value, which is the harmonic mean of recall and precision; R represents the recall, which is the proportion of correctly predicted positive samples to all positive samples; P represents the precision, which is the proportion of correctly predicted positive samples to all positive samples.

5. The farmland extraction method based on high-resolution SAR imagery based on Transformer multi-scale feature fusion according to claim 1, characterized in that: The overall evaluation indicators in step 5 include: Overall accuracy OA: Where i represents a positive example; j represents a negative example; OA represents the overall accuracy, that is, the proportion of correctly labeled pixels to the total pixels; p ii The total number of pixels whose true category is i and identified as category i is the true positive case TP; Kappa coefficient: In the formula, kappa represents the kappa coefficient, which means that the "bias" of the penalty model is used to obtain a more fair model; p o represents the proportion of the number of correctly classified samples of each category to the total number of samples, equivalent to OA; a1, a2, ..., aC represent the number of real samples of each category; b1, b2, ..., bC represent the predicted number of samples of each category; the number of categories is C, the total number of samples is n; p e Represents the sum of the products of the actual and predicted numbers of all categories divided by the square of the total number of samples; Average Intersection-over-Union (MIoU): Where i represents a positive example; j represents a negative example; MIoU represents the average intersection-over-union ratio, which is the average sum of the ratios of the intersection and union of each type of prediction result and the true value; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij Indicates the total number of pixels whose true category is j and identified as category i, that is, false positive FP; Frequency Weighted Intersection over Union (FWIoU): Where i represents a positive example; j represents a negative example; FWIoU represents the frequency-weighted intersection-over-union ratio, which is an improvement of MIoU, setting weights according to the frequency of occurrence of the category; p ii represents the total number of pixels whose true category is i and is identified as category i, i.e., true positive examples TP; p ji The total number of false negative examples FN represents the number of pixels whose true category is i and whose category is j; p ij It represents the total number of pixels whose true category is j and identified as category i, that is, false positive FP.

6. A farmland extraction system based on high-resolution SAR images and Transformer multi-scale feature fusion based on the method of claim 1, characterized in that: include: Data acquisition module: collects SAR images of similar resolution with different imaging parameters and different imaging equipment in different scenes. The SAR images are images of farmland coverage areas, which include farmland areas and non-farmland areas. The farmland coverage area images of each scene are cropped to obtain image blocks, and the same number of image blocks are randomly selected as samples to form a training data set. Data augmentation module: sets a label for each image block in the training dataset, calculates the proportion of farmland in each image block, and performs data augmentation on image blocks where the proportion of farmland exceeds a set threshold; Network training module: Input the data-enhanced image block into the multi-scale feature extraction encoder network E to extract the multi-scale features of the image block; The extracted multi-scale features of the image blocks are input into the decoder network D for training to obtain the prediction result graph. At the same time, the encoder network and decoder network parameters are updated until the training termination condition is met to obtain a trained network model. Result prediction module: collects the SAR image to be predicted, performs cropping, label setting, and data enhancement on the SAR image to be predicted, obtains the image block to be predicted, inputs the image block to be predicted into the trained network model for prediction, and obtains the farmland prediction result map; Performance evaluation module: Use the farmland prediction result map and the label map of the SAR image to be predicted to calculate the category evaluation index and the overall evaluation index to evaluate the network performance.

7. A farmland extraction device based on high-resolution SAR images using Transformer multi-scale feature fusion, characterized in that: include: Memory: used to store a computer program for implementing the farmland extraction method of high-resolution SAR images based on Transformer multi-scale feature fusion as claimed in any one of claims 1 to 5; Processor: used to implement the farmland extraction method of high-resolution SAR imagery based on Transformer multi-scale feature fusion according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a farmland extraction method based on high-resolution SAR images of Transformer multi-scale feature fusion as claimed in any one of claims 1 to 5.

Images