Abandoned land extraction method based on multi-scale multi-time sequence and loss comparison

Abstract

The invention discloses an abandoned land extraction method based on multi-scale multi-time sequence and contrast loss, and the method employs the introduction of a dual-feature extraction module and multi-scale feature fusion, enables a network to learn local features and global features from abundant input data distribution, and improves the network feature learning capability. Secondly, a self-attention module is added behind the feature extraction module, and global features are further enhanced; and finally, the idea of comparative learning is innovatively introduced into a supervised learning process, and a special method for dividing positive and negative samples is defined according to task characteristics of abandoned land segmentation, so that the segmentation accuracy is greatly improved. The abandoned land semantic information can be effectively segmented from the cultivated land range, and the result of the ablation experiment further explains the effectiveness of each module.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image semantic segmentation, and in particular to a method based on multi-scale, multi-time series and contrast loss, for automatically segmenting semantic information of "abandoned land" from cultivated land. Background Art

[0002] With the rapid development of high-resolution remote sensing imaging technology, deep learning algorithms are increasingly being used to detect and segment ground object information in remote sensing images. These technologies can efficiently and accurately extract ground object features in remote sensing images, provide technical support for national and social needs such as resource monitoring and disaster analysis, and have become a hot area for researchers. Among many remote sensing application tasks, the demand for using remote sensing images to monitor abandoned land is becoming increasingly strong. The definition of abandoned land must be based on the standard of cultivated land being abandoned for more than one year. Therefore, the extraction of abandoned land requires a comprehensive judgment based on multi-temporal remote sensing images, which is complex and time-consuming.

[0003] Unfortunately, researchers are currently more focused on the extraction of farmland within the scope of cultivated land, while there are very few algorithms for segmenting abandoned land. There are two reasons for this: 1. Data scarcity: There is currently no public dataset of abandoned land annotations, which limits researchers' ability to train deep neural networks using supervised learning methods. In addition, even if unsupervised learning is tried, the amount of farmland data within the scope of cultivated land is far greater than the amount of abandoned land data, leading to a class imbalance problem; 2. Task complexity: The identification of abandoned land requires the combination of multi-phase remote sensing images of more than one year to accurately distinguish. However, from single-phase remote sensing images, it is difficult to distinguish seasonal fallow from human abandonment based on visual features alone. For example Figure 1 As shown in the figure, the yellow box includes two categories: seasonal fallow land and abandoned land. Figure 1 The current image in (a) is difficult to distinguish and must be combined with Figure 1 (b) can make an accurate judgment. Summary of the invention

[0004] In order to solve the two major problems in the abandoned land segmentation problem, namely, data scarcity and task complexity, the present invention proposes an efficient segmentation method which combines multi-scale feature fusion, multi-time series contrast learning with supervised learning and self-attention mechanism, which is specifically used for the extraction of abandoned land within the scope of cultivated land.

[0005] The specific technical solution of the present invention comprises the following steps:

[0006] (1) Dataset construction: The spectral remote sensing image data of Wenchang City in April 2023, July 2023 and March 2024 were selected as the basic sample set, and the images of different periods were matched according to the longitude and latitude information to ensure the temporal and spatial consistency; farmland and abandoned land were manually labeled to produce the abandoned land segmentation task dataset; images from different periods were introduced for comparative labeling of abandoned land, which not only improved the accuracy of the labels but also enriched the distribution of training data.

[0007] (2) Definition of network model structure: Use the dataset to train a neural network with multi-scale feature fusion and multi-time series input; the neural network consists of a dual feature extraction module, a self-attention module and a decoding module; the dual feature extraction module downsamples the input image through convolution and dilated convolution and outputs multi-level local features; the self-attention module takes the last layer output of the feature extraction module as input and enhances the global features through a multi-head self-attention mechanism; the decoding module unifies the feature scales of the multi-level local features and global features through a linear interpolation method to obtain the final segmentation mask image.

[0008] (3) Design of loss function: Dice loss and CE loss (CrossEntropy loss) are calculated by comparing the output segmentation mask with the true mask. Features of different periods are extracted through the dual feature extraction module, and the contrast loss is calculated based on positive and negative samples.

[0009] Furthermore, the dual feature extraction modules described in step (2) all adopt the structure of ResNet101, and the last stage of ResNet101 adopts the dilated convolution with parameters dilation = 2, 4, 8 instead of ordinary convolution; the two pictures with almost the same seasonal information in March 2024 and April 2023 are extracted with features through the same feature extraction module, and the results are recorded as f1 and f2 respectively; the picture with a large difference in seasonal information in July 2023 is extracted with features through another feature extraction module, and the result is recorded as f3; f1 and f3 are used as inputs of the subsequent self-attention module to participate in the prediction of the mask; f2 is used as an auxiliary feature to participate in the definition of positive and negative samples in the subsequent contrast loss; at the same time, the intermediate layer features output by the second and third stages of the dual feature extraction module are recorded as m1, n1 and m3, n3 respectively, and participate in the multi-scale feature fusion of the subsequent decoding module.

[0010] Furthermore, the self-attention module described in step (2) calculates self-attention using a convolution operation with shared parameters instead of a commonly used fully connected layer. After one layer of convolution compression, the number of self-attention channels is 256 and the number of self-attention heads is 8. Moreover, the self-attention module has only one self-attention layer, and after calculating the self-attention, there is no need to further process the features through a fully connected layer.

[0011] Furthermore, the decoding module described in step (2) completes multi-scale feature fusion; the outputs of f1 and f3 after being processed by the self-attention module are t1 and t3 respectively, t1 and t3 are spliced ​​in the channel dimension, and the spliced ​​features are processed by convolution layer, normalization layer, activation layer and bilinear interpolation function to be spliced ​​with n1, n3 and m1, m3; the spliced ​​features are repeated with the above operation until the scale of the spliced ​​features is consistent with the scale of the original image; finally, the final output mask is obtained through a convolution layer with an output channel of 3.

[0012] Furthermore, the definition of positive and negative samples in the contrast loss described in step (3) is different from that in the traditional method; positive and negative samples are defined by calculating the difference between the ratio of the distance between image features of different seasons and the distance between image features of the same season and different years in the feature space, and the formula is: The calculation result is s. The closer s is to 0, the more the feature difference is caused by seasonal factors, and the corresponding features f1 and f3 are positive samples of each other. On the contrary, the farther s is from 0, the more the feature difference is caused by human abandonment, and the corresponding features f1 and f3 are negative samples. Through the common NCE loss (Noise Contrastive Estimation loss) constraint in contrastive learning, the feature extraction module can distinguish the feature differences caused by human abandonment and seasonal changes.

[0013] Beneficial effects: By inputting multi-time series remote sensing images, the network can extract sufficient features from the rich data distribution, fully characterize the spatiotemporal changes of abandoned land, and prepare for subsequent comparative learning; through multi-scale feature fusion, the network can learn global features and local features at the same time, which not only improves the overall accuracy of semantic segmentation, but also ensures the accuracy of extracted boundaries; through the self-attention mechanism, the network enhances the global feature modeling ability, effectively improves the semantic expression ability and output accuracy of the segmentation mask; through comparative learning, the network can learn robust feature extraction capabilities, and can accurately extract abandoned land even under the interference of seasonal changes. The present invention has a sufficient theoretical basis, a simple network model, is easy to train and deploy, and has accurate segmentation, which can promote the application of deep learning in the field of abandoned land segmentation within the scope of cultivated land. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an example map of seasonal fallow and abandoned land;

[0015] Figure 2 It is the overall flow chart of the present invention;

[0016] Figure 3 This is the overall structure diagram of the dual encoding module, using two ResNet101 networks;

[0017] Figure 4This is the decoder module diagram. DETAILED DESCRIPTION

[0018] Example 1

[0019] refer to Figure 2 The complete process of the present invention is divided into data processing, dual feature extraction module extracts features, self-attention module enhances global features, decoding module fuses multi-scale features, and calculates loss function and updates network parameters. The following is a detailed description of each step:

[0020] For the data processing part, this method first crops and matches the multi-time series remote sensing images according to the longitude and latitude information, and crops the remote sensing images of the entire Wenchang City into many 512×512 slices; then the slices are filtered, and according to the cultivated land range map of Wenchang City, the slices with cultivated land area less than 75% are filtered out; then the labelme tool is used to compare the multi-time series remote sensing images and produce label information for the data set. The label contains 3 categories. The pixel value of 0 indicates that the pixel belongs to the background, the pixel value of 1 indicates that the pixel belongs to the farmland, and the pixel value of 2 indicates that the pixel belongs to the abandoned land. The training data set contains a total of 650 pairs of training samples. One pair of training samples contains label information and images from March 2024, July 2023, and April 2023; finally, after normalization and image-to-tensor preprocessing, we get Figure 2 Input.

[0021] For the feature extraction part of the dual feature extraction module, the images in March 2024, April 2023, and July 2023 are x1, x2, and x3 respectively. Since x1 and x2 are not interfered by seasonal factors, the features are extracted through the same encoder, that is, Figure 3 The upper part. And x3 extracts features through another encoder, that is Figure 3 The lower part. The structure of the encoder is two ResNet101s. The difference is that the downsampling and standard modules in stage 4 use dilated convolution, so the dilation parameters are 2, 4, and 8 respectively. Let the global features of x1, x2, and x3 output by encoder stage 4 be f1, f2, and f3, the local features of x1 and x3 output by encoder stage 2 be m1, m3, and the local features output by encoder stage 3 be n1, n3. Note that x2 is an auxiliary image and does not participate in feature fusion, so its local features do not need to be output. f1, f2, and f3 participate in the calculation of contrast loss. At the same time, m1, m3, n1, n3, f1, and f3 participate in the subsequent multi-scale feature fusion after splicing. The formula is:

[0022] S3=Concat(m1,m3), S4=Concat(n1,n3), S5=Concat(f1,f3). (1)

[0023] Among them, S3, S4, and S5 are the splicing features after stage 3, stage 4, and stage 5 respectively.

[0024] For the self-attention module to enhance the global features, since the number of feature channels of S5 is 2048, it is necessary to first pass through a 1×1 convolution layer to compress the number of feature channels from 2048 to 256. Then, the global self-attention is calculated through the 8-head self-attention module. Therefore, the output channels of the query matrix Q, key matrix K, and value matrix V are 32, 32, and 256 respectively. The self-attention calculation formula is:

[0025]

[0026] Where x is the input of the self-attention module, which is the feature of S5 after being compressed by the convolution layer in this invention, and d k It is a hyperparameter and is set to 1 in the present invention.

[0027] For the decoding module, the multi-scale feature part is integrated, such as Figure 4 As shown, After conversion by the CBR module (convolutional layer, BatchNorm layer, ReLU nonlinear layer), the feature scale is upsampled twice by bilinear interpolation to make it the same as S4, and it is concatenated with the features of S4 after conversion by the CBR module in the channel dimension. The decoding module repeats the above operation until all features are concatenated, and finally upsamples four times by bilinear interpolation to make the output mask consistent with the input image size.

[0028] Regarding the calculation of the loss function, the loss function of the present invention is composed of Dice loss and CE loss (CrossEntropy loss) of supervised learning and NCE loss of contrastive learning. The positive and negative samples in contrastive learning are defined by calculating the difference between the ratio of the distance between image features of different seasons and the distance between image features of different years in the same season in the feature space. The formula is:

[0029]

[0030] Among them, the closer s is to 0, it means that the feature difference is mainly caused by seasonal factors, and the corresponding features f1 and f3 are positive samples of each other; conversely, the farther s is from 0, it means that the feature difference is mainly caused by human abandonment, and the corresponding features f1 and f3 are negative samples; at the same time, the features between different pictures are also different, so they are also negative samples of each other. Due to the small number of positive and negative samples, the positive and negative sample features are stored in the memory bank. The output of the decoding module needs to calculate CEloss and Dice loss with the dataset label. All losses are added with specific weights to get the final loss function:

[0031] L=CE+Dice+0.5×NCE. (4)

[0032] For updating network parameters, this experiment uses the SGD parameter optimizer, with the SGD parameter momentum set to 0.9 and weight_decay set to 5e-4. The SGD learning rate is set to 5e-3, the training rounds are 200, and the learning rate update strategy uses the poly scheduler.

[0033] The results of the ablation experiments in Tables 1 to 3 illustrate the effectiveness of the above methods. The effectiveness of each module is verified by removing or changing a module while keeping the other modules unchanged.

[0034] The experimental results in Table 1 illustrate the necessity of multi-scale feature fusion. Since remote sensing images may have slight offsets in the process of matching according to longitude and latitude, or local area information may be lost due to cloud cover and exposure, the improvement of only shallow feature fusion is limited; and since the semantic segmentation task is a pixel-level image classification task, it is necessary to pay attention to local features, so only using a deep feature extractor cannot obtain the optimal result.

[0035] The experimental results in Table 2 illustrate the effectiveness of single-layer convolutional self-attention on this dataset. Theoretically, using fully connected self-attention, self-attention layer followed by fully connected layer, and multi-layer self-attention superposition are all effective means to improve the network representation ability. However, since the abandoned land dataset is small and cannot support large-scale network training, single-layer convolutional self-attention achieves better performance. At the same time, compared with ASPP, global feature self-attention modeling enables the network to capture long-distance dependencies, and the learned global features are more effective.

[0036] The experimental results in Table 3 illustrate the rationality of the positive and negative sample definition strategy. SeCo and CACo are networks trained using contrastive learning ideas under multi-time series input conditions, but these two methods do not consider the changes in image semantic information caused by short-term human factors when defining positive and negative samples, which is the key to semantic segmentation of abandoned land.

[0037] Table 1 Ablation experiment of the number of feature extractors and feature fusion strategy, using MIoU (Mean Intersection over Union) to measure the segmentation effect. The larger the MIoU, the better the segmentation effect. Among them, single feature extractor without feature fusion means that only the Wenchang image in March 2024 is input and only one feature extractor is trained, so there is no feature fusion; dual feature extractor shallow feature fusion refers to the feature fusion of ResNet101 stage 2 output; dual feature extractor deep feature fusion refers to the feature fusion of the last stage output of ResNet101.

[0038]

[0039] Table 2 Ablation experiments of global feature modeling strategies. ASPP comes from the paper Deeplabv3, which uses global pooling to model global features; self-attention uses the attention mechanism to capture long-distance dependencies, which is also a way to model global features; both convolution and fully connected layers can calculate self-attention, the difference is that convolution shares weight parameters, so the number of parameters to be learned is smaller; single-layer attention followed by fully connected layers and double-layer attention will increase the number of network parameters, leading to overfitting.

[0040]

[0041] Table 3 Ablation experiments of contrastive learning positive and negative sample definition strategies. Among them, non-contrastive loss refers to the use of only cross entropy loss in supervised learning; SeCo and CACo are two effective temporal contrastive learning strategies in the field of remote sensing. SeCo simply regards the same image area that changes due to seasonal factors as positive samples and different image areas as negative samples, without considering the impact of human factors; CACo regards image changes separated by several years (often caused by human factors) as negative samples and image changes separated by several months (often caused by seasonal factors) as positive samples, but this design does not take into account short-term human changes, such as abandoned land.

[0042]

Claims

1. Abandoned land extraction method based on multi-scale, multi-time series and contrast loss, characterized in that: The method comprises the following steps: (1) Spectral remote sensing images of Wenchang City at different times were selected as the basic sample set. Images from different periods were matched one by one according to longitude and latitude information to produce a dataset for the abandoned land segmentation task. (2) The dataset is used to train a neural network with multi-scale feature fusion and multi-time series input; the neural network consists of a dual feature extraction module, a self-attention module, and a decoding module; the dual feature extraction module downsamples the input image through convolution and dilated convolution and outputs multi-level local features; the self-attention module takes the last layer output of the feature extraction module as input and enhances the global features through a multi-head self-attention mechanism; the decoding module unifies the feature scales of the multi-level local features and global features through a linear interpolation method to obtain a segmentation mask, i.e., the extracted range of abandoned land; (3) The cross entropy loss and Dice loss are calculated by comparing the output segmentation mask with the true mask. The features of different periods are extracted through the dual feature extraction module, and the contrast loss is calculated based on the positive and negative samples. The positive and negative samples are defined by calculating the difference between the ratio of the distance between the image features of different seasons and the distance between the image features of different years in the same season in the feature space. The network parameters are updated by calculating the sum of the losses and back-propagating. The range of abandoned land output by the network is closer to the real range.

2. The method for extracting abandoned land based on multi-scale, multi-time series and contrast loss according to claim 1, characterized in that: The images of different periods described in step (1) are three-period images containing information of different seasons; introducing images of different periods for comparative labeling of abandoned land not only improves the accuracy of labels, but also enriches the distribution of training data.

3. The method for extracting abandoned land based on multi-scale, multi-time series and contrast loss according to claim 1, characterized in that: The dual feature extraction modules described in step (2) all adopt the structure of ResNet101, and the last stage of ResNet101 adopts the dilated convolution with parameters dilation=2, 4, 8 instead of ordinary convolution; two pictures with similar seasonal information are extracted features by the same feature extraction module, and the results are recorded as f1 and f2 respectively; pictures with different seasonal information are extracted features by another feature extraction module, and the result is recorded as f3; f1 and f3 are used as inputs of the subsequent self-attention module to participate in the prediction of the mask; f2 is used as an auxiliary feature to participate in the definition of positive and negative samples in the subsequent contrast loss; at the same time, the intermediate layer features of the second and third stage outputs of the dual feature extraction module are recorded as m1, n1 and m3, n3 respectively, and participate in the multi-scale feature fusion of the subsequent decoding module.

4. The method for extracting abandoned land based on multi-scale, multi-time series and contrast loss according to claim 1, characterized in that: The self-attention module described in step (2) calculates self-attention using a convolution operation with shared parameters instead of the commonly used fully connected layer. After one layer of convolution compression, the number of self-attention channels is 256 and the number of self-attention heads is 8. Moreover, the self-attention module has only one self-attention layer, and after calculating the self-attention, there is no need to further process the features through a fully connected layer.

5. The method for extracting abandoned land based on multi-scale, multi-time series and contrast loss according to claim 3, characterized in that: The decoding module described in step (2) completes multi-scale feature fusion; the outputs of f1 and f3 after processing by the self-attention module are recorded as t1 and t3 respectively, t1 and t3 are spliced ​​in the channel dimension, and the spliced ​​features are processed by convolution layer, normalization layer, activation layer and bilinear interpolation function to be spliced ​​with n1, n3 and m1, m3; the above operation is repeated for the spliced ​​features until the scale of the spliced ​​features is consistent with the scale of the original image; finally, the final output mask is obtained through a convolution layer with an output channel of 3.

6. The method for extracting abandoned land based on multi-scale, multi-time series and contrast loss according to claim 5, characterized in that: The difference definition formula in the contrast loss described in step (3) is: The calculation result is s. The closer s is to 0, the more the feature difference is caused by seasonal factors, and the corresponding features f1 and f3 are positive samples of each other. On the contrary, the farther s is from 0, the more the feature difference is caused by human abandonment, and the corresponding features f1 and f3 are negative samples. By comparing the common NCEloss constraints in learning, the feature extraction module distinguishes the feature differences caused by human abandonment and seasonal changes.

Images