Method for detecting semantic change of remote sensing image

Abstract

The invention discloses a remote sensing image semantic change detection method, which belongs to the field of remote sensing image semantic change detection, and comprises the following steps: carrying out double-temporal feature extraction by using a residual network, sending the double-temporal features into a constructed difference enhancement module to model difference information, and obtaining a change weight of the double-temporal features; the change weight and the double-temporal feature are sent to a feature interaction module, interaction among the three branches is completed, the interacted double-temporal feature and a differential feature rich in semantic information are obtained, a binary change graph is obtained from a segmentation head to a semantic segmentation graph and through a change detection head, the semantic segmentation graph is superposed on the obtained binary change graph, and the binary change graph is obtained. And finally generating a semantic change graph. Compared with other existing semantic change detection methods, the method provided by the invention is more excellent, can reduce leak detection conditions under different spectrums similar in semantics and error detection conditions under the same spectrums change semantics, and also has a good effect in detail and edge detection.

Description

Technical Field

[0001] The present invention relates to the field of semantic change detection of remote sensing images, and in particular to a method for semantic change detection of remote sensing images. Background Art

[0002] Semantic change detection (SCD) is an important analytical method in remote sensing image analysis, used to detect changes in the land surface in remote sensing images of the same location at different times. Currently, change detection (CD) is widely used in various fields, including urban planning, land use, and disaster monitoring. Based on the type of semantic annotation, change detection can be divided into binary change detection (BCD) and semantic change detection (SCD). While BCD only requires identifying changes and no changes, SCD is more challenging, requiring not only detecting changes but also identifying the semantics of land cover.

[0003] In recent years, deep learning (DL) has been widely used in image analysis tasks and has achieved remarkable success. Daudt et al. proposed the SCD network as a multi-task learning framework and established a benchmark for SCD research by predicting binary change maps (BCM) and land cover maps (LCM). Since then, SCD has developed rapidly.

[0004] The popular SCD network is based on a three-branch framework: two temporal branches for semantic detection and one change branch for change detection. The input is images T1 and T2 at two time points, which are encoded using a weight-sharing encoding structure. The two temporal branches then predict semantic information to obtain the LCM. After feature fusion, the change branch predicts change to obtain the BCM. The LCM represents the land cover type. The BCM uses 0 and 1 to represent unchanged and changed, respectively. Finally, post-processing is performed during the inference phase, where the BCM is multiplied by the LCM as a mask to obtain the final result: two semantic change maps Y1 and Y2.

[0005] This framework can be briefly considered as a combination of semantic segmentation (SS) and BCD tasks. SS and BCD are both pixel-level segmentation tasks and share many commonalities. Therefore, a multi-task learning framework combining these two can reduce the risk of overfitting and improve the model's generalization ability. However, this framework suffers from insufficient interaction. First, relying on a common encoder for cross-task modeling is insufficient. Second, in the decoder, the two temporal branches learn semantic representations separately, without considering the correlation between the two subtasks in the decoder.

[0006] The three branches lack or have weak interaction, making it difficult for the network to learn the complex relationship between spectrum and semantics. A common phenomenon is missed detection, where objects with similar spectra but different semantic categories are easily identified as unchanged. Another phenomenon is false detection, where objects with the same semantics but significant spectral differences are mistakenly detected as changed. Therefore, a semantic change detection method is needed that can collaboratively learn the complex relationship between spectrum and semantics, thereby reducing missed and false detections. Summary of the Invention

[0007] In order to solve the problems of missed detection and false detection in existing semantic detection, the present invention provides a method for semantic change detection in remote sensing images.

[0008] The present invention adopts the following technical solutions:

[0009] A method for detecting semantic changes in remote sensing images, comprising the following steps:

[0010] Step 1: First, the dual-temporal remote sensing images T1 and T2 are fed into the weight-sharing residual network for dual-temporal feature extraction to obtain deep dual-temporal features F1 and F2;

[0011] Step 2: The bi-temporal features F1 and F2 obtained in step 1 are fed into the constructed difference enhancement module to model the difference information and obtain the change weight w of the bi-temporal features;

[0012] Step 3: Send the change weight obtained in step 2 and the bi-temporal features F1 and F2 obtained in S1 to the feature interaction module to complete the interaction between the three branches and obtain the bi-temporal feature F after interaction. b1 、F b2 and a semantically rich differential feature F b ;

[0013] Step 4: Construct the segmentation head and change detection head, and use the bi-temporal feature F obtained in step 3 b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map;

[0014] Step 5: Superimpose the semantic segmentation map obtained in step 4 on the binary change map to finally generate a semantic change map.

[0015] Preferably, step 2 specifically includes the following steps:

[0016] Step 2.1: Construct a difference enhancement module, which includes one convolutional layer, one pooling layer, two convolutional layers, one activation layer, and one convolutional layer.

[0017] Step 2.2: After the bi-temporal features F1 and F2 are subjected to element-by-element subtraction and channel-by-channel fusion operations, the number of channels is adjusted through the convolution layer, and then the rough differential feature F is obtained after channel-by-channel fusion. a , get the rough differential feature F a It is sent to the pooling layer and two convolutional layers to filter out erroneous information, and a residual connection is used to alleviate the gradient vanishing problem. Finally, it passes through an activation layer and a convolutional layer to reduce the number of channels to 1, and obtain the change weight w.

[0018] Preferably, step 3 specifically includes the following steps:

[0019] Step 3.1: Construct a convolutional block. The convolutional block includes one convolutional layer, one batch normalization layer, one activation layer, one convolutional layer, one batch normalization layer, residual connection, and one activation layer.

[0020] Step 3.2: After the change weight w obtained in step 2 is activated by the activation function, it is multiplied with the bi-temporal features F1 and F2 respectively, and then the semantically enhanced bi-temporal features F are obtained through the convolution block. b1 、F b2 ;

[0021] Step 3.3: F b1 、F b2 Perform channel aggregation to obtain the semantically enhanced differential feature F b , two time branches and one change branch complete the interaction.

[0022] Preferably, step 4 specifically includes the following steps:

[0023] Step 4.1: Construct a decoding block. The decoding block includes one convolutional layer, one batch normalization layer, one activation layer, one convolutional layer, one batch normalization layer, residual connection, and one activation layer.

[0024] Step 4.2: Construct a segmentation header, which includes three decoding blocks, and construct a change detection header, which includes five decoding blocks;

[0025] Step 4.3: Transform the bi-phase feature F b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map.

[0026] The present invention has the following beneficial effects:

[0027] The method for semantic change detection of remote sensing images provided by the present invention designs a difference enhancement module to perform spatiotemporal correlation modeling, obtains change information from different angles, enhances the recognition of change features and suppresses pseudo-changes. A feature interaction module between the three branches is designed to help the network learn the complex relationship between semantics and spectrum, so that the network can adaptively learn target changes under complex circumstances, thereby reducing missed detections and false detections. The present invention is compared with other advanced semantic change detection methods on two data sets (SECOND, LANDSAT). Experiments show that the present invention is superior to the comparison method in the semantic change detection task, can reduce missed detections under similar spectra but different semantics and false detections under the same spectral change semantics, and can also have good effects in the detection of details and edges. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the overall framework diagram of the present invention.

[0029] Figure 2 This is the framework diagram of the difference enhancement module.

[0030] Figure 3 The present invention provides a comparison diagram of semantic change maps of three groups of remote sensing images detected by the method of the present invention and semantic change maps of three groups of remote sensing images detected by other semantic change detection methods. DETAILED DESCRIPTION

[0031] The specific implementation of the present invention will be further described below with reference to the accompanying drawings and specific examples.

[0032] Combine Figures 1 to 3 , a method for semantic change detection in remote sensing images, comprising the following steps:

[0033] Step 1: First, the dual-temporal remote sensing images T1 and T2 are sent to the weight-sharing residual network (ResidualNetwork, ResNet18) for dual-temporal feature extraction to obtain deep dual-temporal features F1 and F2.

[0034] The residual network here is slightly modified, that is, the step size of the last two stages is set to 1, and the original number of channels is kept, while the others remain unchanged.

[0035] Step 2: The bi-temporal features F1 and F2 obtained in step 1 are fed into the constructed Difference Enhancement Module (DEM) to model the difference information and obtain the change weight w of the bi-temporal features.

[0036] like Figure 2 , specifically including:

[0037] Step 2.1: Construct a difference enhancement module. The difference enhancement module consists of one convolution layer, one pooling layer, two convolution layers, one activation layer (ReLU), and one convolution layer.

[0038] Step 2.2: After the bi-temporal features F1 and F2 are subjected to element-by-element subtraction and channel-by-channel fusion operations, the number of channels is adjusted through the convolution layer, and then the rough differential feature F is obtained after channel-by-channel fusion. a , get the rough differential feature F a It is sent to the pooling layer and two convolutional layers to filter out erroneous information, and a residual connection is used to alleviate the gradient vanishing problem. Finally, it passes through an activation layer and a convolutional layer to reduce the number of channels to 1, and obtain the change weight w (this process is supervised by the label).

[0039] Step 3: Send the change weight obtained in step 2 and the bi-temporal features F1 and F2 obtained in S1 to the Feature Interaction Module (FIM) to complete the interaction between the three branches and obtain the bi-temporal feature F after interaction. b1 、F b2 and a semantically rich differential feature F b .

[0040] like Figure 1 , specifically including:

[0041] Step 3.1: Construct a convolutional block. The convolutional block includes one convolutional layer, one batch normalization layer, one activation layer (ReLU), one convolutional layer, one batch normalization layer, residual connection, and one activation layer (ReLU).

[0042] Step 3.2: After the change weight w obtained in step 2 is activated by the activation function (the activation function is Tanh), it is multiplied with the bi-temporal features F1 and F2 respectively, and then the semantically enhanced bi-temporal features F are obtained through the convolution block. b1 、F b2 .

[0043] Step 3.3: F b1 、F b2 Perform channel aggregation to obtain the semantically enhanced differential feature F b , two time branches and one change branch complete the interaction.

[0044] Step 4: Construct the segmentation head and change detection head, and use the bi-temporal feature F obtained in step 3 b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map.

[0045] Specifically include:

[0046] Step 4.1: Construct a decoding block. The decoding block consists of one convolutional layer, one batch normalization layer, one activation layer (ReLU), one convolutional layer, one batch normalization layer, residual connection, and one activation layer (ReLU).

[0047] Step 4.2: Construct a segmentation header, which includes 3 decoding blocks, and construct a change detection header, which includes 5 decoding blocks.

[0048] Step 4.3: Transform the bi-phase feature F b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map.

[0049] Step 5: Superimpose the semantic segmentation map obtained in step 4 on the binary change map to finally generate a semantic change map.

[0050] Table 1 compares the results of semantic change detection on the SECOND dataset using the method of the present invention with those obtained using other semantic change detection methods.

[0051] Table 1 Comparison of semantic change detection

[0052]

[0053] The above table shows the comparative results of the method of the present invention and other methods on the SECOND dataset. It can be seen that HRSCDstr4 uses a simple interactive method to connect the features of the time branch to the change branch, and achieved a score of 9.67%. Other interactive methods, such as SESNet, BiSRNet and SSCD1, also achieved good results. The results in the table show that the method of the present invention is superior to existing methods in both change detection and semantic recognition. Compared with SSCD1, the method of the present invention improved by 0.77% in the intersection-union ratio, 6.69% in SeK, and 3.11% in the score.

[0054] For a more intuitive comparison, Figure 3 The following visualizations show the results of several methods on the SECOND dataset. In Figures a1 and a2 above, the trees and low vegetation within the red dashed boxes have similar spectral characteristics, making them easily identified as unchanged. Using conventional methods to capture only difference information would overlook this issue, leading to detection failure. By incorporating channel concatenation into the DEM, we can preserve the full characteristics of the bi-temporal imagery, enabling the network to better learn this complex situation.

[0055] In Figures b1 and b2 above, low vegetation is easily misidentified as water due to its darker color. Because low vegetation and trees share similar spectral characteristics, they are often misidentified. Figures c1 and c2 above show objects with the same semantic meaning being misidentified due to spectral differences. This demonstrates that the method of the present invention is capable of identifying such spurious changes. This is because the method learns change characteristics from different perspectives using the DEM and uses the FIM to interact between change and semantics, enabling the network to distinguish such spurious changes.

[0056] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A method for detecting semantic changes in remote sensing images, characterized in that: The following steps are involved: Step 1: First, the dual-temporal remote sensing images T1 and T2 are fed into the weight-sharing residual network for dual-temporal feature extraction to obtain deep dual-temporal features F1 and F2; Step 2: The bi-temporal features F1 and F2 obtained in step 1 are fed into the constructed difference enhancement module to model the difference information and obtain the change weight w of the bi-temporal features; Step 3: Send the change weight obtained in step 2 and the bi-temporal features F1 and F2 obtained in S1 to the feature interaction module to complete the interaction between the three branches and obtain the bi-temporal feature F after interaction. b1 、F b2 and a semantically rich differential feature F b ; Step 4: Construct the segmentation head and change detection head, and use the bi-temporal feature F obtained in step 3 b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map; Step 5: Superimpose the semantic segmentation map obtained in step 4 on the binary change map to finally generate a semantic change map.

2. The method for detecting semantic changes in remote sensing images according to claim 1, wherein: Step 2 specifically includes the following steps: Step 2.1: Construct a difference enhancement module, which includes one convolutional layer, one pooling layer, two convolutional layers, one activation layer, and one convolutional layer. Step 2.2: After the bi-temporal features F1 and F2 are subjected to element-by-element subtraction and channel-by-channel fusion operations, the number of channels is adjusted through the convolution layer, and then the rough differential feature F is obtained after channel-by-channel fusion. a , get the rough differential feature F a It is sent to the pooling layer and two convolutional layers to filter out erroneous information, and a residual connection is used to alleviate the gradient vanishing problem. Finally, it passes through an activation layer and a convolutional layer to reduce the number of channels to 1, and obtain the change weight w.

3. The method for detecting semantic changes in remote sensing images according to claim 2, wherein: Step 3 specifically includes the following steps: Step 3.1: Construct a convolutional block. The convolutional block includes one convolutional layer, one batch normalization layer, one activation layer, one convolutional layer, one batch normalization layer, residual connection, and one activation layer. Step 3.2: After the change weight w obtained in step 2 is activated by the activation function, it is multiplied with the bi-temporal features F1 and F2 respectively, and then the semantically enhanced bi-temporal features F are obtained through the convolution block. b1 、F b2 ; Step 3.3: F b1 、F b2 Perform channel aggregation to obtain the semantically enhanced differential feature F b , two time branches and one change branch complete the interaction.

4. The method for detecting semantic changes in remote sensing images according to claim 1, wherein: Step 4 specifically includes the following steps: Step 4.1: Construct a decoding block. The decoding block includes one convolutional layer, one batch normalization layer, one activation layer, one convolutional layer, one batch normalization layer, residual connection, and one activation layer. Step 4.2: Construct a segmentation header, which includes three decoding blocks, and construct a change detection header, which includes five decoding blocks; Step 4.3: Transform the bi-phase feature F b1 、F b2 Send it to each segmentation head to get the semantic segmentation map, and convert the differential feature F b It is sent to the change detection head to obtain a binary change map.

Images