Remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion

Abstract

The invention discloses a remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion, which belongs to the field of remote sensing image semantic change detection, and comprises the following steps: performing multi-scale feature extraction in a high-resolution network, performing multi-scale feature integration by using a bidirectional deep and shallow feature aggregation module, and performing remote sensing image semantic change detection. The method comprises the following steps: acquiring difference characteristics through a high-resolution difference extraction module, refining by using an efficient channel attention module to obtain two semantic segmentation graphs and a binary change graph, and superposing the two semantic segmentation graphs on the binary change graph to finally generate a semantic change graph. Compared with other existing semantic change detection methods, the method provided by the invention is more excellent, especially has better effects in the aspects of identifying the contour of a change area and detecting a subtle change area of a complex scene, can reduce semantic errors caused by visual similarity, and has wider applicability.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image semantic change detection, and in particular to a remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion. Background Art

[0002] In Earth observation applications such as land cover and land use (LCLU) monitoring, resource management, and disaster warning, it is crucial to accurately identify areas where semantic changes have occurred. In addition, it is equally important to classify these semantic changes. Semantic Change Detection (SCD) is the key to addressing these challenges, which can identify the change areas and describe the "from what to what" transition in bi-temporal remote sensing data in detail. The increasing number of open datasets, such as SECOND and LANDSAT, has stimulated the increasing attention of the remote sensing research community to SCD.

[0003] Early SCD research emphasized the value of modeling bi-temporal dependencies to accurately capture semantic changes. Methods such as analyzing change vectors to detect changes and then classifying the LCLU of the affected regions, and iteratively predicting LCLU transitions using the Bayesian rule have demonstrated the evolution of the technology in this area. However, these strategies often face the challenge of error accumulation during the iterative estimation process.

[0004] To address this problem, recent SCD studies often use Convolutional Neural Networks (CNNs) to complete the SCD task. An effective strategy is to assign independent decoding heads to the change detection (BCD) subtask to generate a binary change map (BCM), and to assign independent decoding heads to the semantic segmentation (SS) subtask to generate a semantic segmentation map, and finally generate a semantic change map (SCM) by superimposing the semantic segmentation map on the binary change map. This overall strategy of simultaneously modeling and training subtasks significantly improves the performance and exploits the synergy between them, thus promoting the development of SCD networks characterized by a three-branch decoding architecture. On the other hand, single-branch and dual-branch SCD methods have also been explored in related studies.

[0005] With the advancement of SCD technology, networks with triple decoding branch architectures have become increasingly prominent, and are mainly divided into two types. The first is a network architecture with three encoding branches and three decoding branches, such as HRSCD-str.4. The other is a network architecture using two encoding branches and three decoding heads, such as SSCDl. This design aims to create three task-specific decoding heads that utilize the combined output of the two encoding branches. Studies have shown that the dual encoding branch and triple decoding branch framework has advantages in feature representation capabilities and plays a key role in promoting the performance of SCD networks.

[0006] However, how to optimize the dual encoding architecture to support the triple decoding branches remains an important issue. To improve the feature representations generated by the two encoding branches, the self-attention and cross-attention mechanisms are used in Bi-SRNet proposed by Ding et al., while the attention mechanism for bi-temporal fusion and semantic refinement is adopted in JFRNet proposed by Chang et al. Another method is Smnet proposed by Niu et al., which captures the multi-scale feature representations (MSFR) of bi-temporal images and guides the MSFR to their respective SS decoding branches, while merging the differences between the MSFRs before processing the BCD decoding branches through a multi-scale fusion module. In addition, to promote more efficient information exchange between the triple decoding branches, the last three layers of features of each encoding branch are normalized to a uniform spatial resolution and integrated through CNN.

[0007] At present, most of the networks are constructed using an architecture similar to UNet. Although semantic information can be accumulated, continuous downsampling leads to the loss of details, which is not conducive to pixel-level tasks such as semantic change detection. Although previous studies have used skip connections to solve this problem, in order to ensure task balance, skip connections are often ignored. High Resolution Net (HRNet) can retain high-resolution features with minimal downsampling, thereby reducing the uncertainty of spatial information. However, the high-resolution features output by HRNet may not fully capture semantic details, and how to integrate multi-scale features extracted from HRNet is also a problem. Difference information can guide the decoding process of BCD. Simply using dual-phase feature channel concatenation or element-by-element subtraction to extract difference information cannot accurately capture difference information because element-by-element subtraction is sensitive to noise. Although channel concatenation can avoid the above shortcomings, it is heavily dependent on a large number of training samples and lacks interpretability. Summary of the invention

[0008] In view of the shortcomings of existing remote sensing image semantic change detection, the present invention provides a remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion, which improves the accuracy of remote sensing image semantic change detection.

[0009] The present invention adopts the following technical solution: a remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion, comprising the following steps:

[0010] Step 1: First, the dual-time remote sensing images T1 and T2 are sent to the weight-sharing high-resolution network for multi-scale feature extraction. The multi-scale features of the remote sensing image T1 are extracted as follows: The multi-scale features extracted from the remote sensing image T2 are

[0011] Step 2: Construct a bidirectional deep and shallow feature aggregation module to aggregate the multi-scale features obtained in step 1 Send it to the bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain features The multi-scale features obtained in step 1 Send it to the bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain features

[0012] Step 3: feature and Features Send it to the high-resolution difference extraction module to obtain difference features.

[0013] Step 4: Construct an efficient channel attention module and use the efficient channel attention module to refine the difference features obtained in step 3 to obtain refined difference information.

[0014] Step 5: Build the segmentation head and change detection head to transform the features and Features The two semantic segmentation maps are respectively sent to the segmentation head to obtain two semantic segmentation maps; the difference information is sent to the change detection head to obtain a binary change map.

[0015] Step 6: By superimposing the two semantic segmentation maps on the binary change map, a semantic change map is finally generated.

[0016] Preferably, step 2 specifically includes the following steps:

[0017] Step 2.1: Construct a bidirectional deep and shallow feature aggregation module. The bidirectional deep and shallow feature aggregation module includes 3 downsampling layers, 3 upsampling layers, and 1 Sc layer.

[0018] Step 2.2: After a downsampling layer, Generate channel splicing Will After a layer of downsampling, Generate channel splicing Will After a layer of downsampling, Generated after channel splicing

[0019] Will After a downsampling layer, Generate channel splicing Will After processing through layers of downsampling layers, Generate channel splicing Will After a layer of downsampling, Generated after channel splicing

[0020] Step 2.3: Create the After one layer of upsampling Generate channel splicing Will After one upsampling layer, Generate channel splicing Finally After one upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated.

[0021] The generated in step 2.2 After one layer of upsampling Generate channel splicing Will After one upsampling layer, Generate channel splicing Finally After one upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated.

[0022] Preferably, the downsampling layer includes a convolution layer, a batch normalization layer, an activation layer, and a convolution layer; the upsampling layer includes a convolution layer, a batch normalization layer, an activation layer, and an upsampling layer; the Sc layer includes a convolution layer, a batch normalization layer, an activation layer, and a convolution layer.

[0023] Preferably, the specific process in the high-resolution difference extraction module is: Perform element-by-element subtraction and then add it to the feature and Features Perform channel stitching to obtain rough difference features.

[0024] Preferably, step 4 specifically includes: the efficient channel attention module includes a global pooling layer, a one-dimensional convolution layer, and an activation layer; the difference features of step 3 are sent to the efficient channel attention module, firstly global average pooling is performed, and then the channel attention score is obtained by passing through the one-dimensional convolution layer and the activation layer in sequence, and finally the channel attention score is multiplied by the difference feature to obtain the refined difference information.

[0025] Preferably, the segmentation head comprises a convolution layer, a batch normalization layer, an activation layer, a convolution layer, and an upsampling layer; the change detection head comprises: a residual block layer, a convolution layer, a batch normalization layer, an activation layer, and an upsampling layer.

[0026] The present invention has the following beneficial effects:

[0027] The remote sensing image semantic change detection method based on bidirectional fusion of high-resolution features provided by the present invention uses a high-resolution network as a backbone network to extract multi-scale features, retains high-resolution features with minimal downsampling, thereby reducing the uncertainty of spatial information; in order to generate high-quality information for semantic segmentation and change detection subtasks, the present invention creates a bidirectional deep-shallow feature aggregation module, which merges high-resolution spatial features with semantic information extracted from shallow and deep layers through a bidirectional fusion process, providing features with richer semantics and more accurate spatial information; a high-resolution difference extraction module is constructed, which uses high-resolution features with minimal downsampling to obtain difference features, thereby improving the accuracy of the change detection subtask, and an efficient channel attention module further improves the performance of the change detection subtask. Compared with other existing semantic change detection methods, this method is more excellent, especially in identifying the contours of the change area and detecting the subtle change area of ​​complex scenes, and can reduce semantic errors caused by visual similarity, and this method has a wider applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the overall framework diagram of the remote sensing image semantic change detection method based on bidirectional fusion of high-resolution features.

[0029] Figure 2 The framework diagram of the bidirectional deep and shallow feature aggregation module.

[0030] Figure 3 Framework diagram of the high-resolution difference extraction module and the efficient channel attention module.

[0031] Figure 4 The present invention is a comparison diagram of semantic change graphs after detecting three groups of dual-time remote sensing images using the method of the present invention and semantic change graphs after detecting three groups of dual-time remote sensing images using other semantic change detection methods. DETAILED DESCRIPTION

[0032] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and specific examples.

[0033] Combination Figures 1 to 4 , a remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion, comprising the following steps: Step 1: firstly, the dual-time remote sensing images T1 and T2 are sent to the weight-sharing high-resolution network (HRNet) for multi-scale feature extraction, and the multi-scale features extracted from the remote sensing image T1 are The multi-scale features extracted from the remote sensing image T2 are The high-resolution network here is pre-trained using the ImageNet dataset.

[0034] Step 2: Construct a bidirectional deep and shallow feature aggregation module to aggregate the multi-scale features obtained in step 1 The feature is sent to the weight-sharing bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain the feature The multi-scale features obtained in step 1 The feature is sent to the weight-sharing bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain the feature

[0035] like Figure 2 As shown, the following steps are specifically included: Step 2.1: Construct a bidirectional shallow and deep Features Aggregation Module (BiFAM). The bidirectional shallow and deep features aggregation module includes 3 downsampling layers, 3 upsampling layers, and 1 Sc layer. The downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer (ReLU), and a convolutional layer; the upsampling layer includes a convolutional layer, a batch normalization layer, an activation layer (ReLU), and an upsampling layer (performing bilinear interpolation); the Sc layer includes a convolutional layer, a batch normalization layer, an activation layer (ReLU), and a convolutional layer.

[0036] Step 2.2: After one of the downsampling layers, Generate channel splicing Will After another layer of downsampling, Generate channel splicing Will After the last downsampling layer, Generated after channel splicing It can be seen that the three downsampling layers passed through the above process are three different downsampling layers.

[0037] Will After one of the downsampling layers, Generate channel splicing Will After another layer of downsampling, Generate channel splicing Will After the last downsampling layer, Generated after channel splicing Similarly, the three downsampling layers passed through in the above process are also three different downsampling layers.

[0038] Figure 2 For the convenience of representation, and Use F1 to represent and Use F2 to represent and Use F3 to indicate and Use F4 to indicate; and Use D2 to represent and Using D3 to represent and It is represented by D4.

[0039] Step 2.3: Create the After one layer of upsampling, Generate channel splicing Will After another upsampling layer, Generate channel splicing Finally After the last upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated. It can be seen that the three upsampling layers passed through the above process are three different upsampling layers.

[0040] The generated in step 2.2 After one layer of upsampling, Generate channel splicing Will After another upsampling layer, Generate channel splicing Finally After the last upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated. Similarly, the three upsampling layers in the above process are also three different upsampling layers.

[0041] Figure 2 For the convenience of representation, and Use S3 to represent and Use S2 to represent and It is represented by S1.

[0042] Step 3: feature and Features The images are sent to the High-Resolution Difference Extraction (HRDE) module to obtain difference features.

[0043] The specific process in the high-resolution difference extraction module is as follows: Perform element-by-element subtraction and then add it to the feature and Features Perform channel stitching to obtain rough difference features.

[0044] Step 4: Construct an efficient channel attention module and use the efficient channel attention module (ECA) to refine the difference features obtained in step 3 to obtain the refined difference information.

[0045] Specifically, the efficient channel attention module includes a global pooling layer, a one-dimensional convolution layer, and an activation layer. The difference features in step 3 are sent to the efficient channel attention module. First, global average pooling is performed, and then the channel attention score is obtained by passing through the one-dimensional convolution layer and the activation layer in turn. Finally, the channel attention score is multiplied by the difference feature to obtain the refined difference information. Figure 3 shown.

[0046] Step 5: Build the segmentation head and change detection head to transform the features and Features The two semantic segmentation maps are respectively sent to the segmentation head to obtain two semantic segmentation maps; the difference information is sent to the change detection head to obtain a binary change map.

[0047] The segmentation head consists of one convolutional layer, one batch normalization layer, one activation layer, one convolutional layer, and one upsampling layer (performing bilinear interpolation). The change detection head consists of one residual block layer, one convolutional layer, one batch normalization layer, one activation layer, and one upsampling layer (performing bilinear interpolation).

[0048] Step 6: By superimposing the two semantic segmentation maps on the binary change map, a semantic change map is finally generated.

[0049] Table 1 compares the results of semantic change detection on the SECOND dataset using the method of the present invention with the results of semantic change detection on the SECOND dataset using other semantic change detection methods.

[0050] Table 1 Comparison table of semantic change detection

[0051]

[0052] From the results in Table 1, we can see that the method of the present invention shows excellent performance on the SECOND dataset, and is significantly better than other semantic change detection methods in terms of average intersection-over-union ratio, separation kappa coefficient, F1 score and accuracy.

[0053] Figure 4 The term "our method" refers to the method of the present invention. Figure 4 It can be seen that the method of the present invention performs better in identifying the contour of the changed area and detecting subtle changes in complex scenes.

[0054] As shown in C2 in the figure, the method of the present invention effectively reduces the semantic recognition errors caused by the visual similarity between buildings and terrain in other methods, thereby accurately identifying the changing semantic categories. When compared with Bi-SRNet, which optimizes feature quality through a cross-attention mechanism, the method of the present invention still presents a better semantic segmentation effect.

[0055] from Figure 4 From the comprehensive analysis of Table 1, it can be seen that the method of the present invention significantly improves the detection results of the change detection subtask by using the high-resolution features provided by HRNet, which is reflected in the average intersection and union ratio index. At the same time, the semantic segmentation efficiency of the dual semantic decoding branch is also improved. This conclusion is verified in the separation kappa coefficient and F1 score indicators. This comprehensive improvement highlights the ability of the method of the present invention in providing high-precision change detection and semantic segmentation clarity.

[0056] Furthermore, despite the increase in model parameters due to the introduction of high-resolution features, the overall computational complexity remains manageable, as reflected in the number of floating-point operations per second. This efficiency is attributed to the proposed BiFAM module, which implements feature fusion internally, rather than extending from the encoder to the decoder as in previous UNet-like methods. This approach enhances the broad applicability of the proposed method.

[0057] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion, characterized in that: The following steps are involved: Step 1: First, the dual-time remote sensing images T1 and T2 are sent to the weight-sharing high-resolution network for multi-scale feature extraction. The multi-scale features of the remote sensing image T1 are extracted as follows: The multi-scale features extracted from the remote sensing image T2 are Step 2: Construct a bidirectional deep and shallow feature aggregation module to aggregate the multi-scale features obtained in step 1 Send it to the bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain features The multi-scale features obtained in step 1 Send it to the bidirectional deep and shallow feature aggregation module to integrate multi-scale features to obtain features Step 3: feature and Features Send it to the high-resolution difference extraction module to obtain difference features; Step 4: Construct an efficient channel attention module, and use the efficient channel attention module to refine the difference features obtained in step 3 to obtain refined difference information; Step 5: Build the segmentation head and change detection head to transform the features and Features The two semantic segmentation maps are sent to the segmentation head respectively; the difference information is sent to the change detection head to obtain a binary change map; Step 6: By superimposing the two semantic segmentation maps on the binary change map, a semantic change map is finally generated.

2. The remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2.1: Construct a bidirectional deep and shallow feature aggregation module, which includes 3 downsampling layers, 3 upsampling layers, and 1 Sc layer; Step 2.2: After a downsampling layer, Generate channel splicing Will After a layer of downsampling, Generate channel splicing Will After a layer of downsampling, Generated after channel splicing Will After a downsampling layer, Generate channel splicing Will After processing through layers of downsampling layers, Generate channel splicing Will After a layer of downsampling, Generated after channel splicing Step 2.3: Create the After one layer of upsampling Generate channel splicing Will After one upsampling layer, Generate channel splicing Finally After one upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated. The generated in step 2.2 After one layer of upsampling Generate channel splicing Will After one upsampling layer, Generate channel splicing Finally After one upsampling layer, Channel splicing is performed, and then after passing through the Sc layer, features with rich semantic information and precise spatial information are generated.

3. The remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion according to claim 2 is characterized in that: The downsampling layer includes a convolution layer, a batch normalization layer, an activation layer, and a convolution layer; The upsampling layer includes a convolution layer, a batch normalization layer, an activation layer, and an upsampling layer; The Sc layer consists of one convolutional layer, one batch normalization layer, one activation layer, and one convolutional layer.

4. The remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion according to claim 1 is characterized in that: The specific process in the high-resolution difference extraction module is as follows: Perform element-by-element subtraction and then add it to the feature and Features Perform channel stitching to obtain rough difference features.

5. The remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion according to claim 1, characterized in that: Step 4 specifically includes: the efficient channel attention module includes a global pooling layer, a one-dimensional convolution layer, and an activation layer; the difference features of step 3 are sent to the efficient channel attention module, firstly global average pooling is performed, and then the channel attention score is obtained by passing through the one-dimensional convolution layer and the activation layer in turn, and finally the channel attention score is multiplied by the difference feature to obtain the refined difference information.

6. The remote sensing image semantic change detection method based on high-resolution feature bidirectional fusion according to claim 1, characterized in that: The segmentation head includes one convolution layer, one batch normalization layer, one activation layer, one convolution layer, and one upsampling layer; the change detection head includes: one residual block layer, one convolution layer, one batch normalization layer, one activation layer, and one upsampling layer.

Images