Progressive addition network for multi-modal change detection based on hierarchical feature alignment

Abstract

The invention discloses a hierarchical feature alignment-based progressive addition network for multi-modal change detection, which comprises the following steps of: 1, performing hierarchical processing on two dual-tense images, namely two unmanned aerial vehicle images and a satellite image, through a decoder, and effectively acquiring difference features for hierarchical features between layers by adopting an HFAM (High Frequency Administration and Maintenance) method; and 2, based on the difference features output by the hierarchical feature alignment module, defining the difference features as DFi, then aggregating the difference features of different levels in a progressive manner to obtain fusion features, and finally, based on the fusion features of each level with the same scale, obtaining complementary aggregation features based on coordinated attention (CAm). And finally, the obtained complementary aggregation features are transmitted back to a decoder to obtain a prediction map. And step 3, in the HFA-PANet, a multi-supervision strategy composed of a series of lost items is designed.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image surface change detection, and in particular relates to a progressive addition network for multimodal change detection based on hierarchical feature alignment. Background Art

[0002] With globalization and technological progress, especially the rapid development of remote sensing technology, multimodal change detection (MCD) has been increasingly used in geographic information systems, environmental monitoring, urban planning and other fields. Multimodal change detection involves using image data from different sensors to identify and analyze changes in surface cover, which is particularly important in dealing with complex geographical environments and dynamically changing urban landscapes. Since the data captured by various sensors have different spectral characteristics and resolutions, how to effectively integrate and analyze these heterogeneous data to improve the accuracy and efficiency of change detection has become the focus of current research.

[0003] In this context, the Hierarchical Feature Alignment-based Progressive Addition Network provides a new perspective to address the challenges in multimodal change detection. This approach improves the accuracy and efficiency of change detection by integrating satellite image data at multiple spatial and temporal scales and using advanced deep learning techniques for feature extraction and alignment. The design of HFA-PANet takes into account the differences between images of different modalities, reduces domain shift through a hierarchical feature alignment module (HFAM), and achieves effective alignment of heterogeneous images. In addition, the progressive addition module (PAM) in the network can gradually aggregate feature information at different levels, enhancing the model's ability to distinguish between changed areas and backgrounds.

[0004] This deep learning-based method not only improves the ability to process large-scale image data, but also reduces dependence on expert experience and improves the level of automation. In practical applications, HFA-PANet can play an important role in many fields such as natural disaster warning, land use planning, and urban expansion monitoring, providing scientific and real-time data support for decision makers. With the further improvement of remote sensing technology and computing power, the progressive additive network based on hierarchical feature alignment is expected to play a greater role in the field of multimodal change detection in the future. Summary of the invention

[0005] The purpose of the present invention is to provide a progressive additive network for multimodal change detection based on hierarchical feature alignment, which uses advanced deep learning technology for feature extraction and alignment, thereby improving the accuracy and efficiency of change detection, and taking into account the differences between images of different modalities, reducing domain shift through a hierarchical feature alignment module (HFAM) to achieve effective alignment of heterogeneous images.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a progressive addition network for multimodal change detection based on hierarchical feature alignment, which is specifically implemented according to the following steps:

[0007] Step 1: The two dual-temporal images, the drone images and the satellite images, are processed in layers through a decoder, and then the HFAM method is used to effectively obtain the difference features between the hierarchical features of each layer;

[0008] Step 2: Based on the difference features output by the hierarchical feature alignment module, it is defined as DF i , and then aggregate the difference features of different levels in a progressive manner to obtain the fused features, and finally obtain the coordinated attention (CA)-based fusion features based on each level of the same scale. m )'s complementary aggregation features; finally, the obtained complementary aggregation features are input back to the decoder to obtain a prediction graph;

[0009] Step 3, in HFA-PANet, a multi-supervision strategy consisting of a series of loss terms is designed.

[0010] As a preferred technical solution of the present invention, in step 1, obtaining the difference feature is specifically:

[0011] First, we use the concept of multi-core maximum mean difference MK-MMD to realize feature alignment between heterogeneous BTIs. This method is based on and It can be expressed as:

[0012]

[0013] where φ(·) and Represent the feature mapping kernel function and mathematical expectation respectively; and Represents multiple kernel functions k and features and The distribution difference between the regenerative Hilbert space;

[0014] Then convolve the result of MK-MMD to get

[0015] Finally, the features to be aligned and The absolute value of the difference is calculated to obtain the difference feature; for this, the feature representation can be improved by deploying coordinate attention to consider the spatial position and channel relationship, and the difference feature can be generated in the proposed HFAM It can be expressed as:

[0016]

[0017] where f CA (·) and |·| represent coordinate attention and absolute operation, and DF is the difference feature between the aligned features.

[0018] As a preferred technical solution of the present invention, in step 2, the difference features obtained by aligning the two encoders according to the hierarchical features are classified and defined according to their different scales, and are divided into DF from large to small. i , where i∈[1,5] represents the i-th layer of the encoder.

[0019] As a preferred technical solution of the present invention, step 2 is specifically performed as follows:

[0020] Step 2.1, design PAM, i.e. progressive addition module, to convert DF i Perform layer-by-layer aggregation to obtain fusion features

[0021] Step 2.2: Fusion features Based on, we gradually aggregate the feature differences of each layer again, and based on the features of each layer with the same scale, we use the following convolution formula

[0022]

[0023] Acquire based on coordinated attention (CA m , where m∈{5, 4, 3, 2, 1}) are complementary clustering features;

[0024] Step 2.3 The specific process is to convert CA m Input the decoder and perform operations with the multi-supervisory loss module to obtain the prediction map.

[0025] As a preferred technical solution of the present invention, in step 2.1, if DF 5 and DF 4 Aggregation, the aggregation formula used is:

[0026]

[0027] Among them, the Conv() and Up() formulas are convolution operations and upsampling operations based on bilinear interpolation, respectively.

[0028] As a preferred technical solution of the present invention, step 3 is specifically: the convolution block calculated by the actual image performs multi-scale supervision on the four feature alignment losses formed by HFAM, so as to optimize the network in the training stage, thereby improving the model's ability to detect heterogeneous bit changes.

[0029] As a preferred technical solution of the present invention, the step 3 is specifically:

[0030] Step 3.1 indicates that the two loss functions used for semantic segmentation and change detection tasks are quadratic cross entropy (BCE) and DICE loss, which are calculated as follows:

[0031]

[0032] Where Dice(T,P) is calculated as:

[0033]

[0034] Where T and P refer to the GT map and the prediction map respectively, and H×W is the size of the input image;

[0035] Step 3.2: In addition to the loss in step 3.1, the hierarchical domain alignment (HDA) loss is introduced during the training phase of HFA-PANet and hybrid multi-level (HML) loss To measure the distribution difference between the heterogeneous BTI features of each layer, It can be defined as:

[0036]

[0037] Where n represents the nth convolutional block in the encoder, and the range of n is [1, 5]. The encoder Res2Net includes five layers of convolutional blocks, so represents the distribution difference between the features of the n-th convolutional block in the encoder, which is expressed by the formula calculate;

[0038] Step 3.3, according to which we can minimize To reduce the feature distribution differences of heterogeneous BTI, so as to achieve hierarchical feature alignment of heterogeneous BTI, in addition, It is expressed as:

[0039]

[0040] Where m represents the prediction graph of the mth convolutional block in the decoder, The purpose is to provide multi-scale supervision to the network from multiple levels of prediction results to further enhance the performance of the model;

[0041] Step 3.4, based on the above four loss items, the final loss It can be defined as follows:

[0042]

[0043] α and β are used to adjust middle and proportion.

[0044] As a preferred technical solution of the present invention, in step 3.4, α=0.1 and β=0.1.

[0045] The beneficial effects of the present invention are as follows: the present invention is a progressive addition network for multimodal change detection based on hierarchical feature alignment. Compared with the existing technology, the mainstream methods currently used to implement MCD are divided into two types, one is to implement MCD based on structural relationships or features, and the other is to implement modal alignment between heterogeneous BTIs based on image transformation or regression, and implement MCD in a comparative manner. However, because they are mostly based on pixel points for detection, the consumption is too large for large-scale data sets, so these two methods are only applicable to small-scale multimodal transformation detection in most cases. This method is different from the above methods. Compared with the detection between pixel points, this method designs a hierarchical feature alignment module and a progressive addition module to collect and distinguish the transformations between large-scale data sets, which enhances the network's change representation ability, reduces the time wasted due to the detection object being too large, and can also well reflect the large modal differences in wide-area scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0047] Figure 1 It is a flowchart of the execution of a network model in the present invention using a multiple supervision strategy consisting of a series of loss items to optimize the training phase;

[0048] Figure 2 This is a representative visualization of the results of different methods on the HTCD dataset;

[0049] Figure 3 This is a comparison chart of representative visualization results of different methods on the Wuhan dataset. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, a progressive addition network for multimodal change detection based on hierarchical feature alignment of the present invention is implemented in the following steps:

[0053] Step 1: The two dual-temporal images, the drone images and the satellite images, are processed in layers through a decoder, and then the HFAM method is used to effectively obtain the difference features between the hierarchical features of each layer;

[0054] Step 2: Based on the difference features output by the hierarchical feature alignment module, it is defined as DF i , and then aggregate the difference features of different levels in a progressive manner to obtain the fused features, and finally obtain the coordinated attention (CA)-based fusion features based on each level of the same scale. m ), and finally input the obtained complementary aggregation features back to the decoder to obtain the prediction graph;

[0055] Step 3, in HFA-PANet, a multi-supervision strategy consisting of a series of loss terms is designed.

[0056] The present invention provides a multi-supervision strategy consisting of a series of loss items to optimize the network in the training phase, which is specifically implemented in the following steps: based on the encoder, the multimodal dual-time image is cropped into multiple small-sized image blocks, and HFAM is proposed to deploy it in the convolution block of each layer of the two encoders, and the hierarchical features are aligned to obtain the difference features at each layer and scale. In addition, by introducing coordinate attention, the difference features can be further enhanced by considering the spatial position and channel relationship representation. Based on the above HFAM, the outputted hierarchical difference features of each level are aggregated from deep to shallow in a progressive manner to obtain fusion features, and then convolution operations are performed based on the fusion features of each level with the same scale to obtain complementary aggregation features based on coordinated attention, and finally the obtained complementary aggregation features are output. At the same time, the actual image is subjected to a series of loss processing; then the synthesized convolution block is subjected to multiple supervision with the image after multi-level loss processing, and feedback is obtained by comparing the actual image with the synthesized prediction image, thereby optimizing the network in the training phase and improving the model's ability to detect heterogeneous bit changes.

[0057] The proposed method is based on a deep learning network for multimodal change detection (MCD), especially for heterogeneous bi-temporal images. The core of the method is a novel non-twin hierarchical feature alignment (HFAM) and progressive addition module (PAM) network, namely HFA-PANet.

[0058] The present invention can be used to process images from different sensor modalities, identify and compare changes therein, solve the problem that multi-modal images are difficult to directly compare to identify surface changes, and provide favorable technical support for applications such as surface cover change detection, natural disaster warning and emergency response, and natural resource protection and analysis. The core idea of ​​HFA-PANet is to improve the performance of change detection through hierarchical feature alignment and progressive feature aggregation.

[0059] The specific contents include: extracting the difference features of the original image, progressively aggregating the difference features, and convolution operation on the fusion features. The original image is input into the encoder to obtain the difference features through the hierarchical feature extraction module, and then the fusion features obtained by the progressive addition module are input into the decoder and the constructed loss function is jointly operated to output the prediction map, thereby realizing multimodal image change detection.

[0060] This embodiment selects the method of the present invention and the similar task method for comparison, and illustrates the effect of the multimodal detection method of the present invention.

[0061] During the experiment, the present invention uses two large-scale public MCD databases (HTCD dataset and Wuhan dataset) to test the performance of various models.

[0062] In the experiment, the performance of the model was evaluated using indicators such as precision, recall, F1-Score, and intersection over union. Compared with other deep learning-based methods (FC-EF, FC-Siam-Conc, Fc-Siam-Diff, STANet, LGPNet, SUNet, MSCDNet, and DA-MSCDNet), HFA-PANet showed superior performance.

[0063] On the HTCD dataset, HFA-PANet outperforms other methods in all four evaluations. In the experiment, we selected some popular and latest methods to compare with our HFA-PANet. The results are shown in Table 1, Table 2, Figure 2 , Figure 3 shown.

[0064] Table 1

[0065] Quantitative comparison of different methods on the HTCD dataset (%). The best and second best results are shown in bold and underlined.

[0066]

[0067]

[0068] Table 2

[0069] Quantitative comparison of different methods on the Wuhan dataset (%). The best and second best results are shown in bold and underlined.

[0070]

[0071] Example 2

[0072] like Figure 1 As shown, different from Example 1, in Example 2, in a progressive addition network for multimodal change detection based on hierarchical feature alignment of the present invention, in step 1, obtaining the difference feature is specifically:

[0073] Step 1.1: Input the collected drone images and satellite images in any different time states into two encoders to obtain and and The features output by each convolutional block in the encoder, where C, H, and W represent the number of channels, height, and width of the feature map, respectively;

[0074] Step 1.2: Take the and Input into two convolution blocks to get output features Two of the convolutional blocks are composed of three convolutions respectively;

[0075] The next step is to use the concept of multi-core maximum mean difference MK-MMD to achieve feature alignment between heterogeneous BTIs. This method is based on and It can be expressed as:

[0076]

[0077] where φ(·) and Represent the feature mapping kernel function and mathematical expectation respectively, and Represents multiple kernel functions k and features and The distribution difference between the regenerating Hilbert space,

[0078] Then convolve the result of MK-MMD to get

[0079] Finally, the features to be aligned and The absolute value of the difference is calculated to obtain the difference feature. For this, the feature representation can be improved by deploying coordinate attention to consider the spatial position and channel relationship, and the difference feature can be generated in the proposed HFAM It can be expressed by the formula:

[0080]

[0081] where f CA (·) and |·| represent coordinate attention and absolute operation, and DF is the difference feature between the aligned features.

[0082] Example 3

[0083] like Figure 1 As shown, different from Example 2, in Example 3, a progressive addition network for multimodal change detection based on hierarchical feature alignment of the present invention, in step 2, the difference features obtained by aligning two encoders according to hierarchical features are classified and defined according to their different scales, and are divided into DF from large to small. i , where i∈[1,5] represents the i-th layer of the encoder.

[0084] Example 4

[0085] like Figure 1 As shown, different from Example 3, in Example 4 of the present invention, a progressive addition network for multimodal change detection based on hierarchical feature alignment, step 2 is specifically performed as follows:

[0086] Step 2.1, design PAM, i.e. progressive addition module, to convert DF i Perform layer-by-layer aggregation to obtain fusion features

[0087] Step 2.2: Fusion features Based on, we gradually aggregate the feature differences of each layer again, and based on the features of each layer with the same scale, we use the following convolution formula

[0088]

[0089] Acquire based on coordinated attention (CA m , where m∈{5, 4, 3, 2, 1}) are complementary clustering features;

[0090] Step 2.3 The specific process is to convert CA m Input the decoder and perform operations with the multi-supervisory loss module to obtain the prediction map.

[0091] Example 5

[0092] like Figure 1As shown, different from Example 4, in Example 5 of the present invention, a progressive addition network for multimodal change detection based on hierarchical feature alignment, step 3 is specifically as follows:

[0093] Step 3.1 indicates that the two loss functions used for semantic segmentation and change detection tasks are quadratic cross entropy (BCE) and DICE loss, which are calculated as follows:

[0094]

[0095] Where Dice(T,P) is calculated as:

[0096]

[0097] Where T and P refer to the GT map and the prediction map respectively, and H×W is the size of the input image;

[0098] Step 3.2: In addition to the loss in step 3.1, the hierarchical domain alignment (HDA) loss is introduced during the training phase of HFA-PANet and hybrid multi-level (HML) loss To measure the distribution difference between the heterogeneous BTI features of each layer, It can be defined as:

[0099]

[0100] Where n represents the nth convolutional block in the encoder, and the range of n is [1, 5]. The encoder Res2Net includes five layers of convolutional blocks, so represents the distribution difference between the features of the n-th convolutional block in the encoder, which is expressed by the formula calculate;

[0101] Step 3.3, according to which we can minimize To reduce the feature distribution differences of heterogeneous BTI, so as to achieve hierarchical feature alignment of heterogeneous BTI, in addition, It is expressed as:

[0102]

[0103] Where m represents the prediction graph of the mth convolutional block in the decoder, such as Figure 1 As shown, The purpose is to provide multi-scale supervision to the network from multiple levels of prediction results to further enhance the performance of the model;

[0104] Step 3.4, based on the above four loss items, the final loss It can be defined as follows:

[0105]

[0106] α and β are used to adjust middle and proportion.

[0107] Example 6

[0108] like Figure 1 As shown, different from Example 5, Example 6 of the present invention is a progressive additive network for multimodal change detection based on hierarchical feature alignment, with α=0.1 and β=0.1.

[0109] In the proposed HFA-PANet, in order to ensure that these two losses ( and ) will not converge due to excessive proportions, we set smaller α and β values, i.e., for example, α = 0.1 and β = 0.1. In short, Not only can the proposed HFA-PANET be effectively supervised, but also the domain shift can be reduced via the HDA loss, leading to effective MCDS.

[0110] The above description shows and describes several preferred embodiments of the invention, but as mentioned above, it should be understood that the invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention concept described herein through the above teachings or the technology or knowledge of the relevant field. Changes and modifications made by those skilled in the art should be within the scope of protection of the claims attached to the invention without departing from the spirit and scope of the invention.

Claims

1. Progressive Additive Networks for Multimodal Change Detection Based on Hierarchical Feature Alignment, characterized by: Follow the steps below to implement it: Step 1: The two dual-temporal images, the drone images and the satellite images, are processed in layers through a decoder, and then the HFAM method is used to effectively obtain the difference features between the hierarchical features of each layer; Step 2: Based on the difference features output by the hierarchical feature alignment module, it is defined as DF i , and then aggregate the difference features of different levels in a progressive manner to obtain the fused features, and finally obtain the coordinated attention (CA)-based fusion features based on each level of the same scale. m )'s complementary aggregation characteristics; Finally, the obtained complementary clustered features are fed back to the decoder to obtain the prediction graph; Step 3, in HFA-PANet, a multi-supervision strategy consisting of a series of loss terms is designed.

2. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 1, characterized in that: In step 1, obtaining the difference feature is specifically as follows: First, we use the concept of multi-core maximum mean difference MK-MMD to realize feature alignment between heterogeneous BTIs. This method is based on and It can be expressed as: where φ(·) and Represent the feature mapping kernel function and mathematical expectation respectively; and Represents multiple kernel functions k and features and The distribution difference between the regenerative Hilbert space; Then convolve the result of MK-MMD to get Finally, the features to be aligned and Perform absolute value operation of difference to obtain difference features; To this end, the feature representation can be improved by deploying coordinate attention to consider spatial position and channel relationship, and differential features can be generated in the proposed HFAM. It can be expressed as: where f CA (·) and |·| represent coordinate attention and absolute operation, and DF is the difference feature between the aligned features.

3. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 2, characterized in that: In step 2, the difference features obtained by aligning the two encoders according to the hierarchical features are classified and defined according to their different scales, and are divided into DF from large to small. i , where i∈[1,5] represents the i-th layer of the encoder.

4. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 3, characterized in that: The step 2 is specifically performed as follows: Step 2.1, design PAM, i.e. progressive addition module, to convert DF i Perform layer-by-layer aggregation to obtain fusion features Step 2.2: Fusion features Based on, we gradually aggregate the feature differences of each layer again, and based on the features of each layer with the same scale, we use the following convolution formula Acquire based on coordinated attention (CA m , where m∈{5, 4, 3, 2, 1}) are complementary clustering features; Step 2.3 The specific process is to convert CA m Input the decoder and perform operations with the multi-supervisory loss module to obtain the prediction map.

5. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 4, characterized in that: In step 2.1, if DF 5 and DF 4 Aggregation, the aggregation formula used is: Among them, the Conv() and Up() formulas are convolution operations and upsampling operations based on bilinear interpolation, respectively.

6. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 5, characterized in that: The step 3 is specifically as follows: the convolution block calculated by the actual image performs multi-scale supervision on the four feature alignment losses formed by HFAM, so as to optimize the network in the training phase, thereby improving the model's ability to detect heterogeneous bit changes.

7. The progressive additive network for multimodal change detection based on hierarchical feature alignment according to claim 6, characterized in that: The step 3 is specifically as follows: Step 3.1 indicates that the two loss functions used for semantic segmentation and change detection tasks are quadratic cross entropy (BCE) and DICE loss, which are calculated as follows: Where Dice(T,P) is calculated as: Where T and P refer to the GT map and the prediction map respectively, and H×W is the size of the input image; Step 3.2: In addition to the loss in step 3.1, the hierarchical domain alignment (HDA) loss is introduced during the training phase of HFA-PANet and hybrid multi-level (HML) loss To measure the distribution difference between the heterogeneous BTI features of each layer, It can be defined as: Where n represents the nth convolutional block in the encoder, and the range of n is [1, 5]. The encoder Res2Net includes five layers of convolutional blocks, so represents the distribution difference between the features of the n-th convolutional block in the encoder, which is expressed by the formula calculate; Step 3.3, according to which we can minimize To reduce the feature distribution differences of heterogeneous BTI, so as to achieve hierarchical feature alignment of heterogeneous BTI, in addition, It is expressed as: Where m represents the prediction graph of the mth convolutional block in the decoder, The purpose is to provide multi-scale supervision to the network from multiple levels of prediction results to further enhance the performance of the model; Step 3.4, based on the above four loss items, the final loss It can be defined as follows: α and β are used to adjust middle and proportion.

8. Progressive Additive Networks for Multimodal Change Detection Based on Hierarchical Feature Alignment, characterized in that In step 3.4, α=0.1 and β=0.1.

Images