A change detection method for multi-scale remote sensing images

Abstract

The present invention discloses a method for change detection of multi-scale remote sensing images, in which a dual-temporal remote sensing change detection network MSNet based on a multi-scale attention mechanism is designed. First, the encoder-decoder backbone is improved, a multi-scale attention mechanism is designed, and it is combined with the encoder to extract local multi-scale features in dual-temporal images. Secondly, a global multi-scale feature fusion module is designed to fuse global multi-scale features. Finally, a feature fusion module is designed to achieve effective fusion of dual-temporal features. The network structure can cope with the possible impact of changes in features such as size, shape and texture of the target, realize multi-scale remote sensing image change detection, and enhance the model's ability to capture features of various sizes and resolutions.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and more specifically, relates to a multi-scale remote sensing image change detection method. Background Art

[0002] Remote sensing change detection (RSCD) refers to identifying differences between two or more remote sensing images of the same area. Monitoring these differences is particularly important for understanding surface changes. Change detection using remote sensing images has been widely used in urban planning, disaster assessment, resource management and other fields.

[0003] Traditional RSCD methods rely on labeling change information through human visual recognition and manual outlining, which is not only labor-intensive and inefficient. In order to reduce the manpower burden, researchers began to explore the use of spectral information of remote sensing images to detect changes, using techniques such as change vector analysis (CVA), hybrid change detection, and principal component analysis (PCA). However, these methods usually require the selection of an optimal threshold in the decision-making stage, which not only depends on the specific scenario, but also is a cumbersome and time-consuming process. To overcome these limitations, researchers introduced machine learning techniques such as support vector machines, decision trees, and k-nearest neighbor algorithms into RSCD to achieve automated decision-making. However, these methods still rely heavily on manually extracted features and find it difficult to effectively capture high-level feature representations.

[0004] Thanks to the powerful feature extraction capability of deep learning architecture, existing deep learning-based RSCD methods have made some progress. Daudt et al. first applied the fully convolutional network (FCN) to the change detection task and constructed several U-Net-based networks, such as FC-early fusion (EF), FC-Siam-conc, and FC-Siam-diff. However, due to the single feature extraction at each scale, these networks are difficult to meet the needs of multi-scale changing objects. Inspired by this idea, Peng et al. improved the structure of UNet++ to improve the change extraction capability. However, due to the limited global feature extraction capability of CNN, Chen et al. combined CNN and Transformer to propose BIT, which effectively realized context modeling in the spatiotemporal domain. VCT uses a shared backbone for feature extraction based on BIT and uses graph neural networks to model structured information. Many researchers have also used Transformer to implement change monitoring to improve the global feature extraction capability of the model. However, these methods mainly focus on improving the model's ability to identify changes and non-changes, while ignoring the possible impact of feature changes such as size, shape, and texture of the target.

[0005] Although the existing remote sensing image change detection methods have made some progress, there is still room for improvement. Previous studies have mainly focused on how to improve the model's ability to identify changes and no changes, ignoring the possible impact of changes in features such as the size, shape, and texture of the target. However, the actual identified changes have various forms. Therefore, it is necessary to study the method of multi-scale feature extraction to improve the model's ability to capture features of various sizes and resolutions, so as to enhance the model's adaptability and robustness to different data conditions. Based on this, it is urgent to design a multi-scale remote sensing image change detection method. Summary of the invention

[0006] 1. Technical issues

[0007] Based on the above-mentioned technical defects, the present invention provides a change detection method for multi-scale remote sensing images. In addition to being able to cope with the possible impact of changes in features such as size, shape and texture of the target, the method can also improve the model's ability to capture features of various sizes and resolutions, thereby enhancing the model's adaptability and robustness to different data conditions.

[0008] (II) Technical solution

[0009] The present invention provides a method for multi-scale remote sensing image change detection. The method designs an MSNet network structure. The MSNet network structure specifically includes the following encoder-decoder backbone:

[0010] The encoder includes two groups of weight-sharing encoders, the first group of encoders consists of four encoding modules En_block1 to En_block4 connected in series, the second group of encoders consists of four encoding modules En_block5 to En_block 8 connected in series, the decoder includes two groups of weight-sharing decoders and a group of variable decoders, the first group of decoders consists of three decoding modules De_block1 to De_block3 connected in series, the second group of decoders consists of three decoding modules De_block7 to De_block9 connected in series, the variable decoder consists of three decoding modules De_block4 to De_block6 connected in series, and the nine decoding modules De_block1 to De_block9 are dual-input single-output decoding modules;

[0011] The output end of En_block4 is connected to the input end of De_block1 and the first dual-time image feature fusion module respectively, the output end of En_block8 is connected to De_block7 and the input end of the first dual-time image feature fusion module respectively, the output end of the first dual-time image feature fusion module is connected to the input end of De_block4, the output ends of En_block3 and En_block7 are jump-connected to the input ends of De_block1 and De_block7 respectively, the output ends of En_block2 and En_block6 are jump-connected to the input ends of De_block2 and De_block8 respectively, the output ends of En_block1 and En_block5 are connected to De_block3 and De_block7 respectively. The input end of ck9 is jump-connected, the output ends of De_block1 and De_block7 are connected to the two input ends of the second dual-time image feature fusion module, the output end of the second dual-time image feature fusion module is connected to the input end of De_block4, the output ends of De_block2 and De_block8 are connected to the two input ends of the third dual-time image feature fusion module, the output end of the third dual-time image feature fusion module is connected to the input end of De_block5, the output ends of De_block3 and De_block9 are connected to the two input ends of the fourth dual-time image feature fusion module, the output end of the fourth dual-time image feature fusion module is connected to the input end of De_block6, and the output end of De_block6 finally outputs the change feature after upsampling.

[0012] Preferably, the encoding modules En_block1 to En_block8 also design a multi-scale attention mechanism MSAM from a multi-scale perspective. H×W×C Four mapping layers U1∈R with different receptive field sizes are designed through multi-scale convolution. H×W×C , U2∈R H×W×C , U3∈R H×W×C , U4∈R H×W×C ,The conventional CBR module used in the mapping is a combination of the Conv basic convolution layer, the BN batch standard normalization layer and the ReLU activation layer. The convolution kernel sizes are 1×1, 3×3, 5×5, and 7×7 respectively. U1, U2, U3, and U4 are summed element by element to get U:

[0013] U=U1+U2+U3+U4

[0014] Next, the feature map U is simultaneously subjected to maximum pooling and average pooling to obtain and Then, the elements are summed after passing through the weight-sharing network MLP, and the weight vector M is generated through the Sigmoid function. C; where σ is the Sigmoid function, the MLP structure is a combination of Conv-ReLU-Conv, W0∈R C / r×C , W1∈R C×C / r , W0, W1 are the weights of MLP;

[0015]

[0016]

[0017]

[0018]

[0019] And combine U with the obtained channel attention weight M C (F) to obtain U′,

[0020]

[0021] Then, U′ enters the spatial attention module, where it is first obtained by global average pooling and global maximum pooling functions. and Then the Concat function is used to connect them to perform channel stacking operations, and finally the Sigmoid function is used to generate the weight vector M s ∈R 1×H×W , f 7×7 The filter is a 7×7 Conv convolution operation, and Concat is an image concatenation function;

[0022]

[0023]

[0024] M s (U′)=σ(f 7×7 (Concat[AvgPool(U′),MaxPool(U′)]))

[0025] And U′ and the obtained spatial attention weight M s (U′) multiplied to get U″

[0026]

[0027] in is element-wise multiplication, and U″ is the final output.

[0028] Preferably, the structure of the encoding modules En_block1 to En_block8 is as follows: Figure XAfter two mappings of the 2*CBR module and the residual mapping of the multi-scale attention mechanism MSAM, the shortcut connection of the residual network is realized, and finally the final encoding features are output through the ReLU activation layer.

[0029] Preferably, the structure of the decoding modules De_block1 to De_block9 is as follows: after upsampling, the first feature map X1 is concatenated with the second feature map X2 through a Concat function to map the output to the CBR module, and finally the decoding features are output, and the Conv convolution kernel in the CBR module is 3×3.

[0030] Preferably, the MSNet network structure further includes a global multi-scale feature fusion module, and the global multi-scale feature fusion module is located between En_block4 and De_block1 and between En_block8 and De_block7;

[0031] The structure of the global multi-scale feature fusion module is as follows: Figure X After being input into the CBS module with a convolution kernel of 3×3 and the CBS module with a convolution kernel of 5×5 respectively, the parallel output features are concatenated by the Concat function and input into the CBS module with a convolution kernel of 1×1. The output of the CBS module with a convolution kernel of 1×1 is shortcut-connected to the identity mapping of X, and the final fusion feature is output; among them, the CBS module is a combination of the Conv basic convolution layer, the BN batch standard normalization layer and the SiLU activation layer.

[0032] Preferably, the structures of the first dual-time image feature fusion module to the fourth dual-time image feature fusion module are specifically as follows:

[0033] Input dual-time images A, B∈R H×W×C , A and B perform feature exchange on the channel to generate X1,X2,X3,X4∈R H×W×C ; First, the global feature representation is obtained through the global average pooling operation in the spatial dimension, and then the correlation between channels is modeled through the MLP multi-layer perceptron, and the channel descriptor is generated through the Sigmoid activation function:

[0034] X′ n ∈R 1×1×C =sigmoid(MLP(AvgPool(X n )))

[0035] Then the four global features are concatenated in the second dimension to obtain Then the weight W is obtained by the softmax normalized exponential function in the second dimension:

[0036]

[0037]

[0038] W[:,i,:]∈R C×1×1

[0039] When N is equal to 4, the generated feature weights are multiplied element-by-element with the corresponding inputs and added to obtain the fused feature Y∈R H×W×C :

[0040] Y=W[:,1,:]⊙X1+W[:,2,:]⊙X2+…+W[:,N,:]⊙X N .

[0041] Preferably, the loss function calculation includes: performing bilinear downsampling on the change label with a scale factor of 0.5 to obtain a pseudo label. The overall loss consists of three parts. The loss of the change feature change mask is L change , the image loss at T1 is L T1 , the image loss at T2 is L T2 , the final loss function is expressed as:

[0042] Loss = L change +0.1*L T1 +0.1*L T2

[0043] L change , L T1 and L T2 The three loss functions are calculated using the same loss function, which is a combination of focal loss functions and dice coefficient loss functions; focal loss is expressed as:

[0044] L fl =-(1-p t ) γ log(p t )

[0045] where p t reflects the closeness to the true value of the ground truth category y, p t The larger the value, the closer it is to category y and the more accurate the classification. γ>0 is an adjustable factor. The dice coefficient loss is expressed as:

[0046]

[0047] Where |X∩Y| represents the intersection of X and Y sets, |X| and |Y| represent the segmented ground truth and prediction mask; L fl and L dice The final loss is obtained by combining:

[0048] L=L fl +L dice .

[0049] In another aspect, the present invention also discloses a multi-scale remote sensing image change detection system, comprising: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute any of the multi-scale remote sensing image change detection methods described above.

[0050] (III) Beneficial effects

[0051] Compared with the prior art, the multi-scale remote sensing image change detection method of the present invention has the following advantages:

[0052] (1) The present invention proposes a change detection method for multi-scale remote sensing images, in which a dual-temporal remote sensing change detection network (MSNet) based on a multi-scale attention mechanism is designed. First, a multi-scale attention mechanism is designed and integrated into the encoder to extract local multi-scale features in dual-temporal images. Secondly, a global multi-scale feature fusion module is designed to fuse global multi-scale features. Finally, a feature fusion module is designed to achieve effective fusion of dual-temporal features. The network structure can cope with the possible impact of changes in the size, shape, texture and other features of the target, realize multi-scale remote sensing image change detection, and improve the model's ability to capture features of various sizes and resolutions.

[0053] (2) In addition, the present invention has newly designed a multi-scale attention mechanism module (MSAttention) and a global multi-scale feature fusion module (MSFP), which can extract local multi-scale semantic feature information by embedding MSAM in each encoder and realize the fusion of global multi-scale features through MSFP. Such a design can better consider the characteristics of the target such as size, shape and texture, thereby improving the adaptability and performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall structure diagram of the MSNet network in the multi-scale remote sensing image change detection method of the present invention, and the modules in the figure are mainly based on features;

[0055] Figure 2 This is a diagram of the encoder-decoder backbone structure of the method for multi-scale remote sensing image change detection of the present invention, in which the modules are mainly encoder-decoder;

[0056] Figure 3 It is a structural diagram of the multi-scale attention mechanism of the present invention;

[0057] Figure 4 For the present invention Figure 2 Detailed structural diagram of the encoder and decoder in FIG. 1 ; wherein (a) is a single-input single-output encoding module, and (b) is a dual-input single-output decoding module;

[0058] Figure 5 It is a structural diagram of the global multi-scale feature fusion module of the present invention;

[0059] Figure 6 It is a structural diagram of the dual-time image feature fusion module of the present invention;

[0060] Figure 7 It is the overall workflow diagram of the loss function of the present invention;

[0061] Figure 8 The visualization results of the method of the present invention and other 7 algorithms tested on the LEVIR-CD dataset, where white, green, red, and black are true positive, false negative, false positive, and true negative, respectively, and (a)-(h) are the detection results on LEVIR-CD.

[0062] Fig. 9 The visualization results of the method of the present invention and other 7 algorithms tested on the MSRS-CD dataset. The white, green, red and black colors are true positive, false negative, false positive and true negative, respectively. (a)-(h) are the detection results on MSRS-CD.

[0063] Fig.10 This is the visualization result of the MSNet ablation experiment of the present invention. White, green, red, and black represent true positive, false negative, false positive, and true negative, respectively. (a)-(d) are the detection results on MSRS-CD. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0065] In order to solve the technical problem that the existing RSCD method ignores the possible impact of changes in characteristics such as size, shape and texture of the target, the present invention uses a newly designed MSNet network structure for a multi-scale remote sensing image change detection method.

[0066] 1. Overall structure of the network

[0067] The overall structure of MSNet is as follows Figure 1-2 As shown in Figure 1, the network includes two sets of weight-sharing encoders (four-layer encoding), two sets of weight-sharing decoders (three-layer decoding), and one set of variable decoders (three-layer decoding). Figure 2 The image processing structure diagram mainly based on codec, Figure 1 The structure is expressed by feature graphs. Figure 2 It is a structure expressed by codec modules. Figure 1 The yellow, purple, blue, and green cubes represent features, not encoders and decoders. D and × represent Figure 4 (b) shows a dual-input single-output decoding module. The encoding module acts on the yellow feature generation process. Figure 1 The intermediate decoding module D is the change decoding module, and its final output is the change feature Change mask, which is marked in blue to distinguish it from the green decoder feature. In addition, Figure 1 The yellow in the middle represents the encoded module (i.e. Figure 2 The red module of ) has 4 feature maps, Figure 2 The green three-layer decoding module corresponds to Figure 1 The D in , so there are 9 in total, all of which are decoding modules. Figure 2 The × in the Figure 1 The × in the figure are all dual-time image feature fusion modules.

[0068] See also Figure 1It can be seen that, first of all, the input of MSNet is a pair of remote sensing image graphs T1 and T2, which are respectively encoded into a pair of remote sensing image data with high-level abstract representations through two sets of weight-sharing encoders for change and non-change analysis. Next, for the output of the two sets of encoders, the features are learned and fused through the global multi-scale feature fusion module to better understand the features at different scales from the perspective of multiple scales to improve the performance of change detection. The two branch decoders decode the semantic features of the two moments, and the image features of the two temporal states in each layer are fused through the dual-temporal feature fusion module to capture the correlation information between the two temporal states. Finally, the change features generated by the change encoder are used to accurately detect the change information in the remote sensing image.

[0069] (II) Encoder-Decoder Backbone

[0070] The encoder-decoder backbone structure of the MSNet network is as follows Figure 2 As shown in the figure, dual-temporal features are extracted through weight-sharing encoder modules. These feature representations contain important information in the image, laying the foundation for the subsequent change detection task. After the encoding is completed, the change features are obtained through a series of feature fusion modules. The role of these feature fusion modules is to comprehensively consider the feature representations extracted by the encoder to better capture the change information. This helps the network to accurately restore the change information in the subsequent decoding stage. In the decoding stage, the network adopts 4 dual-temporal feature fusion modules x to fuse the dual-temporal encoder features and generate 4 change features. The role of these dual-temporal feature fusion modules is to comprehensively consider the feature representations of two moments to obtain more comprehensive and accurate change information. Subsequently, the network uses 3 series-connected change decoders De_block4~De_block6 to recover the change information from the change features decoded at each layer. These change decoders can adopt the structure corresponding to the encoder to restore the change features to the original change information in a reverse operation. Through such an encoder-decoder backbone structure, MSNet can effectively extract, fuse and restore dual-temporal features, thereby realizing accurate detection and analysis of change information in remote sensing images.

[0071] See also Figure 2 As shown, the encoder-decoder backbone of the MSNet network of the present invention includes the following improvements:

[0072] The encoder includes two groups of weight-sharing encoders, the first group of encoders consists of four encoding modules En_block1 to En_block4 connected in series, and the second group of encoders consists of four encoding modules En_block5 to En_block 8 connected in series. The decoder includes two groups of weight-sharing decoders and a group of variable decoders, the first group of decoders consists of three decoding modules De_block1 to De_block3 connected in series, the second group of decoders consists of three decoding modules De_block7 to De_block9 connected in series, the variable decoder consists of three decoding modules De_block4 to De_block6 connected in series, and the nine decoding modules De_block1 to De_block9 are dual-input and single-output decoding modules.

[0073] In addition, the input signal of En_block1 is image T1, the input signal of En_block5 is image T2, the output end of En_block4 is connected to the input end of De_block1 and the first dual-time image feature fusion module respectively, the output end of En_block8 is connected to the input end of De_block7 and the first dual-time image feature fusion module respectively, the output end of the first dual-time image feature fusion module is connected to the input end of De_block4, the output ends of En_block3 and En_block7 are jump-connected to the input ends of De_block1 and De_block7 respectively, the output ends of En_block2 and En_block6 are jump-connected to the input ends of De_block2 and De_block8 respectively, and the output ends of En_block1 and En_block5 are connected to De_bl The input ends of ock3 and De_block9 are jump-connected, the output ends of De_block1 and De_block7 are connected to the two input ends of the second dual-time image feature fusion module, the output end of the second dual-time image feature fusion module is connected to the input end of De_block4, the output ends of De_block2 and De_block8 are connected to the two input ends of the third dual-time image feature fusion module, the output end of the third dual-time image feature fusion module is connected to the input end of De_block5, the output ends of De_block3 and De_block9 are connected to the two input ends of the fourth dual-time image feature fusion module, the output end of the fourth dual-time image feature fusion module is connected to the input end of De_block6, and the output end of De_block6 finally outputs the change feature after upsampling, that is, Seg_out3 outputs the change mask Change mask, and the output ends of De_block3 and De_block9 output the segment masks Seg mask1 and Seg mask2 through Seg_out1 and Seg_out2 respectively.

[0074] In another embodiment, in the eight encoding modules En_block1 to En_block8 of the present invention, a multi-scale design is also performed. Figure 3 Multi-scale attention mechanism (MSAM). In order to obtain multi-scale information, for the input X∈R H×W×C Four mapping layers U1∈R with different receptive field sizes are designed through multi-scale convolution. H×W×C , U2∈R H×W×C , U3∈R H ×W×C , U4∈R H×W×C ,The conventional CBR module used in the mapping is a combination of the Conv basic convolution layer, the BN batch standard normalization layer and the ReLU activation layer. The convolution kernel sizes are 1×1, 3×3, 5×5, and 7×7 respectively. U1, U2, U3, and U4 are summed element by element to get U:

[0075] U=U1+U2+U3+U4

[0076] Next, in order to focus the information on the feature map on the channel, we enter the Channel Attention Module. In this module, we first perform maximum pooling and average pooling on the feature map to obtain and Then, the elements are summed after passing through the weight-sharing network MLP, and the weight vector M is generated through the Sigmoid function. C . Where σ is the Sigmoid function, the MLP structure is Conv-ReLU-Conv, W0∈R C / r×C , W1∈R C×C / r , W0, W1 are the weights of MLP.

[0077]

[0078]

[0079]

[0080] And combine U with the obtained channel attention weight M C (F) to obtain U′,

[0081]

[0082] Then, in order to compress and merge the global channel information, U′ enters the Spatial Attention Module. In the module, firstly, the global average pooling and global maximum pooling functions are used to obtain and Then the Concat function is used to connect them to perform channel stacking operations, and finally the Sigmoid function is used to generate the weight vector M s ∈R 1×H×W , where σ is the Sigmoid function, f 7×7 It indicates that the filter is a 7×7 Conv convolution operation, and Concat is an image splicing function.

[0083]

[0084]

[0085] M s (U′)=σ(f 7×7 (Concat[AvgPool(U′),MaxPool(U′)]))

[0086] And U′ and the obtained spatial attention weight M s (U′) is multiplied to get U″, where is element-wise multiplication, and U″ is the final output.

[0087]

[0088] In another embodiment, the MSNet encoder structure is as follows: Figure 4 As shown in Figure (a), En_block1 to En_block8 preferably adopt this encoding structure, which gradually reduces the spatial resolution of the feature map while increasing the abstraction level of the features through the stacking of multiple layers of convolutional modules, and embeds MSAM for feature enhancement, which helps to improve the network's sensitivity to features at different scales, thereby better capturing changing and non-changing information.

[0089] A two-input one-output decoder such as Figure 4 As shown in Figure (b), it corresponds to Figure 1 The 9 decoder modules D, i.e. De_block1 to De_block9, preferably adopt this encoding structure, where X1 represents a low-resolution feature map and X2 represents a high-resolution feature map. In order to identify the change features, the decoder adopts a feature fusion strategy. Specifically, the low-resolution feature map X1 is upsampled twice and fused with the high-resolution feature map X2. This design allows the network to better capture change information by fusing features of different resolutions while retaining spatial information. The feature map X1 represents the low-resolution fused feature or the decoder output feature, i.e. Figure 1 In the figure, the fused features (purple) or decoded features (green) before D, and the feature map X2 is preferred. Figure 1High-resolution encoder features (yellow), or feature maps with higher resolution than X1.

[0090] The structure of the encoding module is as follows: Figure X After two 2*CBR module mappings, Figure 3 After the residual mapping of the multi-scale attention mechanism MSAM is shown, the shortcut connection of the residual network is realized, and finally the final encoding features are output through the ReLU activation layer.

[0091] The structure of the decoding module is as follows: after upsampling, the first feature map X1 is concatenated with the second feature map X2 through the Concat function to map the output to the CBR module, and finally output the decoding features, wherein the Conv convolution kernel in the above CBR module is preferably 3×3.

[0092] (III) Global multi-scale feature fusion module

[0093] In another embodiment, considering that there is usually information of different scales in remote sensing images, covering aspects such as the microstructure and overall layout of objects. In order to enhance the network's ability to understand and process multi-scale data of remote sensing images and comprehensively consider information from different scales, a global multi-scale feature fusion module is designed. This module allows the network to learn features from multiple scales to capture image information more comprehensively. The structure of the global multi-scale feature fusion module is as follows: Figure 5 As shown in the figure. It uses CBS modules with different convolution kernel sizes to obtain feature maps of different receptive fields, thereby obtaining feature information of different scales. These feature maps are then fused together to learn feature information at different scales. This design enables the network to better understand and process information of different scales in remote sensing images, thereby improving the performance of the network in change detection tasks.

[0094] See also Figure 1 and Figure 5 It can be seen that Figure 1 The purple square is the Figure 5 The fusion features processed by the global multi-scale feature fusion module are located at the output end of the two sets of weight-shared encoders. The global multi-scale feature fusion module is specifically located between En_block4 and De_block1 and between En_block8 and De_block7, and the two fusion features obtained by processing are used as input features of the first dual-time image feature fusion module.

[0095] The structure of the global multi-scale feature fusion module is as follows: Figure XAfter being input into the CBS module with a convolution kernel of 3×3 and the CBS module with a convolution kernel of 5×5 respectively, the parallel output features are concatenated by the Concat function and input into the CBS module with a convolution kernel of 1×1. The output of the CBS module with a convolution kernel of 1×1 is shortcut-connected to the identity mapping of X, and the final fusion feature is output; among them, the CBS module is a combination of the Conv basic convolution layer, the BN batch standard normalization layer and the SiLU activation layer.

[0096] (IV) Dual-time image feature fusion module

[0097] In another embodiment, the existing change detection model usually adopts a simple subtraction or convolution enhancement method. The subtraction fusion method is usually easily interfered by noise and cannot accurately fuse the dual-time image features. The convolution enhancement method is prone to error accumulation and cannot effectively achieve feature fusion. For this reason, the present invention proposes a new dual-time image fusion method, which effectively uses the features of the two-time images to achieve feature fusion. The dual-time image feature fusion module is as follows: Figure 6 shown.

[0098] The structure of the dual-time image feature fusion module is as follows:

[0099] Input dual-time images A, B∈R H×W×C , A and B perform feature exchange on the channel to generate X1,X2,X3,X4∈R H×W×C First, the global feature representation is obtained through the global average pooling operation in the spatial dimension, and then the correlation between channels is modeled through the MLP multi-layer perceptron, and the channel descriptor is generated through the Sigmoid activation function:

[0100] X′ n ∈R 1×1×C =sigmoid(MLP(AvgPool(X n )))

[0101] Then the four global features are concatenated in the second dimension to obtain Then the weight W is obtained by the softmax normalized exponential function in the second dimension:

[0102]

[0103]

[0104] W[:,i,:]∈R C×1×1

[0105] Figure 6W1-W4 in the formula are the feature weights W[:,i,:] of each part of W, N is equal to 4, and the generated feature weights are multiplied element by element with the corresponding input, and then added to obtain the fused feature Y∈R H×W×C :

[0106] Y=W[:,1,:]⊙X1+W[:,2,:]⊙X2+…+W[:,N,:]⊙X N

[0107] (V) Loss function calculation

[0108] The overall workflow of the loss function is as follows Figure 7 As shown in the figure, by bilinear downsampling the change label, the scale factor is 0.5, and the pseudo label is obtained. The overall loss consists of three parts. The loss of the change mask is L change , the image loss at T1 is L T1 , the image loss at T2 is L T2 , the final loss function is expressed as:

[0109] Loss = L change +0.1*L T1 +0.1*L T2

[0110] L change , L T1 and L T2 The three loss functions are calculated using the same loss function, which is a combination of focal loss functions and dice coefficient loss functions. Focal loss is expressed as:

[0111] L fl =-(1-p t ) γ log(p t )

[0112] where p t reflects the closeness to the true value of the ground truth category y, p t The larger the value, the closer it is to category y and the more accurate the classification. γ>0 is an adjustable factor. The dice coefficient loss is expressed as:

[0113]

[0114] Where |X∩Y| represents the intersection of X and Y sets, |X| and |Y| represent the segmented ground truth and prediction mask. fl and L dice The final loss is obtained by combining:

[0115] L=L fl +L dice

[0116] (VI) Experimental analysis

[0117] In order to verify the effect of the method of the present invention and the MSNet network, the following experimental analysis (a)-(c) was also performed:

[0118] (a) Experimental analysis of the LEVIR-CD dataset

[0119] The comparison results with the other seven models on the LEVIR-CD dataset are shown in Table 1. The experimental results show that MSNet performs well in multiple key performance indicators. Specifically, MSNet achieves a high score of 92.87% in precision, ranking first among all models, demonstrating its outstanding ability in accurately identifying the actual change area. In addition, MSNet's recall rate is 89.99%, second only to STANet, showing its excellent performance in comprehensively identifying real changes. In terms of the more comprehensive F1 score, MSNet leads with a score of 91.41%, 1.01% higher than the second-ranked Changeformer, effectively proving that it has achieved a good balance between precision and recall. In the intersection-over-union (IOU) indicator that evaluates the accuracy of the model's segmentation of the change area, MSNet leads with a score of 83.60%, further confirming its superior segmentation ability. In terms of overall accuracy, MSNet leads again with a high score of 99.14%, highlighting its strong advantage in overall image recognition capabilities. The outstanding performance of MSNet is due to the advanced technologies used in its model architecture, such as the multi-scale attention mechanism and the global multi-scale feature fusion module, which significantly improve the model's adaptability and recognition accuracy to complex remote sensing image changing scenes.

[0120] From the perspective of qualitative analysis, the inference results of the MSNet network are visualized and compared with the results of other advanced networks. Figure 8 For a clearer display, white, black, red, and green are used to represent true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), respectively. Figure 8 (a) to (h) show the detection results on the LEVIR-CD dataset.

[0121] Table 1

[0122] Quantitative comparison results with other SOTA models on the LEVIR-CD dataset.

[0123] Methods Precision(%) Recall(%) F1 score (%) IOU(%) OA (%) FC-EF 86.91 80.17 83.40 71.53 98.39 FC-Siam-Di 89.53 83.31 86.31 75.92 98.67 FC-Siam-Conc 91.99 76.77 83.69 71.96 98.49 STANet 83.81 91.00 87.26 77.40 98.66 BIT 89.24 89.37 89.31 80.68 98.92 ChangeFormer 92.05 88.80 90.40 82.48 99.04 HANet 91.21 89.36 90.28 82.27 99.02 MSNet 92.87 89.99 91.41 83.60 99.14

[0124] (b) Experimental analysis of the MSRS-CD dataset

[0125] The present invention has built a new benchmark dataset MSRS-CD. The change targets of this dataset have richer scale features and multiple categories such as new buildings, suburban expansion, vegetation changes, and road construction. The comparison results of the present invention and other 7 models on the MSRS-CD dataset are shown in Table 2. MSNet outperforms the other 7 advanced algorithms in five indicators: precision, recall, F1score, IOU, and OA, which are 76.79%, 74.73%, 75.74%, 59.80%, and 93.03% respectively. Precision is 0.58% higher than the second place HANet, Recall is 2.31% higher than the second place BIT algorithm, F1 score is 1.63% higher than the second place BIT algorithm, IOU is 0.87% higher than the second place BIT, and OA is 0.9% higher than the second place HANet, which further verifies its significant advantages in change detection tasks.

[0126] The model is analyzed from a qualitative perspective. Specifically, the proposed MSNet network inference results are visualized and compared with the advanced network visualization. The visualization results on the MSRS-CD dataset are shown in Figure 2. Fig. 9 For better visualization, white, black, red, and green are used to represent TP, TN, FP, and FN, respectively. Fig. 9 (a)-(h) are the detection results on MSRS-CD. The detection results reflect the superiority of MSNet, which is due to the ingenious design of the MSNet network. Through the twin network and the dual-time image feature fusion module, the image features of the current moment and the historical moment can be fused, combining spatial and temporal information, so as to more comprehensively describe the changes in the ground objects. By calculating the attention weights at different scales, the model can more flexibly focus on and emphasize features at different scales, thereby improving the adaptability and effectiveness of the model for feature representation at various scales. This mechanism enables the model to more comprehensively capture the semantic information and spatial structure of the target, thereby improving the performance and robustness of the model. By constructing feature pyramids of multiple scales through the global multi-scale feature fusion module, the model can extract and represent features on feature maps of different resolutions, so that the model can more comprehensively and accurately describe the performance of the target at different scales.

[0127] Table 2 Quantitative comparison results with other SOTA models on the MSRS-CD dataset.

[0128]

[0129]

[0130] (c) Ablation experiment

[0131] The introduction of MSAM and MSFP significantly improves the accuracy of change detection. To verify their effectiveness, ablation experiments of MSNet are designed. In these experiments, Base represents the basic model without MSAM and MSFP. The Base+MSAM model is created as the second baseline by introducing the MSAM module, and Base+MSFP represents the model with MSFP integrated.

[0132] As can be seen from Table 3, the introduction of MSAM layer and MSFP block can improve the model performance on LEVIR-CD dataset. Specifically, after adding MSAM layer, the Precision, recall and F1 score of the dataset are improved by 1.96%, 3.29% and 2.06%, respectively. This shows that MSAM improves the model's perception of information of different scales, thereby enhancing the effect of processing multi-scale input data. In addition, MSFP block can improve the Precision, recall and F1 score of the dataset by 1.17%, 2.05% and 1.36%, respectively. This shows that MSFP block can provide multi-level and multi-scale feature representation, which helps the model capture the multi-scale information of the target at different scales. It is worth noting that compared with the Base model, MSNet, which integrates MSAM layer and MSFP block, improves the Precision, recall and F1 score by 3.20%, 3.26% and 2.65% on LEVIR-CD dataset. The huge improvement of MSNet not only further proves the effectiveness of MSAM block and MSFP layer, but also proves the gain effect of their combination.

[0133] Fig.10The visualization results of the ablation experiment of MSNet on the LEVIR-CD dataset are shown, where white, green, red, and black represent true positive, false negative, false positive, and true negative, respectively. Compared with the baseline model, both the second and third baselines can reduce the underreporting and false positive in the change map. Therefore, by improving the hidden layer feature extraction capability through the MSAM layer and further enhancing the expression of effective information in combination with the MSFP block, MSNet can effectively extract more accurate change maps, further verifying the view that MSNet can greatly improve the completeness and accuracy of the change results through the MSAM layer and MSFP block.

[0134] Table 3

[0135] MSNet ablation experiment results on the LEVIR-CD dataset.

[0136] Method Precision(%) Recall(%) F1 score (%) Base 89.67 86.73 88.76 Base+MSAM 91.63 90.02 90.82 Base+MSFP 90.84 88.78 90.12 MSNet 92.87 89.99 91.41

[0137] It can be seen that the present invention proposes a multi-scale remote sensing change detection network named MSNet. Through the twin network and dual-time image feature fusion module, the image features of the current moment and the historical moment can be fused, thereby combining spatial and temporal information to more comprehensively describe the changes in the ground objects. By introducing a multi-scale attention mechanism and global multi-scale feature fusion, the effective extraction of change features at different scales is achieved.

[0138] The detection method of the present invention described above can be converted into software program instructions, and can be implemented by running a software analysis system including a processor and a memory, or can also be implemented by computer instructions stored in a non-transitory computer-readable storage medium.

[0139] Finally, the method of the present invention is only a preferred embodiment and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting changes in multi-scale remote sensing images, characterized in that: The method designs a MSNet network structure, which specifically includes the following encoder-decoder backbone: The encoder includes two groups of weight-sharing encoders, the first group of encoders consists of four encoding modules En_block1 to En_block4 connected in series, the second group of encoders consists of four encoding modules En_block5 to En_block 8 connected in series, the decoder includes two groups of weight-sharing decoders and a group of variable decoders, the first group of decoders consists of three decoding modules De_block1 to De_block3 connected in series, the second group of decoders consists of three decoding modules De_block7 to De_block9 connected in series, the variable decoder consists of three decoding modules De_block4 to De_block6 connected in series, and the nine decoding modules De_block1 to De_block9 are dual-input single-output decoding modules; The output end of En_block4 is connected to the input end of De_block1 and the first dual-time image feature fusion module respectively, the output end of En_block8 is connected to De_block7 and the input end of the first dual-time image feature fusion module respectively, the output end of the first dual-time image feature fusion module is connected to the input end of De_block4, the output ends of En_block3 and En_block7 are jump-connected to the input ends of De_block1 and De_block7 respectively, the output ends of En_block2 and En_block6 are jump-connected to the input ends of De_block2 and De_block8 respectively, the output ends of En_block1 and En_block5 are connected to De_block3 and De_block7 respectively. The input end of ck9 is jump-connected, the output ends of De_block1 and De_block7 are connected to the two input ends of the second dual-time image feature fusion module, the output end of the second dual-time image feature fusion module is connected to the input end of De_block4, the output ends of De_block2 and De_block8 are connected to the two input ends of the third dual-time image feature fusion module, the output end of the third dual-time image feature fusion module is connected to the input end of De_block5, the output ends of De_block3 and De_block9 are connected to the two input ends of the fourth dual-time image feature fusion module, the output end of the fourth dual-time image feature fusion module is connected to the input end of De_block6, and the output end of De_block6 finally outputs the change feature after upsampling; The encoding modules En_block1 to En_block8 also design a multi-scale attention mechanism MSAM from a multi-scale perspective. H×W×C Four mapping layers U1∈R with different receptive field sizes are designed through multi-scale convolution. H ×W×C , U2∈R H×W×C , U3∈R H×W×C , U4∈R H×W×C ,The conventional CBR module used in the mapping is a combination of the Conv basic convolution layer, the BN batch standard normalization layer and the ReLU activation layer. The convolution kernel sizes are 1×1, 3×3, 5×5, and 7×7 respectively. U1, U2, U3, and U4 are summed element by element to get U: U=U1+U2+U3+U4 Next, the feature map U is simultaneously subjected to maximum pooling and average pooling to obtain and Then the elements are summed after the weight-sharing network MLP, and the channel attention weight M is generated through the Sigmoid function. C (U); where σ is the Sigmoid function, the MLP structure is a combination of Conv-ReLU-Conv, and W0∈R C / r×C , W1∈R C×c / r , W0, W1 are the weights of MLP; And combine U with the obtained channel attention weight M C (U) to obtain U′, Then, U′ enters the spatial attention module, where it is first obtained through global average pooling and global maximum pooling functions and Then the Concat function is used to connect them to perform channel stacking operations, and finally the Sigmoid function is used to generate the spatial attention weight M s (U′)∈R 1×H×W , f 7×7 The filter is a 7×7 Conv convolution operation, and Concat is an image concatenation function; M s (U′)=σ(f 7×7 (Concat[AvgPool(U′),MaxPool(U′)])) And U′ and the obtained spatial attention weight M s (U′) multiplied to get U″ in is element multiplication, U″ is the final output; The MSNet network structure also includes a global multi-scale feature fusion module, which is located between En_block4 and De_block1 and between En_block8 and De_block7; The structure of the global multi-scale feature fusion module is specifically as follows: after the feature map X is respectively input into the CBS module with a convolution kernel of 3×3 and the CBS module with a convolution kernel of 5×5, the parallel output features are concatenated by the Concat function and input into the CBS module with a convolution kernel of 1×1, the output of the CBS module with a convolution kernel of 1×1 is shortcut-connected with the identity mapping of X, and the final fusion feature is output; wherein the CBS module is a combination of a Conv basic convolution layer, a BN batch standard normalization layer, and a SiLU activation layer.

2. The method for multi-scale remote sensing image change detection according to claim 1, characterized in that: The structure of the encoding modules En_block1 to En_block8 is specifically as follows: after the feature map X is mapped twice by the 2*CBR module and then by the residual mapping of the multi-scale attention mechanism MSAM, a shortcut connection of the residual network is realized, and finally the final encoding feature is output through the ReLU activation layer.

3. The method for multi-scale remote sensing image change detection according to claim 2, characterized in that: The structure of the decoding modules De_block1 to De_block9 is specifically as follows: after upsampling, the first feature map X1 is concatenated with the second feature map X2 through the Concat function to map the output to the CBR module, and finally the decoding features are output. The Conv convolution kernel in the CBR module is 3×3.

4. The method for multi-scale remote sensing image change detection according to any one of claims 1 to 3, characterized in that: The loss function calculation includes: by bilinear downsampling the change label, the scale factor is 0.5, and the pseudo label is obtained. The overall loss consists of three parts. The loss of the change feature change mask is L change , the image loss at T1 is L T1 , the image loss at T2 is L T2 , the final loss function is expressed as: Loss=L change +0.1*L T1 +0.1*L T2 L change , L T1 and L T2 The three loss functions are calculated using the same loss function, which is a combination of focal loss functions and dice coefficient loss functions; focal loss is expressed as: L fl =-(1-p t ) γ log(p t ) where p t reflects the closeness to the true value of the ground truth category y, p t The larger the value, the closer it is to category y and the more accurate the classification. γ>0 is an adjustable factor. The dice coefficient loss is expressed as: Where |X∩Y| represents the intersection of X and Y sets, |X| and |Y| represent the segmented ground truth and prediction mask; L fl and L dice The final loss is obtained by combining: L=L fl +L dice .

5. A multi-scale remote sensing image change detection system, characterized in that: include: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the multi-scale remote sensing image change detection method as described in any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the multi-scale remote sensing image change detection method as described in any one of claims 1 to 4.

Images