Remote sensing image change detection network for multi-scale feature extraction and information mining

Abstract

The invention belongs to the field of remote sensing image change detection, and particularly relates to a remote sensing image change detection network for multi-scale feature extraction and information mining. The invention proposes a change detection network MSFMF-CDNet for multi-scale feature information mining fusion, and the method comprises the steps: firstly, designing a selective convolution attention mechanism SCBAM module to improve the distinguishability of multi-scale features based on the observation of different levels of information displayed by hierarchical features; then, a CCSAM module is proposed to refine multi-scale feature global information, and finally, advanced features are utilized to guide a multi-scale fusion HFGFFM module to improve the ability of the model to identify objects of different sizes. Experiments on two common optical remote sensing image CD data sets of LEVIR-CD, WHU-CD and CDD show that compared with other common methods, stronger remote sensing image change detection CD performance is realized.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image change detection, and specifically is a remote sensing image change detection network for multi-scale feature extraction and information mining. Background Art

[0002] Remote sensing image change detection (CD) involves identifying changes in images of the same geographic location taken at different time periods. The availability of a large number of high spatial resolution (HR) and very high resolution (VHR) remote sensing images provides a valuable opportunity to observe ground targets in more detail. Therefore, the application of remote sensing image change detection is becoming more and more widespread, and is applied in many aspects, including land resource management, urban development planning, and disaster assessment.

[0003] Traditional methods for change detection in remote sensing images include algebra-based methods, transformation-based methods, and advanced change detection models. Algebra-based methods generate change maps using basic mathematical calculations such as image differences, image ratios, and change vector analysis (CVA). They obtain remote sensing image change detection (CD) results through threshold segmentation. Transformation-based methods such as principal component analysis (PCA) and triangular cap transform (TCT) transform the original multi-temporal remote sensing images to reduce redundant information and analyze key information for remote sensing image change detection. Advanced change detection models focus on converting image reflectance into physical parameters. Although traditional remote sensing image change detection methods are generally easy to implement and understand, their threshold selection relies on the experience of researchers and may not provide a complete change matrix. Therefore, traditional remote sensing image change detection methods have difficulties in extracting robust features from complex high-resolution satellite image data.

[0004] Convolutional neural networks (CNNs) are widely used in current deep learning-based remote sensing image change detection (CD) algorithms, which can skillfully capture local features of images through convolution operations. These methods usually adopt the Siamese CNNs framework to extract features from dual-temporal optical images, ensuring wide applicability. They can be divided into three categories: pure convolutional models, convolutional variants, and models with attention mechanisms. Pure convolutional models such as FC-EF, FC-Siam-conc, and FC-Siam-diff use fully convolutional networks (FCNs) in CD, which have better feature learning and expression capabilities than traditional methods. However, their reliance on limited receptive fields limits the ability to effectively learn change feature information under different imaging conditions, which may lead to misclassification of true and false changes. Convolutional variant models such as SR-FCN and PGA-SiamNet address this limitation by adopting dilated convolutions and atrous spatial pyramid pooling to expand the receptive field. However, dilated convolutions may affect the continuity of the data, while pyramid pooling may blur the position details of the input image, thereby affecting the ability to accurately model the location and spatial relationship of objects. Models containing attention layers such as IFNet, DSAMNet, STANet, and DASNet enhance global context modeling in bi-temporal images. IFNet and DSAMNet integrate spatial and channel attention modules to aggregate global information and generate more unique bi-temporal features. STANet and DASNet use self-attention modules to capture global dependencies in bi-temporal features extracted by deep CNNs, thereby enhancing the ability to distinguish changing regions. Although the attention mechanism method helps to improve the global feature modeling ability of CNN-based CD methods, the size of the input data and matrix multiplication operations will also lead to a sudden increase in computational complexity, so it is necessary to balance the performance and complexity of the model.

[0005] Remote sensing image change detection (CD) has been rapidly developing in the field of vision tasks. Transformer models and their derivatives have been widely used in the image field, including applications such as image classification and object detection. Taking advantage of the transfer learning ability of model tasks, these models can also be used for remote sensing image change detection tasks through Siamese two-stream structures. In vision tasks, the application of transformer models is generally divided into two categories: pure transformer model variants and models integrating CNN and transformer. Pure transformer model variants, such as ViT, PvT, and Changeformer, are specifically designed for image classification tasks, and the effect on image classification tasks is amazing, but the global uniform representation obtained at different stages is redundant, resulting in limited feature richness. The pyramid structure is introduced in Pvt to supplement the multi-scale feature representation, making the transformer applicable to the task of object detection image segmentation. Similarly, Changeformer adopts a pyramid structure through downsampling for remote sensing image change detection tasks. These pure transformer models use self-attention modules to emphasize global image representation, but directly label the input image through hard segmentation, which may ignore local features such as edges and lines. Models integrating CNN and transformer, such as BIT, MSTDSNet, and ICIF-Net, demonstrate the ability to model both local and global features. These models can be divided into serial and parallel configurations based on their integration methods. For example, BIT serially integrates CNN and transformer modules in a single-scale manner to capture global context from high-level CNN features. However, the coarse-grained nature of the input semantic features in BIT may limit its effectiveness in detecting small-scale changes. MSTDSNet combines CNN with Swin transformers serially and integrates information in a multi-scale manner for change detection in remote sensing images of urban areas. ICIF-Net adopts a dual-branch structure, in which CNNs independently extract multi-scale local features, while transformers process multi-scale global features and integrate them through an attention-based cross-scale fusion module to enhance the exploration of global and local information. These CNN-transformer integration methods demonstrate that local and global features should be modeled dependently under the premise of multi-scale integration, which can improve model performance. Summary of the invention

[0006] In order to solve the problem that a network that simply extracts and fuses local and global feature information does not make sufficient use of multi-scale feature local and global information, resulting in limitations in discrimination ability, the present invention provides a remote sensing image change detection network for multi-scale feature extraction and information mining.

[0007] The present invention is implemented by adopting the following technical scheme: a remote sensing image change detection network for multi-scale feature extraction and information mining, comprising a multi-scale CNN-Backbone encoder, a selective convolutional attention mechanism module, a cascaded cross self-attention mechanism module and an advanced feature-guided multi-scale fusion module, wherein the multi-scale CNN-Backbone encoder extracts different scale features of a dual-phase image, the different scale features of the dual-phase image are input into the selective convolutional attention mechanism module, the features of different stages are further enhanced, the selective convolutional attention mechanism module outputs an optimized and enhanced dual-phase feature image, the optimized and enhanced dual-phase feature image is input into the cascaded cross self-attention mechanism module, the cascaded cross self-attention mechanism module is used to refine the multi-scale feature global information, and new features are obtained, and finally, the advanced feature-guided multi-scale fusion module is used to decode the new features, and a classifier is applied to distinguish the changed features to generate a final change map.

[0008] The above-mentioned remote sensing image change detection network for multi-scale feature extraction and information mining, the multi-scale CNN-Backbone encoder adopts a twin framework, which is composed of two parallel sub-network residual networks with the same structure and shared parameters; the dual-phase images I1 and I2 are fed into the two parallel sub-network residual networks as inputs for processing, and the process of feature extraction of the dual-phase images I1 and I2 includes five stages, the first stage is composed of a convolutional layer BN and Relu, the second stage is composed of a maximum pooling layer and Layer1, and the remaining three stages are composed of Layer2, Layer3, and Layer4 respectively. Finally, different scale features of the dual-phase images are extracted in the five stages. t1 ,F t2 ,F t3 ,F t4 ,F t5 |t=1,2}.

[0009] The above-mentioned remote sensing image change detection network for multi-scale feature extraction and information mining, the selective convolution attention mechanism module is for the 1st, 2nd, and 3rd stage features of the different scale features of the input dual-phase image. i {i=1,2,3} uses spatial attention optimization enhancement to enhance the features F in stages 4 and 5. i {i=4,5} uses channel attention optimization enhancement;

[0010] The process of channel attention optimization and enhancement is as follows: i {i=4,5} and the corresponding channel attention weight map W i c Multiply them together to obtain the optimized and enhanced feature map F i c , channel attention weight map Wi c The calculation details are as follows:

[0011]

[0012] In the above example, AvgPool represents the average pooling operation, MaxPool represents the maximum pooling operation, Fc1 and Fc2 represent the fully connected layers, relu represents the activation function, σ represents the sigmoid function, and W i c {i=4,5} represents the channel attention weight map;

[0013] The process of spatial attention optimization and enhancement is as follows: First, the spatial attention weight map W is calculated by the spatial attention module. i s , then the feature map feature F i {i=1,2,3}F i With W i s Element-by-element multiplication is performed to achieve spatial optimization enhancement. The optimized and enhanced feature map can be expressed as F i s , spatial attention weight map W i s The calculation is as follows:

[0014] W i s =σ(Conv 7×7 (Concat(Avgpool(F i ),Maxpool(F i ))))), where Conv 7×7 represents a convolution operation, the convolution kernel size is 7×7, Concat represents a cascade operation, W i s {i=1,2,3} represents the spatial attention weight map;

[0015] In W i c and W i s After calculating, input feature F i The selective convolution attention mechanism module is optimized and enhanced as follows: i c =F i ·W i c , F i s =F i ·W i s , where · represents element-by-element multiplication; Fi c {i=4,5} and F i s {i=1,2,3} represents the optimized enhanced feature maps at each stage through the channel and spatial modules, respectively;

[0016] Finally, the optimized dual-phase feature image F is obtained. i t , F i t Including F i s and F i c , t=1,2.

[0017] The above-mentioned remote sensing image change detection network for multi-scale feature extraction and information mining, dual-phase feature image F i 1 and F i 2 Input to the cascade cross self-attention mechanism module, the cascade cross self-attention mechanism module refines the multi-scale feature global information, and obtains a new feature denoted as T i {i=1,2,3,4,5};

[0018] The specific operation is expressed as:

[0019] F i cat =Conv1×1(Concat(F i t1 ,F i t2 )),

[0020]

[0021] Q,K,V=W q (F i cat ),W k (Token i ),W v (Token i ),

[0022]

[0023] Where: F i cat It is the feature representation after cascade convolution, Token i For concise semantic representation, Q, K, and V are the process indicators of the attention mechanism, and concat is a cascade operation. is the matrix multiplication representation, W q ,Wk ,W v They are all linear transformation matrices, σ is the softmax normalization operation, and Attention is the specific attention calculation process.

[0024] The above-mentioned remote sensing image change detection network for multi-scale feature extraction and information mining, feature T i {i=1,2,3,4,5} generates the final change map F in the high-level feature-guided multi-scale fusion module pre The specific process is:

[0025]

[0026] F pre =σ(Conv1×1(Conv3×3(F))),

[0027] Among them, + represents the addition of elements at corresponding positions, Up represents the upsampling operation, Conv1*1 and Conv3*3 represent 1*1 convolution and 3*3 convolution operations respectively.

[0028] This paper proposes a remote sensing image change detection network MSFMF-CDNet for multi-scale feature information mining and fusion. First, based on the observation that hierarchical features display different levels of information, a selective convolutional attention mechanism (SCBAM) module is designed to improve the distinguishability of multi-scale features; then, a cascaded cross self-attention mechanism (CCSAM) module is proposed to refine the global information of multi-scale features. Finally, a high-level feature guided multi-scale fusion (HFGFFM) module is used to improve the model's ability to distinguish objects of different sizes. Experiments on two public optical remote sensing image CD datasets, LEVIR-CD, WHU-CD and CDD, show that a stronger remote sensing image change detection CD performance is achieved than other commonly used methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall structure diagram of the network of the present invention.

[0030] Figure 2 This is the structure diagram of the CNN-Backbone encoder.

[0031] Figure 3 is the residual structure diagram, a is the residual structure of direct add connection, and b is the residual structure of 1*1 convolution connection with a step size of s.

[0032] Figure 4 This is the SCBAM module structure diagram.

[0033] Figure 5 Representative visualization results of each CD method on CDD.

[0034] Figure 6 Representative visualization of the results on LEVIR-CD for each CD method.

[0035] Figure 7 This is a process visualization diagram of model feature extraction and the functions of each module. DETAILED DESCRIPTION

[0036] The present invention provides an improved hybrid network based on multi-scale and CNN and Transformer serial combination, called MSFMF-CDNet (A Change Detection Network for Multi-scale Feature Information Mining and Fusion), to fully utilize the multi-scale global-local information in optical remote sensing images for change detection tasks. In the proposed MSFMF-CDNet, a selective convolutional attention mechanism (SCBAM) module for multi-scale features is innovatively designed, called the (SCBAM) block, which strengthens the spatial dependency and channel dependency of CNN multi-scale features from the perspective of deep and shallow feature characteristics. Then, after the bi-temporal features are cascaded encoded, the cascaded cross self-attention mechanism (CCSAM) is used to extract global information from multi-scale features of different resolutions and refine the features. Finally, the (HFGFFM) module is used to make the deep features rich in global information guide the global position dependency of the shallow features. In this way, the multi-scale features containing global-local information in the encoder are fused in the feature decoder, emphasizing the significance of the change area. The proposed MSFMF-CDNet obtains semantic and detailed image representations from coarse-fine and global-local views, and correspondingly improves the robustness of recognition of varying regions of different sizes.

[0037] Network Model

[0038] The ability to extract local and global features is essential for remote sensing image change detection methods. Among many advanced CD methods, CNN has become the mainstream method for extracting local features of remote sensing images. At the same time, the existing Transformer module provides an effective method for extracting global features from remote sensing images. In practical applications, the size of the changed area often varies greatly, especially for remote sensing images with a relatively large imaging time span. Due to factors such as complex scene changes and imaging conditions, higher requirements are placed on the CD method. Many methods have proved that the combination of multi-scale deep and shallow features can improve the model detection effect, but a simple fusion strategy is not enough. To address these problems, the present invention proposes a new serial network that combines CNN and transformer to construct global information based on multi-scale local correlation features. The architecture of the proposed MSFMF-CDNet is shown as follows: Figure 1 As shown in the figure, the network mainly consists of three parts, namely, the encoder that implements basic semantic feature extraction with Siamese architecture, the encoder that uses the attention mechanism to achieve feature enhancement and improvement, and the decoder that guides multi-scale feature fusion with advanced features. First, the multi-scale CNN-Backbone encoder is used to extract preliminary multi-scale features of dual-phase optical remote sensing images. This part of multi-scale features contains the basic semantic background of the remote sensing target and the location details of different scales. Then, the multi-scale local-global feature representation is refined and enriched layer by layer in the feature enhancement and improvement encoder, which enhances the distinction between the changed and unchanged areas. Finally, the multi-scale feature fusion decoding is guided by advanced features, and a classifier composed of two 1×1 convolution operations is applied to distinguish the changed features, generating a change map with sharp details.

[0039] Feature extraction module

[0040] In MSFMF-CDNet, the encoder for feature extraction adopts a twin framework, which consists of two parallel sub-network residual networks with the same structure and shared parameters. Assuming that the optical remote sensing image I t ∈R H×W×3 , t=1,2, the two remote sensing images I1 and I2 acquired at the same geographical location correspond to the time before and after the change respectively. The residual network is mainly composed of a series of convolutional layers, residual blocks and maximum pooling operations. Specifically, the dual-phase images I1 and I2 are fed into the residual network as inputs, and the feature extraction process for I1 and I2 includes five stages, and the output feature maps of the five stages are as follows: Figure 2 As shown in BACKBONE, the first stage consists of a convolutional layer BN and Relu, the second stage consists of a maximum pooling layer and Layer1, and the remaining three stages consist of Layer2, Layer3, and Layer4 respectively. Layer consists of two forms of residual blocks. The residual blocks are as follows Figure 3 As shown. Finally, different scale features of the dual-phase image are extracted in five stages {F t1 ,F t2 ,F t3 ,F t4 ,F t5|t=1,2}, the features of these five stages are downsampled by {1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32} respectively, and their number of feature channels is {64, 64, 128, 256, 512}. In the extraction process, the front stages mainly extract more edge and texture features, while the back stages extract more abstract features, which contain richer semantic information. Based on this division of the extracted features, the 123 stages are shallow (light blue feature maps) and the 45 stages are deep (dark blue features), thus obtaining a more discriminative basic representation of different changing objects in remote sensing images.

[0041] The two residual connections can be expressed as:

[0042]

[0043] in and They correspond to the two forms of residual blocks, j represents the number of times the current residual block is stacked, W1 and W2 represent the weight matrices of the two 3×3 convolution operations in R1(x) and R2(x), W s represents the weight matrix of a 1×1 convolution operation with a step size of s, σ(·) and BN(·) represent the nonlinear activation function (ReLU) and batch normalization layer respectively. So the specific extraction process of each stage can be expressed as follows:

[0044]

[0045] Where W represents the weight matrix of the 7×7 convolution operation, MaxPool represents the maximum pooling operation, and F1, F2, F3, F4, and F5 represent the feature extraction results of each stage respectively.

[0046] Selective Convolutional Attention (SCBAM) and Cascaded Cross Self Attention (CCSAM)

[0047] After using Backbone to extract basic information, it is often necessary to further enhance and improve the extracted features, increase the model's attention to important feature areas, and improve the global attention to dual-phase feature difference information.

[0048] Some existing methods have proposed and proved that the spatial and channel attention mechanisms can be used to enhance the main features, but they ignore the specific characteristics of multi-scale features extracted at different stages, which makes it more difficult to detect the changed area. Shallow features have high resolution, small receptive field, and contain more spatial position information, while deep features have low resolution, large receptive field, and contain more channel object semantic information. Therefore, the selective convolutional attention mechanism (SCBAM) is proposed based on the different characteristics of multi-scale features, such as Figure 4As shown in SCBAM, the features of different stages are further enhanced to improve the detection accuracy of more changing objects. Specifically, spatial attention (SAM) is used for the features of stage 123, and channel attention (CAM) is used for the features of stage 45.

[0049] Figure 4 , SCBAM module, the input is the different scale features of the dual-phase image {F t1 ,F t2 ,F t3 ,F t4 ,F t5 |t=1,2}, each time phase has 5 stages of feature maps, which are processed by CAM (Channel attention module) or SAM (Spatialattention module) and output as weighted feature maps.

[0050] Channel attention module: First, the 45-stage bi-temporal features are channel-refined through the channel attention map. The importance of each channel is encoded in the channel attention map, and its weight is automatically recalibrated in the network. i and the corresponding channel attention mapping weight W i c Multiply them together to get the optimized feature map F i c , highlighting the channels relevant to change detection while suppressing irrelevant channels. In this way, the channel attention module focuses on learning “which channel” is more important from the combined heterogeneous features. The channel attention weight module is as follows Figure 4 As shown in CAM.

[0051] Channel attention weight map W i c The calculation details are as follows:

[0052] W i c =σ(Fc2(Relu(Fc1(Maxpool(F i ))))+Fc2(Relu(Fc1(Avgpool(F i ))))) (3)

[0053] Among them, F i {i=4,5} represents the input feature map (see Figure 4). First, the spatial information of the input feature map is compressed along the spatial axis through the average pooling (AvgPool) operation and the maximum pooling (MaxPool) operation. Assuming there are C feature maps of size H×W, after the pooling operation, the feature map will be compressed into two vectors of size C×1×1. Then, each vector is forwarded to a shared multi-layer perceptron (MLP). The outputs of the shared MLP (two fully connected layers and a relu activation function) are merged using element-wise summation. Finally, a Sigmoid function is attached to assign the attention weight to each channel. σ represents the sigmoid function, and W i c {i=4,5} represents the channel attention weight map.

[0054] Spatial attention module: The 123-stage dual-phase features are further optimized in the spatial dimension through the spatial attention module. Similarly, the importance of each pixel position in the feature map is encoded into a spatial attention map. During the network training process, the spatial attention module is trained to acquire the ability to adaptively recalibrate weights pixel by pixel, and finally outputs a spatial attention map, in which the changed and unchanged pixel positions are assigned higher and lower importance, respectively. Specifically, the spatial attention map W is first calculated by the spatial attention module i s , then for each feature map F in the feature map set i , F i With W i s Element-wise multiplication is performed to achieve space optimization. The optimized feature map can be expressed as F i s After spatial optimization, the changed pixels are emphasized by multiplying them with higher weights, while the unchanged pixels are suppressed by multiplying them with lower weights. The structure of the spatial attention module is shown in Figure 4 As shown in SAM.

[0055] Spatial Attention Map W i s The calculation is as follows:

[0056] W i s =σ(Conv 7×7 (Concat(Avgpool(F i ),Maxpool(F i )))) (4)

[0057] Among them, σ is the sigmoid function, Conv 7×7represents a convolution operation, the convolution kernel size is 7×7, and Concat represents a cascade operation. First, the input feature map F with a dimension of H×W×C is i {i=1,2,3} are average pooled and max pooled in the channel dimension through AvgPool and MaxPool layers respectively. Then, the features of average pooling and max pooling are concatenated and convolved through a 7×7 convolution layer. Finally, after Sigmoid, the final output dimension is H×W spatial attention weight W i s {i=1,2,3}.

[0058] In general, the channel attention module focuses on the important channel information of the feature map, while the spatial attention module focuses on learning the important location area information of the feature map.

[0059] In W i c and W i s After calculating, input feature F i After SCBAM, it is refined as follows:

[0060] F i c =F i ·W i c (5)

[0061] F i s =F i ·W i s (6)

[0062] where F represents element-wise multiplication; i c {i=4,5} and F i s {i=1,2,3} denotes the refined feature maps at each stage through the channel and spatial modules, respectively.

[0063] Finally, the optimized dual-phase feature image F is obtained. i t (F i s 、F i c ), t=1,2.

[0064] CCSAM: Cascaded Cross Attention:

[0065] The purpose of remote sensing image change detection is to find the difference area of ​​the dual-phase image. The feature map of the cascaded dual-phase image contains the characteristic information of the dual-phase image to the maximum extent. Through simple convolution processing and back propagation of the contrast loss between the prediction map and the label map, the difference information of the dual-phase image can be learned. Therefore, MSFMF-CDNet cascades the dual-phase features processed by SCBAM to ensure that the difference features can be better learned.

[0066] After the original features are enhanced using the SCBAM module features, the new dual-phase feature image is obtained and recorded as F i 1 and F i 2 Then, the two are cascaded and the number of channels is unified using 1*1 convolution, which is recorded as F i cat , fed to CCSAM in order to improve the features and learn more difference information global attention, the specific module steps, first the cascaded features are processed by one-by-one point-by-point convolution, the role of point-by-point convolution is to reduce the dimension and obtain the required spatial attention weight map, and then F i cat The spatial dimensions of the corresponding spatial attention weight map are flattened and matrix multiplication is performed to obtain the matrix that can represent F i cat Concise semantic sequence token i , so that the sequence input required to satisfy the transformer cross self-attention is obtained, and finally the concise semantic sequence Token i As key and value and F i cat It is fed as a query to the cross self-attention module to improve the global features of the difference information, and the new features are recorded as T i (i=1,2,3,4,5).

[0067] The specific operation of CCSAM can be expressed as:

[0068] F i cat =Conv1×1(Concat(F i t1 ,F i t2 )) (7)

[0069]

[0070] Q,K,V=W q (F i cat ),Wk (Token i ),W v (Token i ) (9)

[0071]

[0072] F i 1 and F i 2 is the dual-phase feature image after feature enhancement, F i cat It is the feature representation after cascade convolution, Token i For concise semantic representation, QKV is the process identifier of the attention mechanism, and concat is the cascade operation. is a matrix multiplication representation, Wq, Wk, Wv are all linear transformation matrices, σ is the softmax normalization operation, Attention is the specific attention calculation process, T i is the final feature representation.

[0073] High-Level Feature-Guided Multi-Scale Feature Fusion Module (HFGFFM)

[0074] Currently, hierarchical multi-scale is widely used in change detection, but the shallow position-rich information is diluted as the model features extract deep high-level semantic information. A common method is to upsample the hierarchical multi-scale features in sequence to a unified scale and then perform cascade or addition operations. This fails to fully utilize high-level features to locate the changed objects, resulting in inaccurate prediction results. The present invention proposes a HFGFFM module to alleviate this problem.

[0075] So after passing through the SCBAM and CCSAM modules, we get T i {i=1,2,3,4,5} features, T i {i=1,2,3,4} Relative shallow features are respectively i {i=5} Relative deep features are added, so the new It includes the shallow position information of stages 1, 2, 3, and 4 respectively, as well as the deepest global semantic position information. The up-sampled images are fused in the corresponding element-wise addition to obtain F. After two layers of convolution and softmax operation, the predicted image F is obtained. pre , so that the feature map of each level will understand the location information of the changed object and generate the final change map.

[0076] The specific operation of HFGFFM can be expressed as:

[0077]

[0078] F pre =σ(Conv1×1(Conv3×3(F))) (13)

[0079] Among them, + represents the addition of elements at corresponding positions, Up represents the upsampling operation, Conv1*1 and Conv3*3 represent 1*1 convolution and 3*3 convolution operations respectively.

[0080] Loss Function

[0081] There are usually more pixels in the unchanged area than in the changed area. Therefore, during training, the pixels in the unchanged area may cause more losses, causing the model to form a potential unfair bias between changed and unchanged pixels. To alleviate this problem, a hybrid loss combining binary cross entropy loss and dice loss is used as the loss function, and the formula is as follows:

[0082] L=L BCE +L DICE (14)

[0083]

[0084] Where L BCE is the binary cross entropy loss, L DICE is the dice loss, y is the true value 0 or 1, corresponding to unchanged pixels and changed pixels, is the predicted value.

[0085] Dataset

[0086] To verify the effectiveness of the proposed model MSFMF-CDNet, a large number of experiments were conducted on the LEVIR-CD and CDD datasets which are often used for change detection in remote sensing images.

[0087] LEVIR dataset (Google Earth images): An open dataset called LEVIR was developed in the literature, mainly for building change detection. The LEVIR dataset contains 637 pairs of images collected from Texas, USA between 2002 and 2018 through the Google Earth API. Among them, 445 pairs of images are defined as training sets, while 192 pairs of images are defined as test sets. The size of each pair of images is 1024×1024 pixels, with a very high spatial resolution of 0.5 meters. The LEVIR dataset has the following characteristics:

[0088] a) The seasonal and illumination variations are obvious in bi-temporal images, which brings challenges to CD as it can easily cause irrelevant changes in actual detection.

[0089] b) The LEVIR dataset covers various types of buildings, such as villa houses, apartments, garages, and warehouses, which have different shapes, scales, and sizes.

[0090] c) Following the same dataset partitioning of STANet for training, validation, and testing. To reduce GPU memory usage, the images are divided into non-overlapping blocks of 256×256 pixels. This generates 7120 pairs, 1024 pairs, and 2048 pairs of samples for training, validation, and testing, respectively.

[0091] WHUCD dataset (aerial imagery): This dataset mainly includes building changes in Christchurch, New Zealand, where a 6.3 magnitude earthquake occurred in February 2011. The WHU-CD dataset consists of two aerial images collected at the same location in 2012 and 2016. The dataset covers 20.5 square kilometers, has a size of 32,507×15,354 pixels, and a spatial resolution of 0.075 meters. The CD task based on the WHU-CD dataset has the following challenges:

[0092] a) There are different types of building variations, including disappeared buildings and rebuilt buildings, which have various colors, shapes, scales, and ranges. It is a challenge to adapt to different forms of buildings and mine strong building features for the model.

[0093] b) In addition to buildings, the entire remote sensing scene is also covered by complex ground objects, such as grass, trees, roads, bridges, vehicles, and parking lots. Therefore, these ground objects can be easily confused with buildings; similarly, they may also cause many false detection pixels in label comparison.

[0094] c) The images in the WHU-CD dataset are cropped into 256×256 image pairs and divided into training, test, and validation sets in a ratio of 8:1:1.

[0095] CDD dataset (satellite images): The images included are mainly from Google Earth, covering different regions, including Brazil, the United States, Europe, the Middle East, and Asia. It consists of 11 pairs of satellite images taken in different seasons, with spatial resolutions ranging from 0.03 meters to 1 meter per pixel. The dataset includes changing areas of various sizes. The characteristics and processing methods of the CDD dataset are as follows:

[0096] a) The images include not only the changes of buildings, but also the changes of landmarks due to seasonal changes, such as vegetation, snow, etc.

[0097] b) This dataset labels focus more on changes related to the appearance and disappearance of objects rather than differences due to natural factors such as seasonal differences, brightness, etc.

[0098] c) In the experiments, following the data preprocessing steps outlined in CDD, 10,000, 2,998, and 3,000 pairs of 256 × 256 pixel image patches were generated for training, validation, and testing, respectively.

[0099] Experimental setup

[0100] 1) Comparison with State-of-the-Art Methods: The MSFMF-CDNet method is compared with several state-of-the-art remote sensing image change detection (CD) methods, including FC-EF, FC-Siam-Conc, FC-Siam-Di, STANet, BIT, DTCDSCN, SRCDNet, MSPSNet, and ChangeFormer.

[0101] 2) Evaluation Indicators: Four evaluation indicators are used to quantitatively evaluate the performance of the proposed method, including precision, recall, F1-score, mIoU, and overall accuracy (OA). The larger the values ​​of these four indicators, the better the change detection results. In particular, the F1-score takes into account both precision and recall, thus reflecting the overall quality of the method. The calculation of these four indicators is as follows:

[0102]

[0103] Among them, TP, TN, FP and FN respectively represent the number of true positive, true negative, false positive and false negative pixels in the confusion matrix obtained by comparing the predicted change map with the GT (Ground Truth, the actual change map).

[0104] 3) Implementation details: The model of the present invention is implemented using the PyTorch framework and trained on two NVIDIA RTX 3060 GPUs. The model is updated using a stochastic gradient descent (SGD) optimizer with momentum. In the experiment, the momentum was set to 0.99. During training, online data augmentation (i.e., image rotation and image flipping) was randomly applied to the input image pairs in the LEVIR-CD dataset. The learning rate was initially set to 0.001, and the trained model was validated using the validation set at the end of each training epoch. Finally, the model that performed best on the validation set was selected for evaluation on the test set.

[0105] Comparative experiment

[0106] First, the comparison of experimental results on CDD is shown. Table 1 shows the quantitative performance of different remote sensing image change detection algorithms. It can be observed that the MFIMF-CDNet of the present invention shows higher performance than existing competitors in all five evaluation indicators. The highest performance of FC-EF, FC-Siam-conc, and FC-Siam-diff based on CNN in the f1 index is only 87.95%, and its pure convolution structure is still difficult to capture the global pseudo-change information caused by unstable image conditions in CDD. At the same time, the DTCDSCNT based on CNN has reached the second highest level of 92.16% in F1 score. This shows that the use of se-resnet as an encoder and the spatial pyramid global pooling module introduces multi-scale global feature information, which is necessary for identifying false changes. The method of the present invention achieved 94.52% and 94.09% in F1 score and miou, respectively. All indicators have been significantly improved. The main reason for the significant improvement of the performance of the present invention is that the global and local information at multiple scales is fully utilized, which helps MSFMF-CDNet to achieve more accurate and complete change detection results in CDD, covering changes of different sizes.

[0107] Table 1 Experimental results of CDD dataset (143 / 150 epoch 3min / epoch)

[0108]

[0109] Afterwards Figure 5The representative visualization results of each CD method on CDD are shown in Figure 2. It can be seen that FC-EF, FC-SiamConc and FC-Siam-Di are affected by illumination changes and seasonal changes, and are not sensitive to some small targets, resulting in many missed changes in the detection results. Attention-based STANet alleviates the impact of different imaging conditions and reduces omissions to a certain extent. However, the spatial information of remote sensing targets shows a deteriorating trend in the attention module, resulting in the number of false alarms on the predicted boundaries. The DTCDSCD network uses SE-ResNet as the basic encoder module and utilizes the channel relationship of features at different scales. It first performs semantic segmentation tasks and then performs change detection. It relies heavily on the results of semantic segmentation, so it may be easy to make false detections at the edge of the target. SRCDNet reuses super-resolution technology while also using CBAM to enhance features at different scales in channels and space. The overall effect of the results in the comparison figure is good, but it is still not enough to capture long-distance dependencies. It is easy to make false detections in the state of extremely unbalanced samples. As can be seen from the last row of the figure, a small piece of false positives appears. MSPSNet is a deep multi-scale twin change detection network with a parallel convolutional structure and self-attention mechanism. It extracts multi-scale information while building long-distance dependencies. Due to the lack of global information of features at different scales, detection errors in complex scenarios are caused. By comparing the visualized changemap diagrams of each method, the method of the present invention has good effects on targets of complex sizes, shapes, and extremely unbalanced samples, which proves the effectiveness of the method.

[0110] Table 2 provides a quantitative comparison of different CD algorithms on the LEVIR-CD dataset. Unlike CDD, LEVIR-CD focuses on architectural changes, while the illumination differences and mismatch errors between dual-phase images in this dataset severely challenge CD methods. Compared with other methods, the MSFMF-CDNet of the present invention provides a higher F1 score of 90.40% and miou of 90.72%. Changeformer has the same f1 score as the present invention, but its miou is 17.52% lower. Changeformer is a transformer-based change detection model, which to some extent illustrates the importance of hierarchical multi-scale structure, but it has limited ability in local feature extraction. As a pioneer method that combines Transformer with CNN to achieve high-performance CD, BIT significantly outperforms the attention-based STANet in F1 score. Analogous to the BIT model, MSPSNet also utilizes CNN and self-attention mechanism, and pays attention to the multi-scale features ignored by BIT. Its F1 and miou are both higher than bit, which shows the importance of making full use of the relationship between global and local features at different scales.

[0111] Table 2 Experimental results on the LEVIR-CD dataset

[0112]

[0113] exist Figure 6 In , representative visualization results of existing methods and MSFMF-CDNet on LEVIR-CD are shown. Figure 6 As shown in Figure 2, the appearance of the same building and its shadow in the dual-temporal images varies greatly due to different lighting directions and intensities. This interference makes it difficult for CNN-based methods to distinguish between real changes and pseudo changes, resulting in many missed changes in the detection results. Attention-based methods STANet and DASNet are more complete in detecting building changes, but have defects in predicting subtle boundaries of change regions. BIT learns global context-rich features better than STANet and DASNet through transformer modules, but its performance on small building changes (missed detection) is not satisfactory due to the simple single-scale integration of deep CNN and transformer modules. Overall, MSFMF-CDNet more robustly identifies change regions of different sizes by leveraging rich multi-scale global-local representations.

[0114] Table 3 Experimental results of WHU-CD dataset (129 / 150 epoch 1.5 min / epoch)

[0115]

[0116] The performance of different CD algorithms on the WHU-CD dataset is compared in Table 3. In addition to buildings, the entire remote sensing scene in the WHU dataset also contains complex objects such as grass, trees, roads, bridges, vehicles, and parking lots. These objects are easily confused with buildings, which poses a challenge to the model to accurately identify building changes in labels. By comparing with other methods, it can be found that ICIF-Net and MSPSNet have advantages in F1 scores, reaching 88.32% and 88.29% respectively, which proves the advantage of combining CNN with Transformer to extract local and global features. Based on this advantage, MSFIMF-CDNet uses the advantages of hierarchical structure to achieve an F1 score of 85.37%, and at the same time achieves the highest accuracy in Recall and mIoU indicators, which are 87.62% and 86.58% respectively. It shows the importance of building supplementary global information based on local features and proves the structural effectiveness of MSFIMF-CDNet.

[0117]

[0118]

[0119] For the comparison of model efficiency, Table 3 shows the number of model parameters (Params.), floating point operations per second (FLOPs), and the time required to run an epoch on the WHU-CD dataset, using an image size of 256×256×3. Due to the combination of CNN and Transformer at multiple scales, MSFIMF-CDNet performs better in semantic representation, but also requires more resources. The number of parameters is higher than that of models such as BIT and SRCDNET, but lower than that of DTCDSCN, ICIF-Net, and ChangeFmer. In terms of FLOPs, it has an advantage over ICIF-Net with a parallel fusion strategy, and has significant improvements in Recall and mIoU indicators. It is worth noting that the use of residual networks and self-attention mechanisms improves certain performance while increasing the number of parameters, so it is necessary to study some lightweight architectures that take into account efficiency while improving model expression capabilities.

[0120] Ablation experiment

[0121] It can be seen from Table 4 that MSFMF-CDNet has achieved the best evaluation indicators of F1 and mIou on LEVIR-CD. Compared with the Baseline, MSFMF-CDNet has increased by 0.45% and 0.38% respectively. From the perspective of F1 and mIou, after adding SCBAM and HFGFFM separately, the F1 and mIou indicators have decreased compared with the Baseline, and the indicator has improved after adding CCSAM alone, which shows the effectiveness of CCSAM for global feature modeling. When CCSAM and SCBAM are ablated at the same time, the indicator is improved by 0.13 compared with the F1 indicator when CCSAM is ablated alone, which indirectly shows the effectiveness of SCBAM in selectively modeling multi-scale feature space channels. On this basis, after adding HFGFFM, the F1 and mIou indicators are increased by 0.02 and 0.24 respectively, achieving the optimal effect in the ablation experiment, which to a certain extent illustrates the importance of HFGFFM for introducing high-level feature global information into different scale features. Through the above ablation experiments, it can be seen that the combined use of various modules in MSFMF-Net has a significant impact on the improvement of network performance, which proves the effectiveness of each module.

[0122] Table 4 Ablation results of each module

[0123]

[0124]

[0125] At the same time, Figure 7The feature extraction of the model and the process visualization effect diagram of each module are given in Figure 1. Red represents high attention and blue represents low attention. It can be seen that the feature map (the second and fourth rows on the left) obtained after the original features (the first and fifth rows on the left) are SCBAM. The dual-phase original features undergo the convolutional spatial attention mechanism in stages 1, 2, and 3. The changes in the feature map are not obvious. The feature maps in stages 4 and 5 are relatively abstract. From the feature map of stage 4 of image2, it can be seen that the feature map after channel attention is more concentrated than the original feature map, and there are fewer adhesions of some attention points; the second and fourth rows on the left are cascaded to retain the difference information (the third row on the left). It can be seen from stages 1, 2, and 3 that the feature map (the first row on the right) obtained after CCSAM is obvious, and the difference features are well learned, highlighting the difference area; the second row on the right is the feature map after HFGFFM, and the effect is not obvious; by comparing the binary prediction map with the change map in the third row on the right, it can be seen that the edges of the change area are clear and not adhered, and there are only a few false detections.

[0126] This paper proposes a new MSFMF-CDNet method to enhance the performance of remote sensing image change detection in high-resolution remote sensing images. First, the concept of selectively using spatial and channel attention mechanism (SCBAM) based on the characteristics of feature maps at different scales is proposed to enhance the original features. Then, the cascaded cross self-attention mechanism (CCSAM) is used to introduce global feature information. Finally, a high-level feature guided feature fusion module (HFGFFM) is proposed to compensate for the shallow positional spatial information lost after the abstraction of high-level semantic features during feature extraction. Different from the existing remote sensing image change detection algorithms, this paper extends the combination of local features collected from a single scale by CNN and the global context supplemented by transformer to multiple scales. The saliency of the change region is enhanced, and the robustness of the CD performance for change regions of different sizes, especially small change regions, is significantly improved. Comparative experiments on three public datasets show that the present invention achieves higher F1 scores and mIOU values ​​in CDD and LEVIR-CD compared with the most recent CD methods.

Claims

1. A remote sensing image change detection network for multi-scale feature extraction and information mining, characterized by: It includes a multi-scale CNN-Backbone encoder, a selective convolutional attention mechanism module, a cascaded cross self-attention mechanism module and an advanced feature-guided multi-scale fusion module. The multi-scale CNN-Backbone encoder extracts different-scale features of the dual-phase image, and the different-scale features of the dual-phase image are input into the selective convolutional attention mechanism module. The features of different stages are further enhanced. The selective convolutional attention mechanism module outputs the optimized and enhanced dual-phase feature image, which is input into the cascaded cross self-attention mechanism module. The cascaded cross self-attention mechanism module is used to refine the global information of the multi-scale features and obtain new features. Finally, the advanced features are used to guide the multi-scale fusion module to decode the new features, and a classifier is applied to distinguish the changing features to generate a change map with sharp details.

2. The remote sensing image change detection network for multi-scale feature extraction and information mining according to claim 1, characterized in that: The multi-scale CNN-Backbone encoder adopts a twin framework, which consists of two parallel sub-network residual networks with the same structure and shared parameters. The dual-phase images I1 and I2 are fed into the two parallel sub-network residual networks as inputs for processing. The feature extraction process of the dual-phase images I1 and I2 includes five stages. The first stage consists of a convolutional layer BN and Relu, the second stage consists of a maximum pooling layer and Layer1, and the remaining three stages consist of Layer2, Layer3, and Layer4. Finally, the different scale features of the dual-phase images are extracted in the five stages. t1 ,F t2 ,F t3 ,F t4 ,F t5 |t=1,2}.

3. The remote sensing image change detection network for multi-scale feature extraction and information mining according to claim 2 is characterized in that: The selective convolutional attention mechanism module focuses on the features of the 1st, 2nd, and 3rd stages of the input dual-phase image at different scales. i {i=1,2,3} uses spatial attention optimization enhancement to enhance the features F in stages 4 and 5. i {i=4,5} uses channel attention optimization enhancement; The process of channel attention optimization and enhancement is as follows: i {i=4,5} and the corresponding channel attention weight map W i c Multiply them together to obtain the optimized and enhanced feature map F i c , channel attention weight map W i c The calculation details are as follows: W i c =σ(Fc2(Relu(Fc1(Maxpool(F i ))))+Fc2(Relu(Fc1(Avgpool(F i )))))), where AvgPool represents the average pooling operation, MaxPool represents the maximum pooling operation, Fc1 and Fc2 represent the fully connected layers, relu represents the activation function, σ represents the sigmoid function, and W i c {i=4,5} represents the channel attention weight map; The process of spatial attention optimization and enhancement is as follows: First, the spatial attention weight map W is calculated by the spatial attention module. i s , then the feature map feature F i {i=1,2,3}F i With W i s Element-by-element multiplication is performed to achieve spatial optimization enhancement. The optimized and enhanced feature map can be expressed as F i s , spatial attention weight map W i s The calculation is as follows: W i s =σ(Conv 7×7 (Concat(Avgpool(F i ),Maxpool(F i ))))), where Conv 7×7 represents a convolution operation, the convolution kernel size is 7×7, Concat represents a cascade operation, W i s {i=1,2,3} represents the spatial attention weight map; In W i c and W i s After calculating, input feature F i The selective convolution attention mechanism module is optimized and enhanced as follows: i c =F i ·W i c , F i s =F i ·W i s , where · represents element-by-element multiplication; F i c {i=4,5} and F i s {i=1,2,3} represents the optimized enhanced feature maps at each stage through the channel and spatial modules, respectively; Finally, the optimized dual-phase feature image F is obtained. i t , F i t Including F i s and F i c , t=1,2.

4. The remote sensing image change detection network for multi-scale feature extraction and information mining according to claim 3 is characterized by: Dual-phase feature image F i 1 and F i 2 Input to the cascade cross self-attention mechanism module, the cascade cross self-attention mechanism module refines the multi-scale feature global information, and obtains a new feature denoted as T i {i=1,2,3,4,5}; The specific operation is expressed as: F i cat =Conv1×1(Concat(F i t1 ,F i t2 )), Q,K,V=W q (F i cat ),W k (Token i ),W v (Token i ), Where: F i cat It is the feature representation after cascade convolution, Token i For concise semantic representation, Q, K, and V are the process indicators of the attention mechanism, and concat is a cascade operation. is the matrix multiplication representation, W q ,W k ,W v They are all linear transformation matrices, σ is the softmax normalization operation, and Attention is the specific attention calculation process.

5. The remote sensing image change detection network for multi-scale feature extraction and information mining according to claim 4 is characterized in that: Feature T i {i=1,2,3,4,5} generates the final change map F in the high-level feature-guided multi-scale fusion module pre The specific process is: F pre =σ(Conv1×1(Conv3×3(F))), Among them, + represents the addition of elements at corresponding positions, Up represents the upsampling operation, Conv1*1 and Conv3*3 represent 1*1 convolution and 3*3 convolution operations respectively.

Images