Remote sensing image change detection system and method based on multimodal deep learning

Abstract

The present invention relates to a remote sensing image change detection system and method based on multimodal deep learning. The system improves detection accuracy by constructing a twin network that integrates CNN, Transformer and attention mechanism; adopts multi-scale depthwise separable convolution to construct a lightweight backbone network, and combines the soft thresholding channel space attention mechanism to adaptively suppress noise and enhance feature expression; innovatively introduces Harr wavelet transform downsampling to retain high-frequency details, replacing traditional pooling operations to reduce information loss; alleviates twin branch semantic bias through cross-channel feature exchange, and uses Transformer codec to model long-range dependencies; feature splicing and progressive upsampling strategies are used in the decoding stage to enhance edge feature recovery; while reducing model parameters, the system significantly improves the accuracy and anti-interference ability of remote sensing building change detection, and has the advantages of high efficiency and detail preservation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning technology, and in particular relates to a remote sensing image change detection system and method based on multimodal deep learning. Background Art

[0002] Remote sensing imagery, as an essential means of acquiring surface information, is widely used in agriculture, forestry, urban planning, disaster monitoring, and environmental protection. With the rapid development of remote sensing platforms such as satellites and drones, remote sensing image data is characterized by high resolution, large data volumes, and complex scenes. Change detection, a core task in remote sensing image processing, aims to automatically identify areas of change in images at different temporal phases, thereby providing a basis for decision support.

[0003] Currently, change detection in remote sensing images primarily relies on convolutional neural networks (CNNs) for feature extraction. Traditional CNN architectures offer advantages in extracting local features. However, due to the presence of significant redundant information and noise in remote sensing images, traditional downsampling methods based on fixed-scale convolutions and simple pooling operations (such as max pooling or average pooling) often struggle to balance global and detailed information, leading to the partial loss of important features. Furthermore, regions of change in remote sensing images are typically small, making direct training with the binary cross entropy loss function (BCELoss) prone to class imbalance, resulting in poor performance in detecting subtle changes.

[0004] In recent years, to address the shortcomings of CNNs in modeling long-range dependencies, researchers have begun introducing Transformer modules and attention mechanisms to improve global feature modeling and feature selection. However, while directly embedding Transformers into networks can capture long-range dependencies, it also introduces a large number of parameters and is prone to problems such as feature redundancy and noise interference.

[0005] Attention mechanisms are widely used in computer vision. Their basic idea is to assign different weights to different regions or channels, highlighting key features and thereby increasing the network's focus on the target area. Traditional channel attention, spatial attention, and hybrid attention mechanisms (such as CBAM) have improved feature extraction to some extent. However, these methods still have limitations when dealing with noisy and redundant information.

[0006] In summary, due to the existence of multi-scale, subtle changes, and noise interference in actual scenes, existing remote sensing image change detection methods still have certain limitations in detection accuracy and robustness; in addition, how to achieve multi-scale feature extraction and information fusion while ensuring a low number of parameters is also a technical problem that needs to be solved urgently. Summary of the Invention

[0007] In view of the shortcomings and deficiencies of the existing technology, the present invention provides a remote sensing image change detection system based on multimodal deep learning, which improves detection accuracy by constructing a twin network that integrates CNN, Transformer and attention mechanism; adopts multi-scale depth-separable convolution to construct a lightweight backbone network, combined with soft thresholding channel space attention mechanism to adaptively suppress noise and enhance feature expression; innovatively introduces Harr wavelet transform downsampling to retain high-frequency details, replacing traditional pooling operations to reduce information loss; alleviates twin branch semantic bias through cross-channel feature exchange, and uses Transformer codec to model long-range dependencies; feature splicing and progressive upsampling strategies are used in the decoding stage to enhance edge feature recovery; while reducing model parameters, the system significantly improves the accuracy and anti-interference ability of remote sensing building change detection, and has the advantages of high efficiency and detail preservation.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A remote sensing image change detection system based on multimodal deep learning, the system uses a trained multimodal feature extraction network to detect remote sensing image changes; the multimodal feature extraction network is composed of two weight-sharing sub-networks, each of which processes a dual-temporal remote sensing image to extract its own feature information. The sub-networks include an initial convolutional layer, a Siamese encoding module, a Transformer global modeling module, and a cascade decoding module, which are arranged in sequence;

[0010] The Siamese encoding module includes four feature extraction stages, each of which is equipped with a Haar wavelet transform downsampling module, a multi-scale depth-separable convolution module, a convolution module and a soft thresholding attention module in sequence; a channel feature exchange mechanism is introduced in the third feature extraction stage of the two sub-network Siamese encoding modules to exchange the bi-phase features output by the third feature extraction stage in the channel dimension to achieve complementarity and interaction of the bi-phase features; the bi-phase features output by the fourth feature extraction stage are input into the Transformer global modeling module to achieve global dependency modeling; the soft thresholding attention module comprehensively utilizes the channel attention and spatial attention mechanisms;

[0011] The cascade decoding module includes four upsampling stages, and the features output by each feature extraction stage of the Siamese encoding module and the features output by the Transformer global modeling module are input into the cascade decoding module through a UNet-like jump connection.

[0012] As a preferred embodiment of the present invention, the pre-processed dual-temporal remote sensing image is passed through a 3×3 initial convolutional layer to expand the number of input channels from the original value to 32 channels.

[0013] As a preferred embodiment of the present invention, the dual-phase features outputted in the third feature extraction stage are and Perform parameter-free channel feature exchange operation and and The image is randomly divided into N sub-blocks along the channel dimension, and the corresponding sub-blocks are exchanged and reassembled to keep the total number of channels unchanged.

[0014] As a preferred embodiment of the present invention, the multi-scale depth-separable convolution module includes a 3×3 depth-separable convolution module and parallel 5×5 depth-separable convolution modules and 7×7 depth-separable convolution modules; the data processing flow is:

[0015] Step a. First, input the 3×3 depth-wise separable convolution module for convolution processing, followed by batch normalization and ReLU activation function;

[0016] Step b. The data output from step a is split into two paths. One path is convolved with a 5×5 depthwise separable convolution module, followed by batch normalization and ReLU activation. The other path is convolved with a 7×7 depthwise separable convolution module, followed by batch normalization and ReLU activation.

[0017] Step c. The feature maps output by the three sets of depth-wise separable convolutions are added or concatenated pixel by pixel to form a fused multi-scale feature map; the fused multi-scale feature map is passed through a 3×3 convolution module for inter-channel information interaction and outputs a feature map.

[0018] As a preferred embodiment of the present invention, the data processing flow of the soft thresholding attention module is as follows:

[0019] Step a. Channel attention calculation:

[0020] Perform global average pooling and global maximum pooling on the input feature F to obtain two channel description vectors v avg , v max ; Use multi-layer perceptron to process the pooling results and generate channel attention weights through Sigmoid activation function ; Multiply the original features by the channel attention weight element by element to obtain the channel weighted features ;

[0021] Step b. Spatial attention calculation:

[0022] Features after channel weighting Average pooling and maximum pooling are performed in the spatial dimension to obtain two two-dimensional feature maps, which are then channel-joined. The joined features are processed using a 7×7 convolutional layer and a Sigmoid activation function to generate spatial attention weights. The spatial attention weights are element-wise multiplied with the channel-weighted features to obtain spatial weighted features. ;

[0023] Step c. Soft thresholding:

[0024] First, perform global average pooling on the input feature F, and use the multi-layer perceptron to adaptively learn the threshold ;

[0025] ;

[0026] Where W1 and W2 represent the weights of the two fully connected layers; Relu represents the Relu activation function, σ represents the Sigmoid activation function; GAP represents global average pooling;

[0027] Then the spatial weighted features Each element in is soft-thresholded using the following formula:

[0028] ;

[0029] Where sign represents the sign function; Represents the output features.

[0030] As a preferred embodiment of the present invention, the Transformer global modeling module includes a Transformer encoder and a Transformer decoder. The Transformer encoder adopts a multi-head self-attention mechanism and combines position encoding to process input features to capture long-distance dependencies in the image; the output of the Transformer encoder serves as the input of the Transformer decoder. While decoding the output features of the Transformer encoder, the Transformer decoder cross-fuses the features directly output by the fourth feature extraction stage, and finally generates a high-resolution feature map through a fully connected layer and convolution operation.

[0031] As a preferred embodiment of the present invention, each upsampling stage of the cascade decoding module is implemented by a decoder, and each upsampling stage performs the following operations:

[0032] Step a. Perform bilinear interpolation and 2-fold upsampling on the input features;

[0033] Step b. Concatenate the upsampled features with the corresponding feature extraction stage features along the channel dimension;

[0034] Step c. Compress the number of channels of the concatenated feature map to 1 / 2 of the original decoder channel number through 1×1 convolution;

[0035] Step d. Use 3×3 convolution to extract difference features and generate optimized feature maps through ReLU activation function.

[0036] As a preferred embodiment of the present invention, the multimodal feature extraction network uses a hybrid loss function to optimize the output during training, and the hybrid loss function is composed of a binary cross entropy loss and a Dice loss weighted at a 1:1 weight.

[0037] As a further preferred embodiment of the present invention, the Transformer encoder has 4 layers, each layer includes an 8-head self-attention mechanism; the Transformer decoder has 4 layers, each layer includes an encoder-decoder cross attention mechanism.

[0038] As a further preference of the present invention, when upsampling is performed, the specific operation of step b is: splicing the upsampled features of the first stage with the output features of the Transformer decoder along the channel dimension; splicing the upsampled features of the second stage with the output features of the third feature extraction stage of the Siamese encoding module along the channel dimension; splicing the upsampled features of the third stage with the output features of the second feature extraction stage of the Siamese encoding module along the channel dimension; splicing the upsampled features of the fourth stage with the output features of the first feature extraction stage of the Siamese encoding module along the channel dimension.

[0039] The present invention also provides a remote sensing image change detection method based on multimodal deep learning, which includes the following steps:

[0040] Step 1. After remote sensing image data is collected, normalization, enhancement, and alignment are completed through the data preprocessing module;

[0041] Step 2. The pre-processed dual-temporal image is subjected to feature extraction and change detection by the multimodal deep learning-based remote sensing image change detection system, and a change probability map is output;

[0042] Step 3. The post-processing module binarizes the change probability map to generate the final binary change area detection result.

[0043] The present invention has the following significant advantages over the prior art:

[0044] (1) The present invention adopts an Inception-style multi-scale convolution module for multi-scale feature extraction, which can simultaneously capture local details and global context information under convolution kernels of different sizes, solving the problem that traditional fixed convolution kernel size is difficult to take into account multi-scale information; in addition, the multi-scale feature extraction module can not only reduce the number of network parameters, but also make full use of the advantages of parallel convolution kernels to generate more representative feature expressions.

[0045] (2) The present invention introduces a Haar wavelet transform downsampling module in the downsampling stage to decompose the low-frequency and high-frequency information of the image. While reducing the image resolution, it retains the low-frequency and high-frequency key information in the image to the maximum extent, effectively solving the problem of information loss in traditional pooling operations.

[0046] (3) The present invention introduces the Transformer global modeling module in high-level feature processing, uses the self-attention mechanism to capture long-distance dependencies in the image, and realizes the modeling of global long-distance dependencies, making the fusion of bi-temporal features more complete, bridging the semantic gap between different levels, and solving the shortcomings of traditional CNN in capturing long-distance features.

[0047] (4) The present invention designs a soft thresholding attention module, which introduces a soft thresholding operation based on the traditional channel and spatial attention. By dynamically learning the threshold, it effectively suppresses noise information, strengthens the expression of key features, and improves the robustness of the model in complex backgrounds.

[0048] (5) Aiming at the problem of small change areas and class imbalance in remote sensing images, the present invention designs a hybrid loss function combining BCELoss and DiceLoss during the model training stage. The hybrid loss function takes into account both pixel-level and region-level evaluation criteria. It can still achieve high detection accuracy when the change area is small, and improve the class imbalance problem.

[0049] (6) The detection system provided by the present invention has reasonable control over the number of parameters. Although the network structure is relatively complex, the overall number of parameters is still controlled at a low level by using depthwise separable convolution and multi-scale parallel structure, which is convenient for deployment and real-time processing on embedded platforms.

[0050] (7) The remote sensing image change detection system (MSATNet) proposed in this invention, which is based on multi-scale feature extraction, Haar wavelet transform downsampling, Transformer module and soft thresholding attention mechanism, has been verified by a large number of experiments. It has been determined that it has obvious advantages over the existing technologies in detection accuracy, robustness and parameter control. With only 7.90M parameters, it can effectively capture the multi-scale features, detailed information and global dependencies in the image; it is suitable for various remote sensing image change detection application scenarios, such as urban monitoring, environmental change assessment and disaster warning, especially in environments with large amounts of remote sensing image data and subtle and complex change areas; its superior performance and flexible structural design provide new ideas and technical means for subsequent research in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to facilitate the understanding of the technical solution of the present invention, the following is a brief description of the accompanying drawings. Each of the accompanying drawings is a schematic diagram, and its specific structure and process are only used to illustrate the preferred implementation of the present invention and should not be regarded as limiting the scope of protection of the present invention.

[0052] Figure 1 Schematic diagram of the overall architecture of MSATNet provided by the present invention;

[0053] Figure 2 Figure 1 is a flowchart of the first stage of the twin encoding module. (a) is the data processing flow of the first encoding stage (Stage 1); (b) is the data processing flow of the multi-scale feature extraction (DMSConv); (c) is the data processing flow of the soft thresholding attention module (STAM).

[0054] Figure 3 Comparison of change detection results of dual-temporal remote sensing images under different networks. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings and specific steps so that technicians can fully understand the technical solutions of the present invention. This embodiment is only a preferred implementation method and is not intended to limit the scope of protection of the present invention. Technicians can make various modifications and improvements without departing from the spirit of the present invention.

[0056] like Figure 1 、 Figure 2As shown, this embodiment provides a remote sensing image change detection system based on multimodal deep learning. The system adopts a trained multimodal feature extraction network (MSATNet). The overall architecture of the multimodal feature extraction network adopts a Siamese network structure design and is composed of two weight-sharing sub-networks. The two sub-networks process the dual-temporal remote sensing images respectively and extract their respective feature information. The sub-networks include an initial convolutional layer (Stem), a Siamese encoding module (backbone network), a Transformer global modeling module, and a cascade decoding module.

[0057] The Siamese encoding module adopts a weight-sharing depthwise separable convolution structure. The Siamese encoding module includes four feature extraction stages (encoding stages). Each feature extraction stage is equipped with a Haar wavelet transform downsampling module (HWD), a multi-scale depthwise separable convolution module (DMSConv), a 3×3 convolution module (3×3Conv) and a soft thresholding attention module (STAM).

[0058] The multi-scale depthwise separable convolution module includes a 3×3 depthwise separable convolution module and parallel 5×5 depthwise separable convolution modules and 7×7 depthwise separable convolution modules, thereby achieving multi-scale feature extraction and fusing information at each scale to capture local details and global context information at different scales;

[0059] A channel feature exchange (ChannelExchange) mechanism is introduced in the third feature extraction stage of the Siamese encoding modules of the two sub-networks. It is used to exchange the bi-temporal features output by the third feature extraction stage in the channel dimension, realizing the complementarity and interaction of the bi-temporal features. The bi-temporal features output by the fourth feature extraction stage are input into the Transformer global modeling module to implement global dependency modeling, and up-sampled feature fusion is achieved using skip connections and 1×1 convolution.

[0060] The soft thresholding attention module comprehensively utilizes channel attention and spatial attention mechanisms;

[0061] The cascade decoding module includes four upsampling stages, and the features output by each feature extraction stage of the Siamese encoding module and the features output by the Transformer global modeling module are input into the cascade decoding module through a UNet-like jump connection.

[0062] In this embodiment, the data processing flow of the multimodal feature extraction network (MSATNet) is as follows:

[0063] Step a. Input data:

[0064] The dual-temporal remote sensing images are denoted as I1 and I2 respectively.

[0065] The preprocessed dual-temporal remote sensing images (each image) first pass through a 3 × 3 initial convolutional layer to expand the number of input channels from the original value (e.g., 3 or multispectral channels) to 32 channels for subsequent network processing.

[0066] After the initial convolutional layer, batch normalization and ReLU activation function are used to ensure the stability of features during transmission.

[0067] Step b. Dual-phase feature extraction and channel interaction:

[0068] The Siamese encoding module (backbone network) is divided into four feature extraction stages: Stage 1, Stage 2, Stage 3, and Stage 4. Each stage uses the Depthwise Multi-Scale Convolution Module for feature extraction. A downsampling module is introduced between stages. The downsampling module uses the Haar wavelet transform downsampling module (HWD) to replace the traditional pooling operation, which not only reduces the resolution but also retains key details. That is, each stage is connected in sequence to the Haar wavelet transform downsampling module (HWD), the multi-scale depth-separable convolution module (DMSConv), the 3×3 convolution module (3×3Conv), and the soft thresholding attention module (STAM).

[0069] The bi-phase features output from the first feature extraction stage are recorded as and ; The bi-phase features outputted in the second feature extraction stage are recorded as and ; The bi-phase features outputted in the third feature extraction stage are recorded as and , ∈R (H / 8×W / 8×C) and ∈R (H / 8×W / 8×C) ; The dual-phase features output in the fourth feature extraction stage are recorded as and ;

[0070] In order to narrow the semantic difference between the features of the two phases in the third stage, the channel feature exchange mechanism is introduced. The specific approach is: in the channel dimension, and Perform exchange processing and achieve feature information complementation through weighted addition, splicing, etc. Specifically, perform parameter-free channel feature exchange operation, and and The image is randomly divided into N sub-blocks along the channel dimension, and the corresponding sub-blocks are exchanged and reassembled to keep the total number of channels unchanged.

[0071] In this embodiment, and Randomly divided into 4 sub-blocks along the channel dimension, we get { , , , }and{ , , , }; Then randomly select two indexes i,j∈[1,4] and exchange and ; Reassemble the sub-blocks to generate the swapped features and , keeping the total number of channels unchanged.

[0072] Step c. Global dependency modeling:

[0073] The dual-phase features output by the fourth stage and Input the Transformer global modeling module and process it.

[0074] In this embodiment, the Transformer global modeling module includes a Transformer encoder and a Transformer decoder. The Transformer encoder (Enconder) adopts a multi-head self-attention mechanism and combines position encoding to process the input features (weighted summation) to capture the long-range (also known as long-distance) dependencies in the image; specifically, the Transformer encoder contains a 4-layer multi-head self-attention mechanism, each layer consists of a query-key-value projection matrix, a scaled dot product attention calculation and a feedforward neural network; the feature map that has undergone preliminary convolution processing is first marked by the position encoding module before entering the Transformer encoder to ensure that spatial information is not lost, thereby achieving deep fusion of dual-temporal features.

[0075] The Transformer decoder takes the output of the Transformer encoder as input. While decoding the output features of the Transformer encoder, it cross-fuses them with low-level features to achieve a unified expression of global information and local details. Specifically, the Transformer decoder uses a cross-attention mechanism to perform weighted fusion of the Transformer encoder output and low-level features (features directly output in the fourth feature extraction stage), and finally generates a high-resolution feature map through fully connected layers and convolution operations for use in the upsampling stage.

[0076] Step d. Multi-scale feature fusion and decoding:

[0077] After processing by the Transformer global modeling module, upsampling is achieved using skip connections and 1×1 convolutions to restore low-resolution features to high resolution while enhancing edge details and change area information.

[0078] Specifically, the features output by the Transformer global modeling module and the features output by each stage of the Siamese encoding module are input into the cascade decoding module through a UNet-like jump connection. The cascade decoding module includes four upsampling stages (decoding stages). Each upsampling stage is implemented by a decoder, that is, there are four decoders in total. The four decoders are set in sequence, namely Decoder1, Decoder2, Decoder3, and Decoder4. Each upsampling stage performs the following operations:

[0079] Step a. Perform bilinear interpolation and 2x upsampling on the input features;

[0080] Step b. Concatenate the upsampled features with the corresponding encoding stage features along the channel dimension; specifically, in the kth decoding stage (k=1,2,3,4), the upsampled feature map and encoding stage (feature extraction stage) features Splicing, get ; That is: the upsampled features of Decoderl are spliced with the output features of the Transformer decoder along the channel dimension; the upsampled features of Decoder2 are spliced with the output features of the third stage of the Siamese encoding module along the channel dimension; the upsampled features of Decoder3 are spliced with the output features of the second stage of the Siamese encoding module along the channel dimension; the upsampled features of Decoder4 are spliced with the output features of the first stage of the Siamese encoding module along the channel dimension;

[0081] Step c. Use 1×1 convolution to concatenate the feature maps The number of channels is compressed to 1 / 2 of the original decoder channel number;

[0082] Step d. Use 3×3 convolution to extract difference features and generate optimized feature maps through ReLU activation function; the expression of the difference feature extraction stage is:

[0083] ;

[0084] in, For the difference feature, Represents the absolute value.

[0085] In this embodiment, after multi-layer feature extraction, fusion, and upsampling, the network outputs a change detection map C, where the value of each pixel represents the change probability of the region in the dual-phase image.

[0086] Traditional downsampling methods (such as maximum pooling or average pooling) often ignore high-frequency details in the image during downsampling. This embodiment uses Haar wavelet transform to achieve downsampling. The Haar wavelet transform downsampling module (HWD) is designed to combine lossless information encoding and learning to ensure complete information transmission during the downsampling process. The specific data processing steps are as follows:

[0087] Step a. Input feature map X∈R (H×W×C) Perform one-dimensional Haar transform in the row direction to calculate the low-frequency component a x and high-frequency components d x ; That is: for each row of the input image, treat it as a one-dimensional signal , use the following formula to process adjacent pixels:

[0088]

[0089] in, Represents the low-frequency coefficient (also called low-frequency component), Represents high-frequency coefficients (also called high-frequency components).

[0090] Step b. After processing each row, perform one-dimensional Haar transform again along the column direction on the result of row transform, that is, repeat the above steps for each column to achieve two-dimensional Haar wavelet transform. After Haar wavelet transform in row and column directions, the image is decomposed into four sub-bands:

[0091] Low frequency subband LL∈R (H / 2×W / 2×C) ; Where H and W are the height and width of the feature map respectively, and C is the number of channels;

[0092] Horizontal high frequency subband LH∈R (H / 2×W / 2×C) ;

[0093] Vertical high frequency subband HL∈R (H / 2×W / 2×C) ;

[0094] Diagonal high frequency subband HH∈R (H / 2×W / 2×C) ;

[0095] Step c. Concatenate the four subbands along the channel dimension to form X'∈R (H / 2×W / 2×4C) ;

[0096] Step d. Input the spliced data X' into the feature learning block for convolution processing to further extract effective features, achieve the effect of halving the image size and doubling the number of channels, while retaining key information as much as possible; wherein, the feature learning block is composed of a 3×3 convolution layer (output channel 2C), a batch normalization layer and a ReLU activation function connected in sequence, and finally outputs the downsampled feature map X''∈R (H / 2×W / 2×2C) .

[0097] like Figure 2 As shown in (b), in this embodiment, the multi-scale depth-separable convolution module (Depthwise Multi-Scale Convolution Module) adopts a design similar to the Inception structure, and the specific data processing flow is as follows:

[0098] Step a. First, input the 3×3 depthwise separable (DWConv3×3) convolution module for convolution processing, followed by batch normalization (BN) and ReLU activation function;

[0099] Step b. The data output from step a is split into two paths. One path is convolved with a 5×5 depthwise separable (DWConv5×5) convolutional module, followed by batch normalization (BN) and ReLU activation. The other path is convolved with a 7×7 depthwise separable (DWConv7×7) convolutional module, followed by batch normalization (BN) and ReLU activation.

[0100] Step c. Add or concatenate the feature maps output by the three sets of depth-wise separable convolutions pixel by pixel to form a fused multi-scale feature map.

[0101] In this embodiment, the fused multi-scale feature map passes through a 3×3 standard convolution layer (convolution module) to perform inter-channel information interaction and output a feature map.

[0102] In this embodiment, three groups of convolution branches are used, and each group of convolutions is combined with a depth-wise separable convolution strategy to significantly reduce the number of parameters. In addition, each group of convolution branches is equipped with corresponding batch normalization and activation functions to ensure stable gradient transmission during the convolution process.

[0103] Furthermore, in this embodiment, the specific configuration of the Transformer global modeling module is as follows:

[0104] Transformer encoder layer number: 4 layers, each layer includes an 8-head self-attention mechanism, and the hidden layer dimension is 512; Transformer decoder layer number: 4 layers, each layer includes an encoder-decoder cross attention mechanism;

[0105] The positional encoding takes the form of a sinusoidal function with learnable parameters:

[0106]

[0107] Among them, w i is a trainable parameter, pos is the spatial position index, and i is the dimension index; Represents the position encoding output result.

[0108] Attention calculation uses the scaled dot product formula:

[0109]

[0110] Among them, d k is the key vector dimension, Q is the query vector, K is the key vector, and V is the value vector. is the activation function.

[0111] Furthermore, in this embodiment, the soft thresholding attention module comprehensively utilizes the channel attention and spatial attention mechanisms. The data processing flow of the soft thresholding attention module (STAM) is as follows:

[0112] Step a. Channel Attention calculation:

[0113] For the input feature F∈R (H×W×C) Perform global average pooling (GAP, Global Average

[0114] Pooling) and global maximum pooling (GMP), and obtain two channel description vectors v avg , v max .

[0115] The pooling results are processed using a multi-layer perceptron (MLP), and the channel attention weights are generated through a Sigmoid activation function. .

[0116] Specifically, v avg and v max Input two fully connected layers (full connection) plus a sigmoid activation function consisting of an MLP (middle layer dimension C / 4, output layer dimension C), and generate channel attention weights:

[0117]

[0118] in, Represents the Sigmoid function;

[0119] The original features Multiply element-wise with the channel attention weight to obtain the channel-weighted feature .

[0120] Step b. Spatial Attention calculation:

[0121] Features after channel weighting Perform average pooling and maximum pooling in the spatial dimension to obtain two two-dimensional feature maps S avg , S max ∈R (H×W×1) , then the two feature maps are channel-wise spliced;

[0122] The concatenated features are processed using a 7×7 convolutional layer and a Sigmoid activation function, i.e., S avg and S max After splicing, input the 7×7 convolution layer to generate spatial attention weights ;

[0123]

[0124] The spatial attention weight and Multiply element by element to obtain further enhanced features .

[0125] Step c. Soft Threshold Processing (SoftThreashold):

[0126] Spatially weighted features Perform adaptive soft thresholding:

[0127] Specifically, the input feature F is first subjected to global average pooling (GAP, Global Average Pooling); the threshold is adaptively learned using a multi-layer perceptron (MLP) ,Multilayer Perceptron (MLP) includes two fully connected layers (FullyConnection), the first fully connected layer adopts Relu activation function, and the second fully connected layer adopts Sigmoid activation function;

[0128]

[0129] Where W1 and W2 represent the weights of the two fully connected layers; Relu represents the Relu activation function, σ represents the Sigmoid activation function; GAP represents global average pooling;

[0130] Afterwards Each element in is soft-thresholded using the following formula:

[0131]

[0132] Where sign represents the sign function; Represents the output features.

[0133] In this embodiment, the soft thresholding process is performed on the features in the above manner, which can not only suppress noise interference but also retain and enhance important feature information, thereby improving the robustness of subsequent change detection.

[0134] The present invention trains the above-mentioned multimodal feature extraction network to obtain a trained model; during training, a dual-phase remote sensing image dataset is first constructed; specifically, multiple groups of time series remote sensing image pairs are collected, each group containing T1 phase and T2 phase images, and pixel-level change areas are annotated; the original images are preprocessed, including geometric correction, radiation normalization and size standardization, and all images are uniformly cropped or scaled to 256×256 pixel resolution; the dataset is divided into a training set, a validation set and a test set in a ratio of 7:1:2; the divided dataset is then used to train the multimodal feature extraction network, and to further improve the detection accuracy, the network uses a hybrid loss function to optimize the output during the training process.

[0135] In this embodiment, in order to solve the problem of small change area and imbalanced categories, a hybrid loss function with a 1:1 weighting of binary cross entropy loss (BCELoss) and Dice loss (DiceLoss) is designed. , where the BCELoss formula is described as:

[0136]

[0137] Among them, · represents the dot product operation, and They are the predicted change graph and the corresponding label value;

[0138] The Dice loss can be expressed as

[0139]

[0140] in, express norm, the total loss is expressed as

[0141]

[0142] In this embodiment, through the optimization of the hybrid loss function, the network can simultaneously focus on the change information at the pixel level and the region level, thereby improving the detection accuracy of small area changes.

[0143] Specifically, this embodiment uses publicly available remote sensing image change detection datasets (such as LEVIR-CD, CDD, etc.) for experiments; the datasets cover a variety of scenes such as towns, agriculture, and forests, with complex backgrounds and diverse change targets.

[0144] (1) Preprocessing process:

[0145] Each remote sensing image is preprocessed as follows before being input into the network:

[0146] Normalization processing: normalize pixel values to the range of [0,1];

[0147] Data augmentation, including operations such as rotation, flipping, and scaling, to increase the diversity of training data;

[0148] Perform spatial alignment on the dual-phase images to ensure that the input images are matched in the same coordinate system.

[0149] (2) Network parameters:

[0150] The initial convolution layer uses a 3×3 convolution kernel and the number of output channels is set to 32.

[0151] In the backbone network, the multi-scale convolution module in each stage uses three types of convolution kernels: 3×3, 5×5, and 7×7, and adopts a depthwise separable convolution structure, which significantly reduces the number of parameters.

[0152] In the Transformer module, the number of multi-head self-attention heads is set to 8, and the dimension of each head is 64.

[0153] In the soft thresholding attention module, the MLP structure uses two fully connected layers and the number of hidden layer units is 128.

[0154] (3) Training settings:

[0155] The optimizer uses Adam, the initial learning rate is set to 1e-4, and the learning rate decay strategy is to decrease by 20% every 10 epochs.

[0156] The batch size is set to 16 and the number of training rounds is 100 epochs.

[0157] The loss function adopts the above hybrid loss function, and the weights are appropriately balanced between BCELoss and DiceLoss at the beginning of training.

[0158] (4) Comparative experiment and result analysis:

[0159] To verify the effectiveness of this invention, we compared MSATNet with several existing mainstream change detection models (such as FC-EF and AMTNet). The main comparison indicators include precision, recall, F1-Score, and IoU value, as shown in Table 1:

[0160] Table 1 Comparative experiments on the LEVIR-CD dataset

[0161] Methods P(%) R(%) F1(%) IoU(%) FC-EF 86.91 80.17 83.40 71.53 FC-Siam-Conc 91.99 76.77 83.69 71.96 FC-Siam-Diff 89.53 83.31 86.31 75.91 DSIFN / IFN 94.02 82.93 88.13 78.77 BiT 89.24 89.37 89.30 80.68 AMTNet-50 91.82 89.71 90.76 83.08 MSATNet (ours) 91.88 90.86 91.37 84.11

[0162] Experimental results show that MSATNet's detection accuracy is improved, especially in the detection of areas with subtle changes; its robustness is enhanced. By introducing Haar wavelet transform downsampling and soft thresholding attention mechanism, the network's robustness to noise interference is significantly improved, and it can still accurately capture change information in low-contrast or high-noise environments.

[0163] Table 2 shows the comparison of the computational complexity and parameter size of each model (Input size = 256×256)

[0164] Methods FLOPs(G) Params(M) FC-EF 3.58 1.35 FC-Siam-Conc 5.33 1.55 FC-Siam-Diff 4.73 1.35 DSIFN / IFN 82.26 50.44 BiT 8.75 3.49 AMTNet-50 21.56 24.67 MSATNet (ours) 12.49 7.90

[0165] The above results show that although the present invention introduces the Transformer and multi-scale modules, the overall network parameters are still controlled within a reasonable range and are suitable for deployment in embedded devices or resource-constrained environments; the proposed model achieves higher accuracy while maintaining a smaller number of parameters.

[0166] Figure 3 A comparison chart of change detection results of LEVIR-CD dual-temporal remote sensing images is shown. It can be clearly observed that MSATNet is more accurate in detecting edge details and subtle change areas, and has better background noise suppression effect.

[0167] In practical applications, the MSATNet provided by the present invention can be embedded in a remote sensing image processing system to implement automatic change detection tasks. The specific detection method is as follows:

[0168] Step 1. After remote sensing image data is collected, normalization, enhancement, and alignment are completed through the data preprocessing module;

[0169] Step 2. The dual-temporal image is subjected to feature extraction and change detection through MSATNet, and a change probability map is output;

[0170] Step 3. The post-processing module binarizes the change probability map (setting the threshold to 0.5) and generates the final binary change area detection result.

[0171] The above detection results can be used by relevant departments for applications such as land feature monitoring, urban planning or disaster warning.

[0172] This paper designs a novel remote sensing image change detection network that addresses the technical challenges of traditional methods in multi-scale feature extraction, detail preservation, global dependency modeling, and noise suppression. This approach enables efficient and accurate detection of subtle changes in remote sensing imagery with a low number of parameters. This technical solution not only has high academic and application value but also provides reliable technical support for remote sensing monitoring tasks in practical engineering projects.

[0173] The above description of the present invention using specific examples is intended only to facilitate understanding of the present invention and is not intended to limit the present invention. A person skilled in the art of the present invention may make several simple deductions, modifications, or substitutions based on the principles of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A remote sensing image change detection system based on multimodal deep learning, characterized in that: The system uses a trained multimodal feature extraction network to detect changes in remote sensing images; the multimodal feature extraction network is composed of two weight-sharing sub-networks, which process the dual-temporal remote sensing images respectively and extract their respective feature information. The sub-networks include an initial convolutional layer, a Siamese encoding module, a Transformer global modeling module, and a cascade decoding module, which are arranged in sequence; The Siamese encoding module includes four feature extraction stages, each of which is equipped with a Haar wavelet transform downsampling module, a multi-scale depth-separable convolution module, a convolution module and a soft thresholding attention module in sequence; a channel feature exchange mechanism is introduced in the third feature extraction stage of the two sub-network Siamese encoding modules to exchange the bi-phase features output by the third feature extraction stage in the channel dimension to achieve complementarity and interaction of the bi-phase features; the bi-phase features output by the fourth feature extraction stage are input into the Transformer global modeling module to achieve global dependency modeling; the soft thresholding attention module comprehensively utilizes the channel attention and spatial attention mechanisms; The cascade decoding module includes four upsampling stages, and the features output by each feature extraction stage of the Siamese encoding module and the features output by the Transformer global modeling module are input into the cascade decoding module through a jump connection; each upsampling stage of the cascade decoding module is implemented by a decoder, that is, there are four decoders in total, and the four decoders are set in sequence, namely Decoder1, Decoder2, Decoder3, and Decoder4. The upsampled features of Decoder1 are spliced with the output features of the Transformer decoder of the Transformer global modeling module along the channel dimension; the upsampled features of Decoder2 are spliced with the output features of the third feature extraction stage of the Siamese encoding module along the channel dimension; the upsampled features of Decoder3 are spliced with the output features of the second feature extraction stage of the Siamese encoding module along the channel dimension; the upsampled features of Decoder4 are spliced with the output features of the first feature extraction stage of the Siamese encoding module along the channel dimension.

2. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The preprocessed dual-temporal remote sensing images are passed through a 3×3 initial convolutional layer to expand the number of input channels from the original value to 32 channels.

3. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The bi-phase features output from the third feature extraction stage and Perform parameter-free channel feature exchange operation and and The image is randomly divided into N sub-blocks along the channel dimension, and the corresponding sub-blocks are exchanged and reassembled to keep the total number of channels unchanged.

4. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The multi-scale depth-wise separable convolution module includes a 3×3 depth-wise separable convolution module and parallel 5×5 depth-wise separable convolution modules and 7×7 depth-wise separable convolution modules. The data processing flow is as follows: Step a. First, input the 3×3 depth-wise separable convolution module for convolution processing, followed by batch normalization and ReLU activation function; Step b. The data output from step a is split into two paths. One path is convolved with a 5×5 depthwise separable convolution module, followed by batch normalization and ReLU activation. The other path is convolved with a 7×7 depthwise separable convolution module, followed by batch normalization and ReLU activation. Step c. The feature maps output by the three sets of depth-wise separable convolutions are added or concatenated pixel by pixel to form a fused multi-scale feature map; the fused multi-scale feature map is passed through a 3×3 convolution module for inter-channel information interaction and outputs a feature map.

5. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The data processing flow of the soft thresholding attention module is as follows: Step a. Channel attention calculation: Perform global average pooling and global maximum pooling on the input feature F to obtain two channel description vectors v avg ,v max ; Use a multi-layer perceptron to process the pooling results and generate channel attention weights through the Sigmoid activation function; multiply the original features by the channel attention weights element by element to obtain the channel-weighted feature F ′ ; Step b. Spatial attention calculation: The feature F after channel weighting ′ Average pooling and maximum pooling are performed in the spatial dimension to obtain two two-dimensional feature maps, which are then channel-joined. The joined features are processed using a 7×7 convolutional layer and a Sigmoid activation function to generate spatial attention weights. The spatial attention weights are element-wise multiplied with the channel-weighted features to obtain the spatial weighted features F. ″ ; Step c. Soft thresholding: First, perform global average pooling on the input feature F, and use the multi-layer perceptron to adaptively learn the threshold λ c ; λ c =σ(W2(Relu(W1(GAP(F))))); Where W1 and W2 represent the weights of the two fully connected layers; Relu represents the Relu activation function, σ represents the Sigmoid activation function; GAP represents the global average pooling; Then the spatial weighted feature F ″ Each element in is soft-thresholded using the following formula: F out =sign(F ″ )·max(|F ″ |-λ c ,0); Where, sign represents the sign function; F out Represents the output features.

6. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The Transformer global modeling module includes a Transformer encoder and a Transformer decoder. The Transformer encoder uses a multi-head self-attention mechanism combined with position encoding to process input features and capture long-range dependencies in the image. The output of the Transformer encoder serves as the input of the Transformer decoder. While decoding the output features of the Transformer encoder, the Transformer decoder cross-integrates the features directly output by the fourth feature extraction stage, and finally generates a high-resolution feature map through fully connected layers and convolution operations.

7. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The cascade decoding module performs the following operations at each upsampling stage: Step a. Perform bilinear interpolation and 2x upsampling on the input features; Step b. Concatenate the upsampled features with the corresponding feature extraction stage features along the channel dimension; Step c. Compress the number of channels of the spliced feature map to 1 / 2 of the original decoder channel number through 1×1 convolution; Step d. Use 3×3 convolution to extract difference features and generate optimized feature maps through ReLU activation function.

8. The remote sensing image change detection system based on multimodal deep learning according to claim 1, characterized in that: The multimodal feature extraction network uses a hybrid loss function to optimize the output during training, and the hybrid loss function is composed of a binary cross entropy loss and a Dice loss weighted at a 1:1 ratio.

9. A remote sensing image change detection method based on multimodal deep learning, characterized in that: The method comprises the following steps: Step 1. After remote sensing image data is collected, normalization, enhancement, and alignment are completed through the data preprocessing module; Step 2. Perform feature extraction and change detection on the preprocessed dual-temporal image using the remote sensing image change detection system based on multimodal deep learning according to any one of claims 1 to 8, and output a change probability map; Step 3. The post-processing module binarizes the change probability map to generate the final binary change area detection result.

Images