A remote sensing image change detection method based on spatio-temporal feature fusion

Abstract

The application discloses a remote sensing image change detection method based on space-time feature fusion. Based on the cross-time gating mechanism and the cross-scale attention mechanism, the application introduces a time feature fusion module and a space feature fusion module in the remote sensing image change detection task. In the time feature fusion module, the cross-time gating mechanism is used to guide the selective enhancement of the interesting change and the suppression of the non-interesting change, so as to obtain an accurate change representation. The deepest change representation is fused with other shallow change representations through the space feature fusion module. Based on the cross-scale attention mechanism, the high-level representation is used to guide the low-level representation to model the context, so as to enhance the feature expression ability of the low-level representation, capture the fine-grained information of the interesting change and better restore the spatial details of the change representation. The application provides a new solution for the feature fusion mechanism in the change detection task under the complex remote sensing image background, and can effectively improve the change detection effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and computer vision, and particularly relates to a remote sensing image change detection method based on spatio-temporal feature fusion. BACKGROUND

[0002] Object-level change detection is a representative key task in the field of remote sensing, which is often completed on multi-temporal high spatial resolution remote sensing images. Remote sensing image change detection processes a series of spatially co-registered images taken at different time points for a specific area, and the purpose is to label each pixel as “changed” or “unchanged”. This task helps to reveal local development and has important influence on city planning, environmental monitoring, disaster assessment and other work.

[0003] Traditional remote sensing image change detection methods are combined with basic analysis units, that is, difference images are generated by simple transformations (such as change vector analysis and Markov random field model). Then thresholding or clustering analysis is used to obtain change representation. However, these methods cannot effectively detect the key information of remote sensing image change detection due to the dependence on hand-crafted features. The recent rise of deep learning enables convolutional neural networks (CNN) to promote the development of remote sensing image change detection methods based on hierarchical feature extraction. For example, IFNet and DTCDSCN adopt a U-shaped architecture to obtain change feature maps by connecting double-time image features at different spatial scales; SNUNet further adopts dense connections to capture more change details on this basis; SemiCDNet and LGPNet introduce attention mechanisms to provide more expressive change representation, thereby improving the performance of change detection. However, these methods regard remote sensing image change detection as a simple binary classification task, and only rely on feature concatenation or feature subtraction to obtain change representation, and densely connect change representations at different spatial scales to recover spatial details, and there is still much room for improvement in their performance. In addition, the dense connection design of convolutional neural networks leads to complex structures, which requires intensive computation and is not conducive to practical application.

[0004] Compared with the binary classification task, there are great challenges in change detection. First, there are a large number of false changes. In the dual-time image, interference factors (such as weather, season, light) often cause many changes that are not of interest. In many cases, a large number of changes that are not of interest are caused by interference factors such as weather, light, and season. The definition of "change" may vary from person to person and subjective consciousness. Therefore, how to emphasize the changes of interest and ignore the changes that are not of interest is particularly important for change detection. Second, the change details are high. In most cases, the number of change pixels in change detection is much smaller than the number of unchanged pixels, and it is very challenging to locate the change objects with fine structural details. Therefore, it is necessary to effectively capture the fine-grained spatial detail information of the change object.

[0005] In view of the first point, most of the previous solutions are to provide change representation through simple feature splicing or feature subtraction, and the results are often unsatisfactory. In view of the second point, most of the previous work is to connect the dense multi-scale change representation to obtain rich semantic information and spatial details. However, in the process of dense connection of high and low change representations, the low-quality boundaries of high-level features and irrelevant backgrounds of low-level features introduce more interference information, resulting in a decline in change detection results. At the same time, the use of this kind of dense connection often has a large number of parameters and complex calculation.

[0006] Therefore, for the change detection task with complex background, it is a technical problem to be solved at present to design a network with superior performance to improve the performance of image change detection. SUMMARY

[0007] A remote sensing image change detection method based on spatio-temporal feature fusion is provided. The method aims to emphasize the changes of interest in the change detection task and ignore the changes that are not of interest, and effectively capture the fine-grained spatial detail information of the change object to solve the major problems existing in the current change detection.

[0008] The specific technical solutions of the present application are as follows:

[0009] A remote sensing image change detection method based on spatio-temporal feature fusion, which combines two remote sensing images at different time points to be changed into a dual-time image, inputs a change detection model composed of an encoder module, a temporal feature fusion module (TFF), a spatial feature fusion module (SFF), and a lightweight decoder module, and obtains the change detection result between the two time points of the remote sensing image;

[0010] The encoder module first extracts different scale dual-time feature representations from shallow to deep from the dual-time images respectively by using a backbone network; then fuses the dual-time feature representations of each scale based on a cross-time gating mechanism by using an independent time feature fusion module, so as to obtain the change representation after time feature fusion at the corresponding scale, so as to emphasize the change of interest; secondly, the deepest change representation output by the last time feature fusion module is input into an independent spatial feature fusion module together with the shallow change representation output by each of the remaining time feature fusion modules, so as to capture the fine-grained information of the change representation by using the cross-scale attention and restore the spatial details thereof, so as to obtain the spatio-temporal feature fusion change representation; finally, the deepest change representation output by the time feature fusion module and the spatio-temporal feature fusion change representation output by all the spatial feature fusion modules are input into a lightweight decoder module together;

[0011] The time feature fusion module takes the dual-time feature representation as input, first performs a subtraction operation on the two time feature representations in the dual-time feature representation to obtain a rough change representation containing the difference information of the two time feature representations; then, the rough change representation is spliced with the dual-time feature representation respectively, and a deep separable convolution is used for feature integration, each integrated feature representation obtains a weight map through convolution and Sigmoid activation, and the two weight maps are used for weighted summation with the dual-time feature respectively, and then a deep separable convolution is performed to obtain the dual-time feature representation with the change of interest strengthened; finally, the dual-time feature representation with the change of interest strengthened is spliced, so as to obtain the final fine change representation as the output of the time feature fusion module;

[0012] The spatial feature fusion module takes the deepest change representation output by the last time feature fusion module and the shallow change representation output by one of the time feature fusion modules as input, up-samples the deepest change representation and splices it with the shallow change representation to obtain a fusion feature representation, takes the fusion feature representation as the query Q and the key K, and takes the shallow change representation as the value V, first calculates the weights of the 1*1 convolution result of the query Q and the 1*1 convolution result of each key K through similarity calculation, and normalizes all the weights to convert them into attention weights, then performs weighted summation on the normalized attention weights and the 1*1 convolution result of the value V, enhances the feature expression capability of the low-level change representation, and obtains an enhanced change representation; finally, the enhanced change representation is spliced with the original input shallow change representation again, so as to finally obtain the spatio-temporal feature fusion change representation and take it as the output of the spatial feature fusion module;

[0013] In the lightweight decoder module, firstly, the deepest layer change representation output by the last temporal feature fusion module and the spatio-temporal feature fusion change representation output by the remaining three spatial feature fusion modules are respectively up-sampled to the same scale and then spliced along the channel direction; then the channel attention mechanism is used to enhance the spliced change representation; finally, the enhanced change representation is up-sampled to the size of the input remote sensing image, and the change detection result between the final dual-time images is obtained.

[0014] Preferably, the backbone network is a ResNet-18 model, which includes a total of 4 residual blocks.

[0015] Preferably, in the backbone network, the first three residual blocks output three shallow feature representations of different scales, and the fourth residual block outputs the deepest change representation.

[0016] Preferably, the number of scales to be fused in the temporal feature fusion module is 4, and the four scale dual-time feature representations output by the backbone network need to be fused respectively; each scale dual-time feature representation and Firstly, a coarse change representation is obtained by subtraction = Then, respectively splice and , and integrate the features of the two spliced results through a depth separable convolution, respectively, to obtain , wherein , wherein represents a splicing operation, represents a depth separable convolution; then calculate the weight of and the weight of wherein represents a 1x1 convolution, is an activation function; finally, calculate the final fine change representation wherein represents element-wise multiplication.

[0017] Preferably, the change detection model is trained in advance using labeled training data before being used for actual change detection tasks.

[0018] Preferably, the training data all need to be randomly data-augmented.

[0019] As preferred, the loss function used in the change detection model training is a hybrid loss function, which is obtained by weighted summation of a weighted cross-entropy loss function and a dice loss function.

[0020] As preferred, the similarity calculation is realized by a dot product method.

[0021] Compared with the prior art, the present application has the following obvious benefits:

[0022] The application discloses a remote sensing image change detection method based on space-time feature fusion. The present application provides a new solution for the feature fusion mechanism in the change detection task under the complex remote sensing image background, and can effectively improve the change detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 It is a structure diagram of the STNet model;

[0024] Figure 2 It is a schematic diagram of the time feature fusion module;

[0025] Figure 3 It is a schematic diagram of the space feature fusion module;

[0026] Figure 4 It is a flow chart of the training and testing of the STNet model in the embodiment of the present application;

[0027] Figure 5 It is a test visualization result in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In order to facilitate a comprehensive understanding of the present application, many specific details of the present application are set forth in the following description. However, since the present application can be implemented in many other ways different from the methods described herein, those skilled in the art can make similar improvements without departing from the spirit of the present application, and therefore the present application is not limited to the specific embodiments shown below. The technical features of each embodiment of the present application can be combined accordingly without conflict.

[0029] Since the remote sensing image has a relatively complex background, the remote sensing image change detection task is easily affected by environmental interference factors and produces many changes that are not of interest. In many cases, a large number of changes that are not of interest are generated by distracting factors such as weather, light, etc., and the definition of "change" may differ depending on the application and subjective consciousness of a person. Therefore, how to emphasize the changes of interest and ignore the changes of non-interest is particularly important for change detection. In order to solve the above problems, a remote sensing image change detection method based on spatio-temporal feature fusion, namely STNet, is proposed in this paper.

[0030] Therefore, the remote sensing image change detection method based on spatio-temporal feature fusion provided by the present application is specifically as follows: inputting a double-time image to be subjected to a change detection task into a change detection model composed of an encoder module, a temporal feature fusion module, a spatial feature fusion module and a lightweight decoder module, and obtaining a change detection result. The double-time image in the present application is a set of two remote sensing images at different time points, and therefore the obtained change detection result is the change detection result between the two remote sensing images at different time points.

[0031] In the above-mentioned encoder module, first, a backbone network is used to extract double-time feature representations of different scales from shallow to deep from the double-time image; then, an independent temporal feature fusion module (TFF) is used to fuse the double-time feature representations at each scale based on a cross-time gating mechanism, so as to obtain the change representation after temporal feature fusion at the corresponding scale, so as to emphasize the changes of interest; secondly, the deepest change representation output by the last temporal feature fusion module is input into an independent spatial feature fusion module (SFF) together with the shallow change representations output by each of the remaining temporal feature fusion modules, so as to capture the fine-grained information of the change representation and restore its spatial details by using a cross-scale attention, thereby obtaining a spatio-temporal feature fusion change representation; finally, the deepest change representation output by the temporal feature fusion module and the spatio-temporal feature fusion change representations output by all the spatial feature fusion modules are input into a lightweight decoder module together, so as to extract the final change detection result.

[0032] It should be noted that there are multiple levels of feature extraction layers in the above backbone network, so the time feature representation extracted in the remote sensing image at each time point is also multi-scale. In the present application, the feature representation output by the final output of the backbone network is called the deepest change representation, and the feature representations extracted by the remaining intermediate feature extraction layers are all shallow feature representations. In an embodiment of the present application, the backbone network is a ResNet-18 model, which includes a total of 4 residual blocks. Therefore, when using the ResNet-18 model as the backbone network, for each remote sensing image at a time point, the first 3 residual blocks output three different scale shallow feature representations, and the fourth residual block outputs the deepest change representation. Therefore, for a double-time image composed of two remote sensing images, after passing through the ResNet-18 model, the time feature representation extracted in the remote sensing image at each time point at 4 different scales includes 4 scale feature representations from shallow to deep. Thus, each scale in the 4 scales will form two time feature representations corresponding to the two remote sensing images respectively, which are called double-time feature representations at this scale in the present application.

[0033] Specifically, the above-mentioned time feature fusion module TFF also has multiple, and the double-time feature representation at each scale in the backbone network needs to be input into a time feature fusion module. Therefore, for each TFF module, it takes the double-time feature representation at one scale as input, first performs a subtraction operation on the two time feature representations in the double-time feature representation to obtain a rough change representation containing the difference information of the two, then uses the rough change representation to respectively splice with the double-time feature representation, and uses depth separable convolution to integrate the features to obtain two integrated feature representations. Each integrated feature representation obtains a weight map through convolution and sigmoid activation, and uses two weight maps to respectively perform weighted summation with the double-time feature, thereby obtaining a double-time feature representation with interest change strengthened. Finally, the double-time feature representation with interest change strengthened is spliced, thereby obtaining the final fine change representation, which is the output of the time feature fusion module.

[0034] It should be noted that the fine change representations output by the 4 TFF modules also have a difference in depth, and the last TFF module outputs the deepest change representation, and the remaining 3 TFF modules output shallow change representations.

[0035] In addition, the above-mentioned spatial feature fusion module SFF has three modules, respectively corresponding to the three output shallow change representations of the TFF module. Each SFF module takes the deepest change representation output by the last time feature fusion module and the shallow change representation output by one of the time feature fusion modules as input, upsamples the deepest change representation, splices it with the shallow change representation to obtain a fused feature representation, takes the fused feature representation as the query Q and the key K, and takes the shallow change representation as the value V, calculates the weight by similarity between the 1*1 convolution result of the query Q and the 1*1 convolution result of each key K, normalizes all weights to convert them into attention weights, then performs weighted summation on the normalized attention weights and the 1*1 convolution result of the value V to obtain an enhanced change representation, thereby using the high-level change representation to guide the low-level representation to perform context modeling and enhance the feature expression capability of the low-level change representation. Finally, the enhanced change representation is spliced with the shallow change representation of the original input again, thereby obtaining the spatio-temporal feature fusion change representation as the output of the spatial feature fusion module.

[0036] In the above-mentioned LightDecoder module, first, the deepest change representation output by the last time feature fusion module and the spatio-temporal feature fusion change representations output by the remaining three spatial feature fusion modules are upsampled to the same scale and then spliced along the channel direction; then the change representation after splicing is enhanced by using the channel attention mechanism; finally, the enhanced change representation is upsampled to the size of the input remote sensing image, and the final change detection result between the dual-time images is obtained.

[0037] The data processing flow and calculation formula specifically performed in the STNet model in the embodiment of the application will be described in detail below. Figure 1 The overall structure diagram of the STNet model is shown in FIG. 1. The STNet model includes an encoder module and a light decoder module. The encoder module is used to obtain more accurate change representations, and the light decoder module is used to restore the spatial details of the dual-time images.

[0038] Specifically, for the encoder module, the input is the dual-time images to be completed for the change detection task, and the dimensions of the dual-time images are both , The Batch size of the input is The feature channel number of the input image is N. In the training stage, the value of the Batch size is determined according to the sample amount of each batch, The value of N in the present application is 3, , respectively. Firstly, the backbone network is ResNet-18 model. After the dual-time image is respectively extracted by the backbone network, each remote sensing image obtains 3 shallow feature representations and 1 deep feature representation of the deepest layer, so that a group of two dual-time feature representations is obtained for each scale. The 3 shallow feature representations are the outputs of the first three residual blocks of the ResNet-18 model, and the dimensions are , , . The deep feature representation is the output of the fourth residual block of the ResNet-18 model, and the dimension is , is the feature channel number of the deep feature representation. Then, the dual-time feature representations of different scales output by the backbone network are respectively input into different TFF modules which are independent of each other, to obtain the change representation after completing time fusion, and the dimension is , and are the height and width of the feature map input into the TFF, respectively; then, the deep change representation output by the TFF is input into the SFF together with the other three shallow change representations, to obtain the change representation after completing spatial fusion, and the dimension is the same as that of the input shallow change representation.

[0039] In the embodiment, for the TFF module, the purpose is to guide the selective enhancement of the change of interest and the suppression of the change of non-interest through a cross-time gating mechanism, so as to obtain better dual-time feature fusion. The number of scales to be fused by the TFF module is 4, and for the 4-scale dual-time feature representations output by the backbone network, each needs to be fused by a TFF module, so a total of 4 TFF modules are needed. As shown in Figure 2 , the specific method in each TFF module is as follows: firstly, the dual-time features corresponding to the input scale are subtracted to obtain a rough change representation containing the difference information between the two, i.e. , wherein and are the dual-time feature representations of one scale. Then, is spliced with , , and these change representations are integrated by depth separable convolution to obtain , , wherein , , wherein represents a splicing operation, represents depth separable convolution. Then, the weights of and are calculated, which are ,​ where denotes a 1x1 convolution, is an activation function. Finally, the improved fine variation representation is obtained as i.e. , where denotes an element-wise multiplication operation, denotes a concatenation operation.

[0040] The reason why the deep separable convolution is used instead of the normal convolution in the time feature module (TFF) of the present application is to greatly reduce the parameters and calculations of the model without affecting the performance of the model, which ensures the lightweight of the model.

[0041] In the present embodiment, for the SFF module, the purpose is to use the high-level variation representation to guide the low-level variation representation to conduct context modeling, enhance the feature expression ability of the low-level variation representation, and thus capture the fine-grained information of the interest variation, so as to better recover the spatial details of the variation representation, as shown in Figure 3 The present embodiment adopts three SFF modules in total, and the input of the i-th SFF module is the deepest variation representation output by the last TFF module and the shallow variation representation output by the i-th TFF module. In order to facilitate the description of the cross-scale attention mechanism used in the present application, the following first gives the general paradigm of the attention mechanism: given a feature Q = [Q1, Q2, …, Q N ]∈R N×C , where N and C are the number of pixels and the number of channels in the query Q respectively. Similarly, there are key K ∈ R N×C and value V ∈ R N×C . Then the weighted sum of each input element is calculated i.e. . Where each corresponding weight coefficient is calculated using normalization processing, i.e. . Where denotes the dot product attention arranged in proportion, i.e. The above-mentioned , , are the parameter matrices required for 1*1 convolution. For the self-attention mechanism, Q, K, and V are equivalent to the representation of a specific layer, while for the cross-scale attention mechanism proposed in the present application, Q and K are the fusion features obtained by upsampling and splicing the deep and shallow change representations of the deepest layer, and V is the shallow change representation. Therefore, the SFF module takes the deep change representation output by the fourth TFF module and the shallow change representation output by one of the TFF modules as input, upsamples the deep change representation to the same size as the shallow change representation, splices the deep change representation with the shallow change representation to complete feature fusion, obtains the fused features as Q and K, and takes the shallow change representation as V. When the attention weight coefficient is obtained according to the dot product attention mechanism , the attention weight and the 1*1 convolution result of V are weighted and summed to obtain the weighted sum , which can be used to form the enhanced change representation. The enhanced change representation is spliced with the shallow change representation of the original input again, so as to finally obtain the spatio-temporal feature fusion change representation. This operation enables the high-level representation to provide richer and more accurate context information, guides the low-level representation to model the context, and enhances the feature expression ability of the low-level representation. Therefore, the simultaneous use of TFF and SFF modules can effectively fuse the semantic information of the high-level representation and the spatial details of the low-level representation. It should be noted that in the processing of the above-mentioned SFF module, the dimensions of the data need to be reshaped according to actual needs (a reshape function can be used to achieve this).

[0042] In the lightweight decoder module, the change representations of different scales that have completed spatio-temporal feature fusion are first upsampled to the same scale and spliced to obtain the fused change representation , which is , where represents the change representation output from the first SFF module, represents the change representation output from the second SFF module, represents the change representation output from the third SFF module, represents the change representation output from the last TFF module, represents the splicing operation in the channel dimension. Then, the channel attention mechanism is used to selectively enhance the change representation, so that it pays more attention to the channel and weights different channel change representations, so as to obtain the enhanced change representation, which has a dimension of . Finally, the enhanced change representation is upsampled to the original image size, and the final change detection result between the two remote sensing images is obtained, and the dimension of the final output image is .

[0043] It should be further noted that the change detection model STNet described above must be trained in advance using labeled training data before being used for actual change detection tasks. In order to expand the training sample size, the training data can be randomly augmented. In addition, the loss function used in the training of the change detection model in the present application is a hybrid loss function, i.e., the sum of the weighted cross-entropy loss function and the dice loss function. The specific training and verification processes can be in accordance with the training and verification methods in the change detection model in the prior art, and therefore will not be described here.

[0044] The above-mentioned remote sensing image change detection method based on spatio-temporal feature fusion will be applied to a specific embodiment to demonstrate the technical effects achieved.

[0045] Embodiment

[0046] The specific network structure of the change detection model STNet used in this embodiment is shown in Figure 1 , which has been described in detail above and will not be described here. The specific process of this embodiment is shown in Figure 4 , i.e., the specific process of the change detection task can be divided into three stages, namely data preprocessing, model training and verification, testing and visualization. The testing and visualization results are shown in Figure 5 .

[0047] 1. Data preprocessing stage

[0048] This embodiment takes the remote sensing image change detection LEVIR-CD dataset and WHU dataset as an example.

[0049] For the LEVIR-CD dataset, which contains 637 pairs of high-resolution dual-time remote sensing images, the size is and the spatial resolution is 0.5 m. The original pictures are preprocessed, i.e., the images are first cut into non-overlapping pictures of size , and they are randomly divided into a training set (7120 pictures), a verification set (1024 pictures) and a test set (2048 pictures), and then the cut images are randomly augmented, i.e., random flipping and random flipping operations.

[0050] For the WHU dataset, which contains a pair of dual-time aerial images with a size of , the spatial resolution is 0.075 m. The original pictures are preprocessed, i.e., the images are first cut into non-overlapping pictures of size , and they are randomly divided into a training set (6096 pictures), a verification set (762 pictures) and a test set (762 pictures).

[0051] 2. Model training and verification

[0052] Step 1, obtain the training set data and the validation set data as described above, and batch the training set and the validation set according to a certain batch size, and the total number is N.

[0053] Step 2, sequentially select a batch of samples with index from the training data set and the validation data set, wherein The change detection model STNet is trained using each batch of training samples, and is verified and evaluated after each training. During the training process, the hybrid loss function of each training sample is calculated, and , wherein and are the weighted cross-entropy loss function and the dice loss function, respectively. On this basis, according to the total loss of each batch of training samples, the parameters of the network are comprehensively adjusted until the training is completed. After a given number of iterations is completed, the model converges, the training is completed, and the verification results during the training are obtained. At the same time, the input image is randomly horizontally, vertically flipped and rotated during training for data augmentation.

[0054] 3, test and visualization

[0055] The dual-time images of the test data set are input into the trained change detection model STNet, and the changed parts of the two images are obtained by testing the output, and a two-class image is output, completing the change detection task.

[0056] The proposed STNet model is evaluated on the above remote sensing image change detection LEVIR-CD data set and WHU data set using five common indicators: F1-score (F1), recall (Rec), precision (Pre), intersection over union (IoU) and overall accuracy (OA). In addition, two measurement standards are adopted, including the number of parameters (Params) in millions (M) and the number of floating point operations per second (Flops) in gigas (G), to measure the spatial complexity and computational cost of the network.

[0057] In this example, the test visualization result is shown in Figure 5 , and the test data result is shown in the following table.

[0058] Table 1 Test data result

[0059] Dataset F1 Pre. Rec. IoU OA LEVIR-CD 90.52 92.06 89.03 82.09 99.36 WHU 87.46 87.84 87.08 77.72 98.85

[0060] Table 2 Parameter quantity and calculation speed

[0061] Model Params(M) Flops(G) STNet 14.6 9.61

[0062] As Figure 5As shown in the above table, the change detection model STNet of the present application can well complete the change detection task for remote sensing images. Through selective enhancement of the change of interest and suppression of the change of non-interest, a more accurate change representation is obtained to emphasize the change of interest, and meanwhile, the high-level representation is used to guide the context modeling of the low-level representation to enhance the feature expression ability of the low-level representation and better recover the spatial details of the change representation. In addition, the method has good performance in terms of parameter cost and calculation cost. Therefore, the present application provides a new solution for the feature fusion mechanism in the change detection task in the complex remote sensing image background, and can effectively improve the change detection effect.

[0063] The above embodiments are only a preferred solution of the present application, but not used to limit the present application. Those skilled in the related art can also make various variations without departing from the core of the present application. Therefore, any other related technical solutions obtained by equivalent conversion or equivalent substitution method shall fall within the protection scope of the present application.

Claims

1. A method for detecting changes in remote sensing images based on spatiotemporal feature fusion, characterized in that: The remote sensing images of two different time points to be changed are combined into a dual-time image, which is then input into a change detection model consisting of an encoder module, a temporal feature fusion module, a spatial feature fusion module, and a lightweight decoder module to obtain the change detection results between the two time points. The encoder module first uses a backbone network to extract dual-temporal feature representations from dual-temporal images at different scales from shallow to deep. Then, an independent temporal feature fusion module is used to fuse the dual temporal feature representations at each scale based on a cross-temporal gating mechanism, thereby obtaining the temporal feature fusion change representation at the corresponding scale to emphasize the changes of interest. Next, the deepest change representation output by the last temporal feature fusion module and the shallow change representation output by each of the other temporal feature fusion modules are input into an independent spatial feature fusion module. Cross-scale attention is used to capture the fine-grained information of the change representation and restore its spatial details to obtain the spatiotemporal feature fusion change representation. Finally, the deepest change representation output by the temporal feature fusion module and the spatiotemporal feature fusion change representations output by all the other spatial feature fusion modules are input into the lightweight decoder module. The temporal feature fusion module takes a dual temporal feature representation as input. First, it subtracts the two temporal feature representations in the dual temporal feature representation to obtain a coarse change representation containing the difference information between the two. Then, it concatenates the coarse change representation with the dual temporal feature representation and integrates the features using depthwise separable convolution. Each integrated feature representation obtains a weight map through convolution and sigmoid activation. The two weight maps are then weighted and summed with the dual temporal features respectively. Finally, it is subjected to depthwise separable convolution to obtain a dual temporal feature representation enhanced with interest changes. Finally, the dual temporal feature representations enhanced with interest changes are concatenated to obtain the final refined change representation as the output of the temporal feature fusion module; The spatial feature fusion module takes the deepest change representation output by the last temporal feature fusion module and the shallow change representation output by one of the temporal feature fusion modules as input. The deepest change representation is upsampled and concatenated with the shallow change representation to obtain a fused feature representation. This fused feature representation is used as the query Q and key K, and the shallow change representation as the value V. First, the 1*1 convolution results of query Q and each key K are weighted using similarity calculations, and all weights are normalized to convert them into attention weights. Then, the normalized attention weights and the 1*1 convolution results of value V are weighted and summed to enhance the feature representation capability of the low-level change representation, resulting in an enhanced change representation. Finally, this enhanced change representation is concatenated again with the original input shallow change representation to obtain the spatiotemporal feature fusion change representation, which is then used as the output of the spatial feature fusion module. In the lightweight decoder module, the deepest change representation output by the last temporal feature fusion module and the spatiotemporal feature fusion change representation output by the other three spatial feature fusion modules are first upsampled to the same scale and then stitched together along the channel direction. Then, the channel attention mechanism is used to enhance the stitched change representation. Finally, the enhanced change representation is upsampled to the size of the input remote sensing image, and the final change detection result between the two temporal images is obtained.

2. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The backbone network is a ResNet-18 model, which contains a total of 4 residual blocks.

3. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 2, characterized in that, In the backbone network, the first three residual blocks output shallow feature representations at three different scales, and the fourth residual block outputs the deepest change representation.

4. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The temporal feature fusion module requires fusion of four scales. For the four scales of dual-temporal feature representations output by the backbone network, fusion is required for each scale. and First, obtain the roughness representation by subtraction. = Then respectively and , The two concatenated results are then combined using depthwise separable convolutions to obtain the following features: , ,in , ,in This indicates a splicing operation. Represent depthwise separable convolution; then compute weights and weights ,in Represents a 1×1 convolution. Use an activation function; finally, compute the final refined representation. ,in This indicates element-wise multiplication.

5. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The change detection model is trained in advance using labeled training data before being used in actual change detection tasks.

6. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 5, characterized in that, All training data must undergo random data augmentation.

7. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The loss function used in training the change detection model is a hybrid loss function, which is obtained by weighted summation of weighted cross-entropy loss function and dice loss function.

8. The remote sensing image change detection method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The similarity calculation is performed using the dot product method.

Images