Remote sensing image change detection method, device and system and storage medium

Abstract

The invention discloses a remote sensing image change detection method, device and system and a storage medium. The method comprises the following steps: S1, acquiring a historical remote sensing image data set; s2, preprocessing the historical remote sensing image data set; s3, training a symmetric dual-frequency feature enhanced network hybrid model according to the preprocessed historical remote sensing image data; and S4, inputting the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model to carry out remote sensing image change detection. By adopting the technical scheme of the invention, more complete and continuously changing regions can be detected, and adjacent changing regions of interest can be distinguished more accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a remote sensing image change detection method, device, system and storage medium. Background Art

[0002] Compared with traditional surveying and mapping data, remote sensing images have various advantages, such as a wide detection range, high frequency, few restrictions on data collection, and rich information. Therefore, remote sensing images are a very suitable data type for large-scale land use and change research. From the perspective of data sets, change detection tasks can be divided into several types, including synthetic aperture radar (SAR) change detection, multispectral change detection, hyperspectral change detection, very high resolution change detection, and heterogeneous image change detection. In recent years, very high resolution images have become increasingly easy to obtain. They provide more detailed information about ground objects than medium and low resolution images, especially man-made structures. Therefore, change detection methods on very high resolution remote sensing images are applicable to agricultural surveys, disaster assessments, land cover monitoring, urban expansion research, and urban internal change analysis. Convolutional neural networks (CNNs) and Transformers are currently widely used frameworks for remote sensing image change detection. However, they are prone to missing detections and discontinuities in the monitored areas, and there are problems with low detection accuracy. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a remote sensing image change detection method, device, system and storage medium.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A remote sensing image change detection method includes:

[0006] Step S1, obtaining a historical remote sensing image data set;

[0007] Step S2, preprocessing the historical remote sensing image data set;

[0008] Step S3, training a symmetric dual-frequency feature enhanced network hybrid model according to the preprocessed historical remote sensing image data;

[0009] Step S4, inputting the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model for remote sensing image change detection.

[0010] Preferably, preprocessing the historical remote sensing image data set includes: image segmentation and image enhancement.

[0011] Preferably, the symmetric dual-frequency feature enhanced network hybrid model includes: a feature extraction module, a frequency module I, a token-based enhanced Transformer module, a frequency module II, and a detection head module; wherein, the feature extraction module consists of a dual-branch HCT; the frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; the frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetric to the frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

[0012] The present invention also provides a remote sensing image change detection device, including:

[0013] An acquisition module, configured to acquire a historical remote sensing image data set;

[0014] A preprocessing module, configured to preprocess the historical remote sensing image data set;

[0015] A training module, configured to train the symmetric dual-frequency feature enhanced network hybrid model according to the preprocessed historical remote sensing image data;

[0016] A detection module, configured to input the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model for remote sensing image change detection.

[0017] Preferably, the preprocessing module preprocessing the historical remote sensing image data set includes: image segmentation and image enhancement.

[0018] Preferably, the symmetric dual-frequency feature enhanced network hybrid model includes: a feature extraction module, a frequency module I, a token-based enhanced Transformer module, a frequency module II, and a detection head module; wherein, the feature extraction module consists of a dual-branch HCT; the frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; the frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetric to the frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

[0019] The present invention also provides a remote sensing image change detection system, including: a memory and a processor, where a computer program run by the processor is stored on the memory, and the computer program executes a remote sensing image change detection method when run by the processor.

[0020] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes a remote sensing image change detection method when running.

[0021] The present invention adopts a hybrid network of CNN and Transformer with symmetric dual-frequency feature enhancement to effectively mine the change information of interest. First, a dual-branch feature extraction module is used to accurately extract the original local and global features of the dual-temporal remote sensing images; second, a frequency module I is constructed to extract the frequency components of these original features; third, an enhanced token mining module based on KAN is used to learn better semantic information. Finally, the frequency components of the semantic change information beneficial to the final change detection are mined from the frequency module II. By adopting the technical solution of the present invention, more complete and continuous changed areas can be detected, and adjacent change regions of interest can be distinguished more accurately. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0023] Figure 1 It is a flowchart of the remote sensing image change detection method according to the embodiment of the present invention;

[0024] Figure 2 It is a schematic structural diagram of a symmetric dual-frequency feature enhanced network hybrid model;

[0025] Figure 3 It is a schematic structural diagram of the feature extraction module;

[0026] Figure 4 It is a schematic structural diagram of the CKSA block. Detailed Embodiments

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0028] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Example 1:

[0030] As Figure 1 shown, an embodiment of the present invention provides a remote sensing image change detection method, including:

[0031] Step S1, obtaining a historical remote sensing image data set;

[0032] Step S2, preprocessing the historical remote sensing image data set, where the preprocessing includes: image segmentation and image enhancement;

[0033] Step S3, training a symmetric dual-frequency feature enhanced network hybrid model according to the preprocessed historical remote sensing image data;

[0034] Step S4, inputting the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model for remote sensing image change detection.

[0035] As an implementation manner of the embodiment of the present invention, as Figure 2 shown, the symmetric dual-frequency feature enhanced network hybrid model includes: a feature extraction module, a frequency module I, a token-based enhanced Transformer module, a frequency module II, and a detection head module; among them, the feature extraction module is composed of a dual-branch HCT; the frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; the frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetric to the frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

[0036] Inspired by the hybrid model and the CMTFNet model, a dual-HCT module, called the feature extraction module, is constructed to fuse local features and global features. The two branches of the feature extraction module share parameters and have the same structure. Each branch is an HCT, and the HCT includes: an encoder and a decoder, as Figure 3As shown in the figure. The encoder consists of a 7×7 CNN and four ResNet50 blocks. Among them, the CNN scales the input remote sensing image (3×H0×W0) to the feature map e0 (64×H0 / 4×W0 / 4). Each subsequent ResNet 50 extracts local features step by step to obtain e1 (256×H0 / 4×W0 / 4), e2 (512×H0 / 8×W0 / 8), e3 (1024×H0 / 16×W0 / 16), and e4 (2048×H0 / 32×W0 / 32). The decoder part consists of 3 Transformer blocks, which are used to decode multi-scale global context features, and generate feature maps d3 (512×H0 / 16×W0 / 16), d2 (512×H0 / 8×W0 / 8), and d1 (256×H0 / 4×W0 / 4) respectively. In addition, in order to fuse the local hierarchical features and the global context features, fusion operations are performed after the first three decoder modules respectively. It should be noted that, for the convenience of subsequent feature fusion operations, a CNN is used to perform dimensional transformation on e3 and e4, obtaining e3’ (512×H0 / 16×W0 / 16) and e4’ (512×H0 / 32×W0 / 32) respectively. In the three fusion operations, the first two fusions capture rich local features and global context information, but lack spatial details. Therefore, the third fusion is also very important for integrating the spatial features from the first CNN module. The decoder generates multi-scale global context information and gradually restores the spatial resolution of the features by fusing the hierarchical features obtained from the CNN module. In addition, a learnable variable is used during the fusion process to balance the importance of local features and global context information. Therefore, the contributions of these two elements to the output can be expressed as:

[0037] O = α×O E +(1 - α)×O D

[0038] In the formula, O represents the output result of the fusion operation, α represents the learnable variable, O E represents the feature output of the encoder part, while O D represents the global context information output by the decoder.

[0039] Combining the spectral layer and the multi-head attention mechanism enables the model to achieve state-of-the-art performance. Therefore, a joint module consisting of frequency module I, a token-based enhanced Transformer module, and frequency module II is designed, and these modules follow the feature extraction module. Frequency module I generates first-order frequency features, which help to represent the frequency information in each original feature image. In this study, frequency module I mainly consists of a fast Fourier transform (FFT) layer, a weighted gating unit, and an inverse Fourier transform (IFFT) layer, and its expression is:

[0040] X FFT = IFFT(Gate(FFT(X FE )))

[0041] In the formula, X FFT represents the output of frequency module I, and X FE is the original feature map extracted by the feature extraction module. FFT, Gate, and IFFT represent fast Fourier transform, weighted gating unit, and inverse Fourier transform respectively.

[0042] The FFT layer converts the feature map from the physical space to the frequency spectrum space. The weighted gating unit, as a learnable weight parameter in the neural network, adjusts its weights through backpropagation during the training process, effectively identifying the frequency domain features in the feature map, thereby determining the importance of each frequency component in the feature representation. The inverse FFT converts the feature map from the frequency spectrum space back to the spatial domain, thereby generating frequency features with enhanced details, called first-order frequency features.

[0043] Finally, the output of frequency module I is connected through a residual connection to retain the characteristics in the original feature image, as shown in the following formula:

[0044] X HFFT = X FFT + X FE

[0045] In the formula, X HFFT represents the output of frequency module I.

[0046] The feature extraction module extracts and fuses multi-scale features from the dual-temporal remote sensing images. Then, the first-order frequency features are obtained using frequency module I. The token-based enhanced Transformer module is used for semantic token extraction and high-level semantic information perception. The token-based enhanced Transformer module consists of two units: the channel and spatial attention (CKSA) block based on KAN and the Transformer unit.

[0047] Semantic token interoperability helps to interact with change information in the remote sensing change detection task. Semantic tokens represent high-level concepts of interest in changes and are one of the key elements in change detection. In addition, the applicability and effectiveness of KAN have been verified in the fields of computer vision and remote sensing. Inspired by the ability of the KAN layer to promote customized activation learning at the network edge and calculate the contribution of each input channel, the CKSA block was first designed, as Figure 4As shown in the figure. The CKSA block mainly consists of two parts: a channel attention unit and a spatial attention unit. In the channel attention unit, the output of the frequency module I is used as the input, and max pooling (MaxPool) and average pooling (AvgPool) operations are respectively performed, and the dimension is compressed (Flatten) into a one-dimensional array. The KAN is used to replace the original fully connected neural network to efficiently extract the contribution values of different channels. Finally, the one-dimensional array is mapped (View) back to a three-dimensional array, and each channel obtains a weight value. The weight value of each channel is multiplied by the output of the frequency module I to obtain a feature map with channel weight information. In the spatial attention unit, the feature map with channel weight information is compressed in the channel dimension (two compression methods: max and mean), and the compression results are concatenated (Concat). Then, a convolutional layer is used to learn the spatial position relationship, and the relationship result is reprojected (Reporoject) into a feature map with 1 channel. Finally, the weight value is multiplied by the feature map after channel attention position by position to obtain a feature map with the same dimension as the output result of the frequency module I. At the end of the CKSA module, the feature map with both channel attention weights and spatial attention weights needs to be transformed (Reporoject) again to generate two concentrated token sets for accurately learning semantic tokens within the module. The CKSA block is mainly composed of channel and spatial attention units. Specifically, the KAN layer is used to replace the fully connected layer. In CKSA, the process of converting the feature image into tokens can be expressed as:

[0048] X CKSA = SA(CKA(X HFFT ))

[0049] In the formula, X CKSA represents the output of the CKSA module, CKA represents the channel attention unit based on KAN, and SA represents the spatial attention unit.

[0050] CKSA obtains tokens of image features. These tokens contain rich details of the changes in the feature image but lack the semantic information of the interaction relationships between tokens. The Transformer can make full use of the high-level global semantic relationships in the token space. Therefore, a Transformer block is introduced in the subsequent stage of the token-based enhanced Transformer module. First, two sets of tokens obtained from CKSA are concatenated to form a token cluster, and then it is input into the encoder of the Transformer to capture the global semantic context between these tokens. Since the token cluster concatenates the semantic token sets along the second dimension (dim = 1), it can be compared to binding two bundles of token sets together. Therefore, the Transformer encoder can extract the internal relationships within a set of tokens and the interrelationships between two sets of semantic tokens. The output of the Transformer encoder not only has rich high-level semantic information within the tokens but also has rich global semantic information between the tokens.

[0051] The high-level semantic context information is divided into two sets of contexts, and the dimension of each set of contexts is the same as that of the tokens. These two sets of contexts encapsulate the compact semantic context and are used to express the high-level information of the change hotspots. Subsequently, the Transformer decoder restores these two sets of contexts to the pixel space to generate a two-branch semantic pixel map. The two-branch pixel map containing high-quality semantic information enables each pixel in the map to be represented by these two sets of contexts. This representation effectively highlights the pixel values of interest in the semantic map.

[0052] The high-level semantic pixel map effectively reveals the semantic hotspots in the feature space. Subsequently, by subtracting a pair of high-level semantic pixel maps, a semantic difference map is obtained, which can be used to represent the changed semantic information. The subtraction between the high-level semantic pixel maps may produce positive and negative results. To ensure that all values are non-negative, any negative result is converted to its absolute value, defined as follows:

[0053] X SUB =|X SPFM1 -X SPFM2 |

[0054] In the formula, X SPFM1 and X SPFM2 respectively represent the high-level semantic pixel maps output by the token-based enhanced Transformer module, and X SUB represents the semantic difference map.

[0055] The HFFT of frequency module I and the BFFT of frequency module II are symmetrically distributed in the early and late stages of the model; similar to HFFT, BFFT also includes an FFT layer, a weighted gate, an IFFT layer, and a residual connection. BFFT is used to generate a second-order semantic difference frequency map. The formula of frequency module II is expressed as follows:

[0056] X BFFT = IFFT(Gate(FFT(X SUB )))+X SUB

[0057] In the formula, X BFFT represents the semantic difference frequency map containing second-order semantic difference frequency information output by frequency module II.

[0058] Frequency module II first scales the semantic difference map to match the dimensions of the original remote sensing image. BFFT converts the physical space of the rescaled semantic difference map into the spectral space, where the frequency information of the semantic difference map is depicted. Then, it restores the depicted detailed information to the physical space, thereby generating second-order semantic difference frequency information.

[0059] The second-order semantic difference information in the semantic difference frequency map represents the final semantic information generated by the model and performs a skip connection with the semantic difference map to enhance the frequency information of the semantic difference map. The semantic difference map with enhanced frequency information is directly used in the detection head module to distinguish the changed area and the background area. The detection head uses a fully convolutional network to generate a change map, and the dimension of this change map is R H0×W0×2 , where H0 and W0 represent the height and width of the original dual-temporal remote sensing image, respectively.

[0060] The symmetric dual-frequency feature enhanced network hybrid model uses a cross-entropy loss function, and the loss function is expressed as:

[0061]

[0062] In the formula, L represents the cross-entropy loss function, G represents the ground truth, and P represents the predicted value.

[0063] The embodiment of the present invention is a symmetric dual - frequency feature - enhanced network hybrid model, aiming to effectively extract original multi - scale image features, first - order frequency components, semantic tokens, and second - order semantic difference frequency information from dual - temporal remote sensing images. Specifically, inspired by the performance improvement of the CNN and Transformer hybrid model (HCT), the embodiment of the present invention constructs ResNet and Transformer modules in the initial layer, aiming to extract and fuse multi - scale image features, including local and global features, rather than the low - level image features commonly used in other change detection methods. Subsequently, a residual fast Fourier transform (HFFT) is integrated into the shallow layer of the symmetric dual - frequency feature - enhanced network hybrid model to provide a frequency attention mechanism. This layer is specifically used to perceive the first - order frequency features by analyzing the different frequency components of each feature image. Next, a channel attention mechanism based on Kolmogorov - Arnold (KAN) combined with spatial attention (CKSA) blocks is used to extract semantic tokens. Then, an encoder - decoder structure is used to learn the global spatial context and high - level semantic information. In addition, a symmetric residual fast Fourier transform (BFFT) module is used to extract semantic difference frequency information in the deep structure of the symmetric dual - frequency feature - enhanced network hybrid model. The embodiment of the present invention introduces a dual - branch hybrid method of CNN and Transformer blocks to extract local and global features. These features are then fused to generate a feature image, significantly improving the ability to represent low - level features. The channel attention mechanism of CKSA designed based on KAN is used to calculate the contribution of each input channel, and the spatial attention mechanism is used to perceive hotspots, enhancing the ability to learn tokens. Symmetric HFFT and BFFT layers are used. The frequency components of the feature image are refined through the HFFT layer to obtain the first - order frequency features and through the BFFT to obtain the second - order semantic difference frequency information. The first - order frequency features and the second - order semantic difference frequency information are crucial for improving the expression of low - level features and high - level change information.

[0064] Embodiment 2:

[0065] The embodiment of the present invention also provides a remote sensing image change detection device, including:

[0066] An acquisition module, configured to acquire a historical remote sensing image dataset;

[0067] A pre - processing module, configured to pre - process the historical remote sensing image dataset;

[0068] A training module, configured to train a symmetric dual - frequency feature - enhanced network hybrid model according to the pre - processed historical remote sensing image data;

[0069] The detection module is used to input the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model for remote sensing image change detection.

[0070] As an implementation manner of the embodiment of the present invention, the preprocessing module preprocesses the historical remote sensing image dataset, including: image segmentation and image enhancement.

[0071] As an implementation manner of the embodiment of the present invention, the symmetric dual-frequency feature enhanced network hybrid model includes: a feature extraction module, a frequency module I, a token-based enhanced Transformer module, a frequency module II, and a detection head module; wherein, the feature extraction module consists of a two-branch HCT; the frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; the frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetric to the frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

[0072] Embodiment 3:

[0073] The embodiment of the present invention also provides a remote sensing image change detection system, including: a memory and a processor, wherein a computer program run by the processor is stored on the memory, and the computer program executes the remote sensing image change detection method when run by the processor.

[0074] Embodiment 4:

[0075] The embodiment of the present invention also provides a storage medium, on which a computer program is stored, and the computer program executes the remote sensing image change detection method when running.

[0076] The above embodiments are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A remote sensing image change detection method, characterized in that: include: Step S1, obtaining a historical remote sensing image dataset; Step S2, preprocessing the historical remote sensing image data set; Step S3, training a symmetric dual-frequency feature enhanced network hybrid model according to the preprocessed historical remote sensing image data; Step S4: input the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model to perform remote sensing image change detection.

2. The remote sensing image change detection method as claimed in claim 1, characterized in that: Preprocessing of historical remote sensing image datasets includes image segmentation and image enhancement.

3. The remote sensing image change detection method as claimed in claim 1, characterized in that: The symmetric dual-frequency feature enhanced network hybrid model includes: feature extraction module, frequency module I, token-based enhanced Transformer module, frequency module II and detection head module; among them, the feature extraction module is composed of a double-branch HCT; frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetrical with frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

4. A remote sensing image change detection device, characterized in that: include: Acquisition module, used to acquire historical remote sensing image datasets; Preprocessing module, used to preprocess historical remote sensing image datasets; A training module, used to train a symmetric dual-frequency feature enhanced network hybrid model based on preprocessed historical remote sensing image data; The detection module is used to input the remote sensing image data of the target area into the trained symmetric dual-frequency feature enhanced network hybrid model for remote sensing image change detection.

5. The remote sensing image change detection device as claimed in claim 4, characterized in that: The preprocessing module preprocesses the historical remote sensing image dataset including image segmentation and image enhancement.

6. The remote sensing image change detection device as claimed in claim 4, characterized in that: The symmetric dual-frequency feature enhanced network hybrid model includes: feature extraction module, frequency module I, token-based enhanced Transformer module, frequency module II and detection head module; among them, the feature extraction module is composed of a double-branch HCT; frequency module I is used to extract the frequency details of multi-level features and generate first-order frequency features. The token-based enhanced Transformer module uses the first-order frequency features as input to obtain semantic information; frequency module II is located in the deep layer of the symmetric dual-frequency feature enhanced network hybrid model, symmetrical with frequency module I, and is used to extract the frequency components of the semantic information of the token-based enhanced Transformer module and generate second-order semantic difference frequency information; the detection head module is used to generate a change map.

7. A remote sensing image change detection system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the remote sensing image change detection method according to any one of claims 1 to 3 is executed.

8. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the remote sensing image change detection method according to any one of claims 1 to 3.

Images