Heterogeneous remote sensing image deep learning change detection method and device, and storage medium

Abstract

The invention discloses a heterogenous remote sensing image deep learning change detection method and device, and a storage medium. The method comprises the following steps: S1, obtaining heterogenous remote sensing image data; s2, obtaining a matching optimization-based heterogenous image change detection model according to the heterogenous remote sensing image data; and S3, inputting remote sensing image data to be processed into the matching optimization-based heterogenous image change detection model to realize end-to-end heterogenous remote sensing image change region accurate detection. By adopting the technical scheme of the invention, the high-precision identification of the change area between the different-source images is realized.

Description

Technical Field

[0001] This invention belongs to the field of information processing technology, and in particular relates to a method, apparatus, and storage medium for detecting changes in heterogeneous remote sensing images using deep learning. Background Technology

[0002] With the rapid development of remote sensing technology, change detection of multimodal and multitemporal heterogeneous remote sensing images has become a core requirement in fields such as geological disaster monitoring and environmental change analysis. However, due to significant differences in the imaging principles, radiometric responses, spatial resolutions, and noise characteristics of different sensors, heterogeneous remote sensing images often face problems such as accumulated matching errors, loss of differential semantic features, and limited feature interaction in traditional change detection, which seriously affects the accuracy of ground feature change detection. Summary of the Invention

[0003] This invention provides a method, apparatus, and storage medium for detecting changes in heterogeneous remote sensing images using deep learning.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A deep learning-based change detection method for heterogeneous remote sensing images includes: Step S1: Acquire heterogeneous remote sensing image data; Step S2: Based on the heterogeneous remote sensing image data, obtain a heterogeneous image change detection model based on matching optimization; Step S3: Input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve accurate end-to-end detection of heterogeneous remote sensing image change areas.

[0005] As a preferred approach, the heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature collaborative change detection module. The multi-dimensional feature matching stage utilizes a dual-branch network structure to acquire image spatial-spectral features and performs feature fusion and matching. The semantic feature interaction module employs self-attention and cross-attention mechanisms to perform in-depth information interaction and refinement of the fused features. The feature collaborative change detection module further integrates and refines the multi-scale fused features to achieve accurate localization and classification of changed regions.

[0006] As a preferred option, the loss function of the heterogeneous image change detection model based on matching optimization is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

[0007] The present invention also provides a heterogeneous remote sensing image deep learning change detection device, comprising: The first processing module is used to acquire remote sensing image data from different sources. The second processing module is used to obtain a heterogeneous image change detection model based on matching optimization based on heterogeneous remote sensing image data. The third processing module is used to input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve end-to-end accurate detection of heterogeneous remote sensing image change areas.

[0008] As a preferred approach, the heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature collaborative change detection module. The multi-dimensional feature matching stage utilizes a dual-branch network structure to acquire image spatial-spectral features and performs feature fusion and matching. The semantic feature interaction module employs self-attention and cross-attention mechanisms to perform in-depth information interaction and refinement of the fused features. The feature collaborative change detection module further integrates and refines the multi-scale fused features to achieve accurate localization and classification of changed regions.

[0009] As a preferred option, the loss function of the heterogeneous image change detection model based on matching optimization is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

[0010] The present invention also provides a storage medium storing a computer program, which executes a deep learning change detection method for heterogeneous remote sensing images during runtime.

[0011] This invention focuses on the collaborative optimization of image matching and change detection in heterogeneous remote sensing images. It proposes an end-to-end integrated "matching-detection" network framework based on deep learning, which realizes multimodal feature extraction, semantic information compensation, and deep integration of matching and detection processes, providing a highly robust solution for ground feature change detection in complex scenarios. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0013] Figure 1 This is a flowchart of the deep learning change detection method for heterogeneous remote sensing images according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a heterogeneous image change detection model based on matching optimization. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Example 1: like Figure 1 As shown, this embodiment of the invention provides a deep learning-based change detection method for heterogeneous remote sensing images, comprising: Step S1: Acquire heterogeneous remote sensing image data; Step S2: Based on the heterogeneous remote sensing image data, obtain a heterogeneous image change detection model based on matching optimization; Step S3: Input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve accurate end-to-end detection of heterogeneous remote sensing image change areas.

[0017] Furthermore, the core idea of ​​the matching-optimized heterogeneous image change detection model is to fully utilize multi-dimensional information such as spatial and spectral data, and through multi-level feature fusion and semantic interaction mechanisms, achieve end-to-end accurate detection of heterogeneous remote sensing image change areas. For example... Figure 2As shown, the heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature-coordinated change detection module. In the multi-dimensional feature matching stage, the model utilizes a dual-branch network structure to acquire image spatial-spectral features and performs feature fusion and matching to mitigate differences in modality and resolution between heterogeneous images, reducing step-by-step matching errors. Then, the self-attention and cross-attention mechanisms of the semantic feature interaction module are used to perform deep information interaction and refinement of the fused features, enhancing the globality and consistency of feature representation. Finally, the feature-coordinated change detection module further integrates and refines the multi-scale fused features, achieving accurate localization and classification of changed regions. The MSCDNet model, in its design, not only considers the inherent multi-dimensional feature differences of heterogeneous images but also takes into account the complementarity of global and local information, thus exhibiting strong robustness and adaptability in change detection tasks.

[0018] The working process of the multidimensional feature matching module is as follows: The multidimensional feature matching module employs a pseudo-Twin network structure to extract spatial and spectral features from heterogeneous remote sensing images, enabling cross-modal image matching. In the feature extraction stage, this module uses a dual-branch design similar to the SSSM network, with one branch focusing on extracting spatial structure information and the other on acquiring spectral features. Through multi-scale processing, these two branches capture global spatial structure and local fine-grained spectral information, respectively, and effectively integrate them in the fusion layer to generate a high-dimensional fused feature map. Simultaneously, to further enhance the expressive power of the fused features and eliminate noise and pseudo-matches between heterogeneous images, this embodiment introduces a semantic attention branch structure after acquiring the fused features for semantic matching correction. This attention module automatically assigns greater weight to regions with high semantic information and discriminative power in the fused features through an adaptive weighting mechanism, while suppressing low-quality features significantly affected by image radiometric differences and geometric distortions. The attention branch not only strengthens the semantic consistency of features from a global perspective but also effectively improves the accuracy of matching points during local matching correction. Subsequently, based on the corrected semantic information, the module enters the localization and matching stage. By utilizing local spatial relationships and global geometric constraints, key points are meticulously corrected, thereby outputting features optimized by localization and matching. This step ensures that the same object in the image can obtain consistent feature representations at the same spatial location under different modalities by minimizing the geometric errors between matching points, providing a more stable and robust feature input for subsequent change detection tasks.

[0019] Furthermore, a dual-branch design is used for spatial-spectral feature extraction and fusion, mainly consisting of two parts: spatial-spectral feature extraction and spatial-spectral feature fusion. Throughout the module, all convolutional operations employ batch normalization (BN) layers and ReLU activation function layers to ensure the stability and normalization effect of features after nonlinear activation, thereby improving the network's convergence speed and generalization ability. In the spatial-spectral feature extraction stage, the model adopts a parallel design, simultaneously performing spatial and spectral feature extraction. Specifically, the spatial feature extraction branch uses a 7×7 convolutional layer as the initial convolutional layer, expanding the number of channels in the input image from the original to 64. This operation effectively captures a large range of spatial structural information, providing a good initial representation for subsequent deep feature extraction. Then, this branch uses two residual blocks to perform deep feature extraction on the image, resulting in a feature map with a scale of 1 / 8 of the original image. In this process, each residual block contains two 3×3 convolutional operations, and information is effectively transferred through residual connections, ensuring the preservation of low-level detail information while enhancing the expression of high-level semantic features. Following the residual block, to further enhance the robustness of feature representation and the ability to fuse information across scales, a 1×1 convolutional layer is introduced into the spatial feature extraction branch. This layer primarily serves for channel fusion and feature recombination. Subsequently, an upsampling operation is used to enlarge the obtained feature map to twice the current scale, and it is then element-wise added and convolved with the features output from the residual block at the same scale. This achieves effective fusion of shallow and deep features, further enhancing the representational ability of the feature map at different levels. This process is repeated multiple times until a spatial feature map with a scale half that of the original image is generated. Unlike the spatial feature extraction branch, the spectral feature extraction branch focuses on local details and frequency domain information. This branch uses a 3×3 convolution as the initial convolutional layer, using a smaller convolutional kernel to more sensitively capture subtle texture changes and spectral features in the image. Subsequently, the network structure of the spectral branch remains consistent with that of the spatial branch, ultimately generating the corresponding spectral feature map. In the spatial-spectral feature fusion stage, this invention proposes a multi-scale fusion strategy, breaking away from the limitation of traditional network designs that only utilize the final feature map for fusion. The core idea of ​​this stage is to utilize the multi-scale feature maps obtained during feature extraction, and to achieve a gradual fusion of coarse-scale to fine-scale feature maps by performing concatenation, upsampling, and other operations on spatial and spectral features at each scale. The model starts with a coarse-scale feature map at 1 / 8 of the original image scale. It first performs channel-level concatenation of the coarse-scale features output from the spatial and spectral branches, and then uses 1×1 convolution to fuse and reorganize the concatenated features, thereby improving both the semantic information and the detail representation of the fused features.Subsequently, the feature map is upsampled to twice its original size and then element-wise added to a pre-extracted feature map at the same scale. This is followed by convolution to further fuse and refine the features. This process is repeated to obtain a fine-scale fused feature map at half the scale of the original image. The fused feature maps at 1 / 8 and 1 / 2 scales are then output to subsequent modules to perform image matching tasks on heterogeneous remote sensing images. In the SSSM module, spatial-spectral features are effectively integrated through a multi-scale, hierarchical fusion strategy. This results in feature maps that possess rich semantic information as well as a large amount of fine-grained spatial and spectral details, improving the accuracy of feature matching and enhancing the model's adaptability to complex backgrounds and heterogeneous image information.

[0020] Furthermore, the semantic attention branch structure performs semantic correction on the key point selection process through deep feature transformation to obtain feature maps with rich semantic information, thereby achieving semantic matching optimization. The semantic attention branch module mainly includes three processing stages: semantic feature extraction, correction information acquisition, and semantic correction.

[0021] In the semantic feature extraction stage, to improve the capture and utilization of high-level semantic information during heterogeneous image matching, an independent semantic feature acquisition branch is introduced into the SACM architecture. This branch mainly consists of 1×1 convolutional layers, Batch Normalization (BN) layers, and the ReLU activation function. The 1×1 convolutional layers are used to reconstruct and linearly transform the high-dimensional feature maps output by the feature extraction module, effectively integrating the semantic information between channels. Batch Normalization (BN) layers are introduced to stabilize the network training process, reduce internal covariate shifts, and make the feature distribution more stable, thereby improving the model's generalization ability. Simultaneously, the ReLU activation function is used to introduce nonlinear transformations, further enhancing the sparsity and discriminative power of feature representation.

[0022] During the information acquisition phase, the semantic attention branch module calculates feature weights for each acquired semantic feature pixel by pixel. A And semantic offsets. Both are passed through a 1×1 convolutional layer and a sigmoid activation function. Extraction is performed, where deep semantic features are calculated during feature weighting. S The semantic attention weight map is mapped to a single channel, and the output semantic attention coefficients range from 0 to 1. These coefficients are used to reweight the detection response element-wise, as shown in the formula below. The semantic offset is calculated using a convolutional network, which extracts the shift correction information of the two channels from deep semantic features.

[0023] Meanwhile, during the semantic correction stage, based on the semantic attention weight graph A The semantic offset and keypoint detection are performed using the following formulas respectively.D and positioning results Perform semantic correction.

[0024] in, D s and These are the results of key point correction and offset correction, respectively. This is the adjustment coefficient.

[0025] The semantic feature interaction module works as follows: The semantic feature interaction module, as a crucial component of the model's intermediate layers, aims to further enhance the expressive power of fused features. Through the extraction of high-level semantic information and the interaction between features, it achieves information complementarity across different scales and modalities, thereby improving change detection accuracy. The module employs multi-layered self-attention and cross-attention mechanisms to capture global dependencies between regions in the image and interrelationships between cross-modal features, thus reducing feature inconsistencies caused by differences in sensor performance, acquisition conditions, and radiometric response in heterogeneous images. Self-attention is used to extract global feature interactions in the image, while cross-attention is used to transfer information between matched image pairs. Both utilize multi-head attention and feedforward neural network computations, further enhancing features through residual connections and normalization. This allows the model to fully utilize fused spatial-spectral feature information, achieving more accurate feature alignment and matching.

[0026] The initial fused features are obtained based on the multi-dimensional feature matching module. The semantic feature interaction module uses sinusoidal positional encoding to add positional information to the input features, thus providing spatial priors for subsequent attention calculations and ensuring that spatial positional information is fully preserved during feature interaction. Subsequently, the module inputs the positionally encoded feature sequence into a multi-layer self-attention unit, thereby enhancing the semantic representation capability within this modality and forming a more robust global feature description. Simultaneously, the module also establishes feature connections between different modalities or temporal images directly through a cross-modal cross-attention mechanism. This allows the entire feature interaction process to not only fully explore local details but also consider global structural information, resulting in optimized fused features with higher discriminative power and robustness.

[0027] The working process of the feature co-change detection module is as follows: The feature-cooperative change detection module is primarily used for feature change identification. It utilizes feature maps obtained from the front-end network, after matching optimization and multi-dimensional feature interaction fusion, to achieve high-precision change localization and description. To reduce the overall model complexity while fully utilizing deep semantic information, this module employs an improved UNet++ network structure. Compared to the traditional UNet, UNet++ achieves effective fusion of features at different levels through nested, dense skip connections, preserving rich local detail information while providing global semantic context. To ensure sufficient extraction of deep semantic information and reduced loss, the number of downsampling layers is reduced, thereby improving the computational efficiency of change detection while preserving spatial resolution and detail information.

[0028] The module's design comprises two key components: feature fusion and cross-scale information integration, and efficient convolutional design. For feature fusion, the module utilizes densely nested skip connections to effectively fuse multi-scale information, ensuring the preservation of fine-grained features such as edges and textures in changing regions while suppressing information loss due to downsampling. Furthermore, to further enhance the model's lightweight nature and computational speed, separable convolutions replace standard convolutions. By decomposing channel convolutions and spatial convolutions, computational complexity is reduced, providing efficient and accurate support for the entire change detection process.

[0029] The loss function of the heterogeneous image change detection model based on matching optimization is as follows: To simultaneously improve the accuracy and robustness of change detection, this invention proposes a multi-scale change detection loss design with a joint loss function. This loss function not only considers the class imbalance problem but also fully utilizes the complementary information of multi-scale features. Since the number of training samples in non-change regions is much larger than that in change regions during change detection training, the model may over-focus on easily classified samples and under-weight difficult samples. To address this issue, an improved focus loss function is adopted, which makes the network training process more focused on difficult samples, thus mitigating the performance degradation caused by class imbalance. The formula is shown below: in, t Indicates the position of image pixels. p t This represents the predicted probability that the pixel at that location belongs to the change category. α t These are the corresponding weighting factors, and γ ( γ ≥ 0) is an adjustable focus parameter used to control the degree of focus of the loss between easy and difficult samples.

[0030] In deep feature fusion, especially in change detection tasks, features at different scales often represent global semantic information and local detail information, respectively. Therefore, the focus loss is calculated for multiple prediction branches with long-skip connections in the upper layers of the network, and the total change loss function is constructed in the form of a weighted sum, the expression of which is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

[0031] The embodiments of the present invention have the following technical effects: (1) To address the error accumulation problem caused by step-by-step matching in image matching of heterogeneous remote sensing images, this invention proposes a spatial-spectral collaborative matching optimization algorithm (SSSM) based on pseudo-twin networks. By utilizing the dual-branch network structure of pseudo-twin networks, spatial and spectral features of heterogeneous remote sensing images are extracted in parallel, and deep fusion is performed through multi-scale feature networks to achieve collaborative extraction and matching of deep features of cross-modal images.

[0032] (2) To address the issues of semantic information loss and feature independence in heterogeneous remote sensing images, this invention proposes a semantic enhancement matching optimization model (SACM) based on convolutional neural networks. By introducing a multi-scale semantic attention branch, the deep semantic features of heterogeneous remote sensing images are adaptively refined and contextualized to enhance the high-level semantic consistency of land cover categories, edge structures, etc. among heterogeneous images and improve the distinguishability of matching features.

[0033] (3) To address the issues of matching error propagation and limited feature interaction in change detection of heterogeneous remote sensing images, this invention proposes an end-to-end change detection model, MSCDNet, which features coordinated matching and detection. By integrating multi-dimensional feature matching and semantic feature interaction mechanisms, a multi-scale feature alignment network is constructed to improve the model's matching accuracy for different feature representations of the same ground object. Furthermore, the feature-coordinated change detection network is used to identify change features in heterogeneous remote sensing images.

[0034] Example 2: This invention also provides a heterogeneous remote sensing image deep learning change detection device, comprising: The first processing module is used to acquire remote sensing image data from different sources. The second processing module is used to obtain a heterogeneous image change detection model based on matching optimization based on heterogeneous remote sensing image data. The third processing module is used to input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve end-to-end accurate detection of heterogeneous remote sensing image change areas.

[0035] As one embodiment of the present invention, the heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature collaborative change detection module. The multi-dimensional feature matching stage utilizes a dual-branch network structure to acquire image spatial-spectral features and performs feature fusion and matching. The semantic feature interaction module uses self-attention and cross-attention mechanisms to perform deep information interaction and refinement on the fused features. The feature collaborative change detection module further integrates and refines the multi-scale fused features to achieve accurate localization and classification of changed regions.

[0036] As one embodiment of the present invention, the loss function of the heterogeneous image change detection model based on matching optimization is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

[0037] Example 3: This invention also provides a storage medium storing a computer program that executes a heterogeneous remote sensing image deep learning change detection method during runtime.

[0038] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A deep learning-based change detection method for heterogeneous remote sensing images, characterized in that, include: Step S1: Acquire heterogeneous remote sensing image data; Step S2: Based on the heterogeneous remote sensing image data, obtain a heterogeneous image change detection model based on matching optimization; Step S3: Input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve accurate end-to-end detection of heterogeneous remote sensing image change areas.

2. The method for detecting changes in heterogeneous remote sensing images using deep learning as described in claim 1, characterized in that, The heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature collaborative change detection module. The multi-dimensional feature matching stage uses a dual-branch network structure to acquire image spatial-spectral features and perform feature fusion and matching. The semantic feature interaction module uses self-attention and cross-attention mechanisms to perform in-depth information interaction and refinement on the fused features. The feature collaborative change detection module further integrates and refines the multi-scale fused features to achieve accurate localization and classification of changed regions.

3. The method for detecting changes in heterogeneous remote sensing images using deep learning as described in claim 2, characterized in that, The loss function of the heterogeneous image change detection model based on matching optimization is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

4. A deep learning-based change detection device for heterogeneous remote sensing images, characterized in that, include: The first processing module is used to acquire remote sensing image data from different sources. The second processing module is used to obtain a heterogeneous image change detection model based on matching optimization based on heterogeneous remote sensing image data. The third processing module is used to input the remote sensing image data to be processed into the heterogeneous image change detection model based on matching optimization to achieve end-to-end accurate detection of heterogeneous remote sensing image change areas.

5. The heterogeneous remote sensing image deep learning change detection device as described in claim 4, characterized in that, The heterogeneous image change detection model based on matching optimization consists of a multi-dimensional feature matching module, a semantic feature interaction module, and a feature collaborative change detection module. The multi-dimensional feature matching stage uses a dual-branch network structure to acquire image spatial-spectral features and perform feature fusion and matching. The semantic feature interaction module uses self-attention and cross-attention mechanisms to perform in-depth information interaction and refinement on the fused features. The feature collaborative change detection module further integrates and refines the multi-scale fused features to achieve accurate localization and classification of changed regions.

6. The heterogeneous remote sensing image deep learning change detection device as described in claim 5, characterized in that, The loss function of the heterogeneous image change detection model based on matching optimization is: in, w i Indicates the first i The weights corresponding to the layer prediction results L fi Indicates the first i Focus loss in layer output.

7. A storage medium, characterized in that, The storage medium stores a computer program, which executes the heterogeneous remote sensing image deep learning change detection method as described in any one of claims 1-3 when the computer program is running.

Images