Remote sensing semantic change detection method based on space-time semantic feature fusion

Abstract

The invention belongs to the technical field of remote sensing information extraction, and particularly relates to a remote sensing semantic change detection method based on space-time semantic feature fusion. Aiming at the problems of insufficient semantic extraction and inconsistent change characteristics in the existing semantic change detection, the invention provides a semantic enhancement and change consistency network, the semantic extraction capability under a complex ground feature category is improved by introducing a multi-scale adaptive module, and the high-efficiency zero-sample characteristic of SegmentAnything Model 2 (SAM2) is combined, so that the semantic change detection efficiency is improved. And a semantic alignment module is designed to enhance the consistency of change information. Finally, from the perspective of perception-analysis-extraction, the semantic change detection of the high-resolution remote sensing image is realized, the accuracy and practicability of remote sensing semantic change detection are further improved, and the method has important research value.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing information extraction, and in particular relates to a remote sensing semantic change detection method based on spatiotemporal semantic feature fusion. Background Art

[0002] Change detection is a fundamental technology in Earth observation, widely used to monitor changes in the Earth's surface caused by natural or human factors over time. Its goal is to quantitatively and qualitatively analyze the state of the same geographic area over time. Currently, common change detection methods focus on identifying the location and shape of the changed area, but lack the ability to provide specific information about the type of change. Therefore, multi-classification change detection (also known as semantic change detection), which can simultaneously identify both the location and type of change, has become a research hotspot and has demonstrated significant value in a variety of fields, including urban planning, environmental monitoring, and disaster assessment.

[0003] With the continuous advancement of remote sensing technology, change detection methods have developed into two main categories: pixel-based and object-oriented. These methods typically perform comparative analysis based on the spectral characteristics, color, texture, and geometric structure of multi-temporal remote sensing images. Pixel-based algorithms, such as image differencing, ratio methods, principal component analysis, and change vector analysis, perform well for low- and medium-resolution imagery. However, because their minimum analysis unit is a single pixel, they are susceptible to interference from the imaging environment and changes in ground object morphology. This problem is particularly pronounced when processing multi-source, heterogeneous imagery. To adapt to the development of high-resolution imagery, object-oriented methods have emerged. Representative approaches include direct object comparison, post-classification comparison, and multi-temporal segmentation. These methods can combine spectral and spatial information of the target for analysis and offer advantages in improving change detection accuracy. However, detection effectiveness often relies on the accuracy of the initial classification or segmentation.

[0004] In recent years, the introduction of deep learning has significantly advanced remote sensing change detection technology. It can mine complex, nonlinear features based on statistical patterns in multi-temporal data, and demonstrates powerful capabilities in abstract semantic understanding and complex scene recognition. Common approaches include generative adversarial networks (GANs), joint sparse representations, and spatial feature modeling based on convolutional neural networks. However, these methods generally lack the ability to adequately model the relationships between change categories, making it difficult to understand "object-to-object" changes. To address this challenge, semantic change detection methods have been proposed. Key research approaches include: first, feeding multi-scale features extracted by the encoder network into the decoder for change recognition; and second, fusing feature information at different scales to enhance the granularity and completeness of detection. These methods have significantly improved semantic modeling and extraction capabilities, laying a solid theoretical and technical foundation for subsequent research in semantic change detection.

[0005] Under current technological conditions, semantic change detection methods based on deep learning face the following challenges: 1) Existing methods fail to adequately extract fine-grained features for complex surface categories, limiting recognition accuracy. 2) During change detection, semantic inconsistencies exist between segmentation results and the final change detection results. 3) In remote sensing scenes characterized by high inter-category similarity and significant intra-category variability, SAM2 lacks the ability to model local neighborhood semantics, impacting overall detection performance.

[0006] Therefore, achieving accurate object recognition in remote sensing images is a challenging task, requiring significant improvements in robustness, accuracy, and universality. In light of this, the inventors aim to provide a remote sensing semantic change detection method based on spatiotemporal semantic feature fusion, which can improve the accuracy of semantic change detection in remote sensing images and achieve high-precision remote sensing semantic change detection. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above problems existing in traditional technologies and provide a remote sensing semantic change detection method based on spatiotemporal semantic feature fusion.

[0008] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0009] The present invention provides a remote sensing semantic change detection method based on spatiotemporal semantic feature fusion, comprising the following steps:

[0010] S1. Collect and process remote sensing semantic change detection samples;

[0011] S2. Construct a remote sensing image semantic change detection model based on twin spatiotemporal multi-feature fusion;

[0012] S3, model implementation and model algorithm training;

[0013] S4. Test and evaluate the accuracy of the trained model.

[0014] Furthermore, in step S1, a sample library is created based on remote sensing images. First, various public remote sensing image data sets are downloaded from the Internet. Then, the large amount of collected image data is sorted and damaged or poor-quality files are removed. Finally, data preprocessing is performed, which includes image cropping, scaling, and rotation operations to remove irrelevant content and adjust the image to an appropriate size to complete the construction of the sample library.

[0015] Furthermore, in step S2, in order to overcome the limitations of existing methods, a semantic enhancement and change consistency network is designed for semantic change detection; in order to enhance the network model's ability to evaluate and perceive intra-class and inter-class differences in complex surface cover categories, a multi-scale adaptive module is introduced to comprehensively extract features from different scales, while fully utilizing the zero-sample capability of SAM2.

[0016] Furthermore, in step S2, in order to solve the consistency problem between the segmentation results and the final detection results in the change detection process, a semantic alignment module is designed; through the interactive self-attention mechanism, the change information is continuously supervised, thereby realizing the multi-dimensional deep fusion and consistency enhancement of the change features, and effectively improving the information transmission and integration capabilities of the features in multi-time series remote sensing images.

[0017] Furthermore, in step S2, the model mainly includes: a feature extraction module and a multi-scale feature fusion module.

[0018] Furthermore, the feature extraction module: Aiming at the characteristics of pseudo changes caused by imaging conditions, production activities, external environmental factors, and the complexity of multi-classification tasks and multiple change detection directions in semantic change detection, a feature extraction enhancement module based on time and space at different scales is designed to effectively classify complex landforms; in the feature extraction stage, the pre-trained ResNet34 is used as the learnable backbone network, and multi-level local features are extracted from the dual-phase remote sensing image pairs based on the twin weight sharing structure; at the same time, the efficient Hiera encoder structure in SAM2 is introduced, and its parameters are kept completely frozen; the Hiera encoder extracts the features through the feature pyramid Multi-scale global features, and pre-trained on tens of millions of samples based on MAE; the coarse change detection results generated by the deep abstract features in the ResNet34 branch are used as prior guiding features and injected into the fusion features of each layer to continuously supervise the semantic feature extraction process of the encoder and guide the model to focus more on the changed areas; in order to effectively fuse the features extracted by the R34 encoder and the SAM2 encoder, a multi-scale adapter is designed to align and fuse the potential features of the two; finally, the final layer features are extracted from the R34 encoder for coarse detection, and continuous feedback guidance is provided through the coarse detection results to improve the performance of downstream change detection tasks.

[0019] Furthermore, the multi-scale feature fusion module: The deep features of the model are accurate in semantic information, but too much detail and boundary information is lost; using multi-scale semantic information, we can better grasp the deep and shallow features and prevent the loss of edge and detail information due to the network being too deep; as the network becomes deeper, the gap between shallow and deep features is often too large, and directly fusing the two will inevitably generate a lot of noise, which is not enough to effectively aggregate the feature information passed down from upstream; for this reason, a multi-scale feature selective interactive aggregation module is designed to use upstream features at different scales for feature fusion.

[0020] Furthermore, the multi-scale features extracted by the encoding network are input into the decoder for change recognition. The decoding stage first fuses the multi-scale bi-temporal features; on this basis, a set of residual modules are used to extract change information, and then a semantic alignment operation is performed; finally, in the output stage, multi-dimensional features are used for constrained alignment.

[0021] Furthermore, in step S3, the twin convolutional neural network constructed in step S2 is used as the basic framework, that is, the neural network architecture of two or more identical sub-networks is used to implement model training in the Windows system and Pytorch environment. The weight information is shared during training, which requires fewer parameters and is less prone to overfitting.

[0022] Furthermore, in step S4, the results of the model test are comprehensively and systematically evaluated, and the quantitative and qualitative results are analyzed and studied; the model is tested using the SECOND test set, which covers the following six object classes: water bodies, ground, low vegetation, trees, buildings, and sports fields; finally, mIoU, Sek, and OA are used as evaluation indicators to evaluate the results on all categories, and the average value is taken as the final accuracy.

[0023] The beneficial effects of the present invention are:

[0024] To address the issues of insufficient semantic extraction and inconsistent change features in existing semantic change detection, this paper proposes a semantic enhancement and change consistency network. By introducing a multi-scale adaptive module, it improves semantic extraction capabilities for complex feature categories. Combining the efficient zero-shot nature of SegmentAnything Model 2 (SAM2), a semantic alignment module is designed to enhance the consistency of change information. Ultimately, from a "perception-analysis-extraction" perspective, this embodiment achieves semantic change detection in high-resolution remote sensing imagery, further improving the accuracy and practicality of remote sensing semantic change detection, and possesses significant research value.

[0025] Of course, any product implementing the present invention does not necessarily need to achieve all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 Schematic diagram of the model structure of the present invention;

[0028] Figure 2 Schematic diagram of the structure of the feature extraction module in the present invention;

[0029] Figure 3 Schematic diagram of the structure of the multi-scale feature fusion module in the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] like Figure 1 As shown, this embodiment provides a remote sensing semantic change detection method based on spatiotemporal semantic feature fusion, comprising the following steps:

[0032] Step S1: Collect remote sensing image semantic change detection samples

[0033] To create a sample library based on remote sensing imagery, we first downloaded various public datasets of remote sensing images from the internet. Next, we organized the large amount of collected image data, removing damaged or low-quality files. Data preprocessing then took place, including operations like cropping, scaling, and rotating the images to remove irrelevant content and resizing them to an appropriate size, completing the construction of the sample library.

[0034] Step S1: Construction of remote sensing semantic change detection method based on spatiotemporal semantic feature fusion

[0035] To overcome the limitations of existing methods, this example proposes a novel twin convolutional network for semantic change detection. To enhance the evaluation and perception of intra- and inter-class differences, this example introduces a difference feature enhancement module. This module comprehensively captures features from the spatiotemporal dimensions. To address the interrelationships between multi-temporal and multi-level features, this example investigates a feature selection module and an interactive attention module, which achieve multi-dimensional deep fusion and interaction of change features. This enhances the ability to transfer and integrate information between features within multi-temporal remote sensing images.

[0036] The main structure and components of the network model of this embodiment are described in detail below. The model of this embodiment mainly includes: a feature extraction module and a multi-scale feature fusion module.

[0037] Feature extraction module: To address the pseudo-changes caused by factors such as imaging conditions, production activities, and the external environment, as well as the complex multi-classification tasks and multiple change detection directions in semantic change detection, this embodiment proposes a feature extraction enhancement module based on time and space at different scales, which can effectively classify complex objects.

[0038] like Figure 2 As shown in the figure, in the feature extraction stage, this embodiment uses a pre-trained ResNet34 (R34) as a learnable backbone network. Based on a twin weight-sharing structure, it extracts multi-level local features from dual-temporal remote sensing image pairs. At the same time, the efficient Hiera encoder structure from SAM2 is introduced, and its parameters are kept completely frozen. The Hiera encoder extracts multi-scale global features through a feature pyramid and is pre-trained on tens of millions of samples based on MAE (Li, 2022; Dosovitskiy, 2020). Compared with the traditional ViT architecture, Hiera removes many redundant components while strengthening its multi-scale feature extraction capabilities. This enables SAM2 to maintain its simplicity and efficiency while possessing stronger spatial reasoning capabilities, surpassing earlier SAMs. In addition, this embodiment utilizes the rough change detection results generated by the deep abstract features in the ResNet34 branch as a priori guiding features, which are injected into the fused features at each layer to continuously supervise the encoder's semantic feature extraction process and guide the model to focus more on the changed areas. To effectively fuse the features extracted by the R34 and SAM2 encoders, this embodiment designs a multi-scale adapter to align and fuse their latent features. Finally, this embodiment extracts final-layer features from the R34 encoder for coarse detection. This coarse detection result provides continuous feedback guidance to improve the performance of downstream change detection tasks. The relevant formula is as follows:

[0039]

[0040] Multi-scale feature fusion module: The deep features of the model are accurate in semantic information, but too much detail and boundary information is lost. By using multi-scale semantic information, we can better grasp the deep and shallow features and prevent the loss of edge and detail information due to the network being too deep. As the network becomes deeper, the gap between shallow and deep features is often too large. Directly fusing the two will inevitably generate a lot of noise, which is not enough to effectively aggregate the feature information passed down from upstream. Based on this, this embodiment proposes a multi-scale feature selective interactive aggregation module. Feature fusion is performed using upstream features at different scales.

[0041] like Figure 3 As shown, in the decoding stage, this embodiment first fuses multi-scale bi-temporal features. Based on this, a set of residual modules extracts change information, followed by a semantic alignment operation. Finally, in the output stage, multi-dimensional features are used for constrained alignment. The above process can be described by the following formula:

[0042]

[0043] f msf =Cat(f i ) (7)

[0044]

[0045]

[0046] Step S3: Model implementation and training

[0047] Regarding the experimental software environment configuration, all experiments in this example were conducted on the Windows operating system. The network model in this example and the models used in the comparative experiments were based on PyTorch as a deep learning framework. The deep learning environment for model training was configured using Anaconda, using Python version 3.7. The final development platform for the experiment was Pycharm. During the training and testing of the network model, dependent libraries such as Numpy and OpenCV were primarily used. To increase the model's runtime speed, GPU acceleration technology was employed, utilizing the parallel programming model CUDA and the accelerated computing library cnDNN to accelerate parallel computing on the GPU, speeding up the processing of complex image processing tasks.

[0048] In terms of experimental hardware environment configuration, all experiments in this embodiment are run on a high-performance computer equipped with two Nvidia A6000 graphics cards for parallel acceleration, a 16-core Intel Core i7-13700F CPU, 128GB of Kingston DDR4 3600MHz memory, a Kingston 2TB solid-state drive, and a Seagate 4TB mechanical hard drive.

[0049] In the model training phase, this example trains the proposed model on the SECOND dataset using the Sgdm optimizer. The initial learning rate is set to 0.02, which is exponentially decayed over the set number of epochs. The weight decay is 0.0003, and the total batch size is 8. Data augmentation techniques, including horizontal and vertical flips and random rotations, are used during training to increase the scale and diversity of the training data.

[0050] Step S4: Model testing and evaluation

[0051] This example tests the model using the SECOND test set, which covers the following six object classes: water, ground, low vegetation, trees, buildings, and sports fields. Finally, this experiment uses mIoU, Sek, and OA as evaluation metrics to evaluate the results across all classes, taking the average as the final accuracy.

[0052] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A remote sensing semantic change detection method based on spatiotemporal semantic feature fusion, characterized in that: The following steps are involved: S1. Collect and process remote sensing semantic change detection samples; S2. Construct a remote sensing image semantic change detection model based on twin spatiotemporal multi-feature fusion; S3, model implementation and model algorithm training; S4. Test and evaluate the accuracy of the trained model.

2. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 1 is characterized in that: In step S1, a sample library is created based on remote sensing images. First, various public remote sensing image datasets are downloaded from the Internet. Then, the large amount of collected image data is sorted and damaged or poor-quality files are removed. Finally, data preprocessing is performed, which includes cropping, scaling, and rotating the image to remove irrelevant content and adjust the image to an appropriate size to complete the construction of the sample library.

3. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 2 is characterized in that: In step S2, to overcome the limitations of existing methods, a semantic enhancement and change consistency network is designed for semantic change detection. To enhance the network model’s ability to assess and perceive intra-class and inter-class differences in complex land cover categories, a multi-scale adaptive module is introduced to comprehensively extract features from different scales while fully utilizing the zero-shot capability of SAM2.

4. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 3 is characterized in that: In step S2, in order to solve the consistency problem between the segmentation results and the final detection results in the change detection process, a semantic alignment module is designed; through the interactive self-attention mechanism, the change information is continuously supervised, thereby achieving multi-dimensional deep fusion and consistency enhancement of the change features, effectively improving the information transmission and integration capabilities of features in multi-time series remote sensing images.

5. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 4 is characterized in that: In step S2, the model mainly includes: a feature extraction module and a multi-scale feature fusion module.

6. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 4 is characterized in that: Feature extraction module: To address the problems of pseudo-changes caused by imaging conditions, production activities, and external environmental factors, as well as the complexity of multi-classification tasks and the multiple change detection directions in semantic change detection, a feature extraction enhancement module based on spatiotemporal features at different scales is designed to effectively classify complex objects. In the feature extraction stage, a pre-trained ResNet34 is used as a learnable backbone network, and multi-level local features are extracted from dual-temporal remote sensing image pairs based on a twin weight sharing structure. At the same time, the efficient Hiera encoder structure in SAM2 is introduced, and its parameters are kept completely frozen. The Hiera encoder extracts multi-scale global features through a feature pyramid and is pre-trained on tens of millions of samples based on MAE. The rough change detection results generated by the deep abstract features in the ResNet34 branch are used as prior guidance features and injected into the fusion features of each layer to continuously supervise the semantic feature extraction process of the encoder and guide the model to focus more on the changed areas. In order to effectively fuse the features extracted by the R34 encoder and the SAM2 encoder, a multi-scale adapter is designed to align and fuse the potential features of the two. Finally, the final layer features are extracted from the R34 encoder for coarse detection, and continuous feedback guidance is provided through the coarse detection results to improve the performance of downstream change detection tasks.

7. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 6, characterized in that: Multi-scale feature fusion module: The deep features of the model are accurate in semantic information, but they lose too much detail and boundary information. By utilizing multi-scale semantic information, we can better grasp the deep and shallow features and prevent the loss of edge and detail information due to the network being too deep. As the network becomes deeper, the gap between shallow and deep features is often too large. Directly fusing the two will inevitably generate a lot of noise, which is not enough to effectively aggregate the feature information passed down from upstream. To this end, a multi-scale feature selective interactive aggregation module is designed to use upstream features at different scales for feature fusion.

8. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 7 is characterized in that: The multi-scale features extracted by the encoding network are input into the decoder for change recognition. The decoding stage first fuses the multi-scale bi-temporal features. On this basis, a set of residual modules are used to extract change information, and then a semantic alignment operation is performed. Finally, in the output stage, multi-dimensional features are used for constrained alignment.

9. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 8, characterized in that: In step S3, the twin convolutional neural network built in step S2 is used as the basic framework, that is, a neural network architecture with two or more identical sub-networks is used to implement model training in a Windows system and Pytorch environment. Weight information is shared during training, requiring fewer parameters and less prone to overfitting.

10. The remote sensing semantic change detection method based on spatiotemporal semantic feature fusion according to claim 9, characterized in that: In step S4, the results of the model test are comprehensively and systematically evaluated, and the quantitative and qualitative results are analyzed and studied. The model is tested using the SECOND test set, which covers the following six object classes: water, ground, low vegetation, trees, buildings, and sports fields. Finally, the results are evaluated on all categories using mIoU, Sek, and OA as evaluation metrics, and the average is taken as the final accuracy.

Images