Remote sensing semantic change detection method based on multi-feature fusion

Abstract

The present application belongs to the technical field of remote sensing information extraction, and particularly relates to a remote sensing semantic change detection method based on multi-feature fusion. The present application proposes a new type of remote sensing semantic change detection twin convolution network; firstly, in order to enhance the evaluation and perception ability related to the differences between classes and classes, a difference feature enhancement module is introduced. The module comprehensively captures features from the time dimension. Then, in order to solve the mutual relationship between multi-temporal and multi-level features, the feature selection interaction module and the module are studied, realizing the multi-dimensional deep fusion and interaction of the change features. This enhances the information transmission and integration ability between the features in the multi-temporal remote sensing image. Finally, from the perspective of "perception-analysis-extraction", the semantic change detection of high-resolution remote sensing image is realized, which further improves the accuracy and practicability of remote sensing semantic change detection, and has important research value.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing information extraction technology, specifically relating to a remote sensing semantic change detection method based on multi-feature fusion. Background Technology

[0002] Change detection, as one of the most widely used fundamental technologies in the field of Earth observation, aims to quantitatively and qualitatively detect surface changes at different points in time caused by human activities and natural factors in the same geographical location. Most existing detection methods can only show the location and morphology of changes in land features, but cannot provide detailed information on the type of change. Therefore, seeking a new change detection method capable of identifying the area and type of change has become a research hotspot. This method is called multi-class change detection, or semantic change detection. This type of semantic change detection method plays a crucial role in many fields such as urban management, environmental monitoring, and damage assessment.

[0003] With the advancement of technology, change detection methods in remote sensing images have generally fallen into two main categories: pixel-based and object-oriented. These methods primarily use image characteristics such as spectrum, color, texture, and geometric morphology of multi-temporal remote sensing images as the basis for change analysis. Pixel-based methods include image differencing, image ratio analysis, principal component analysis, and change vector analysis. These methods can detect change information well in low-to-medium resolution remote sensing images, but they use only a single pixel as the smallest unit of analysis, making them susceptible to the influence of imaging conditions and the morphology of the target itself, especially in multi-source heterogeneous remote sensing images. As sensor resolution continues to improve, object-oriented methods have been proposed, mainly including direct object comparison, object comparison after classification, and multi-temporal segmentation. These methods combine the spectral and spatial morphology of the target for change detection, which can effectively improve the detection accuracy of high-resolution remote sensing images, but they are highly dependent on the performance of classification or segmentation.

[0004] In recent years, deep learning technology has been widely applied to change detection in remote sensing imagery. It can perform nonlinear feature recognition and extraction based on the statistical consistency of multi-temporal observation data, and possesses powerful capabilities in perceiving deep, abstract feature information. It can effectively handle changes in complex scenes or conditions, such as generative adversarial networks, joint sparse representation, and spatial structure extraction based on convolutional neural networks. However, existing deep learning change detection methods lack necessary support regarding inter-class relationships of change, i.e., "object-to-object" changes. To address this issue, semantic change detection methods have been proposed. Current research approaches involve, on the one hand, using the feature extraction results from the backbone network during the encoding stage as input to the change detection decoding; and on the other hand, extracting and fusing features from different scales during the encoding stage as input to the change detection decoding. These methods have improved the refinement of remote sensing semantic change detection classification and the completeness of extraction results. The proposed semantic change detection framework and methods provide reliable technical and theoretical support for subsequent research.

[0005] Under current technological conditions, deep learning-based semantic change detection methods mainly face the following challenges: 1) First, in the dual-temporal feature fusion stage, existing works mostly focus on processing the feature space dimension while neglecting the importance of temporal information. For semantic change detection tasks where some detection regions exhibit large intra-class differences but small inter-class differences, these methods struggle to decouple, extract, and effectively fuse spatial and temporal information to locate change regions and suppress false detections. 2) Shallow features in deep learning models typically contain rich detailed information, while deep features extract more abstract contextual information. Existing methods cannot effectively bridge the semantic gap between shallow and deep features, leading to useful semantic information potentially being interfered with by a large amount of redundant information, directly affecting the accuracy of semantic change detection results. Therefore, achieving accurate object identification in remote sensing images is a highly challenging task, requiring significant improvements in robustness, accuracy, and universality.

[0006] Therefore, the inventors hope to provide a remote sensing semantic change detection method based on multi-feature fusion to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to overcome the above-mentioned problems in traditional technologies and provide a remote sensing semantic change detection method based on multi-feature fusion, thereby improving the accuracy of remote sensing image semantic change detection and achieving high-precision remote sensing semantic change detection.

[0008] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0009] This invention provides a remote sensing semantic change detection method based on multi-feature fusion, comprising the following steps:

[0010] Step 1: Collect semantic change detection samples from remote sensing images;

[0011] Step 2: Construction of a remote sensing image semantic change detection model based on twin spatiotemporal multi-feature fusion;

[0012] Step 3: Model Implementation and Algorithm Training;

[0013] Step 4: Model testing and evaluation;

[0014] Furthermore, step one includes the following sub-steps:

[0015] 1) Analysis of ground feature characteristics in different time phases of remote sensing images;

[0016] 2) Analysis of changes in ground features in remote sensing images.

[0017] Furthermore, in step two, during the construction of the remote sensing image semantic change detection model, a difference feature enhancement module is introduced into the Siamese convolutional network to enhance the evaluation and perception capabilities related to intra-class and inter-class differences. This difference feature enhancement module can comprehensively capture features from the spatiotemporal dimensions. A feature selection module and an interactive attention module are designed to achieve multi-dimensional deep fusion and interaction of change features, thereby enhancing the information transmission and integration capabilities among features in multi-temporal remote sensing images.

[0018] Furthermore, in step two, in response to the characteristics of complex multi-classification tasks and multiple change detection directions in semantic change detection caused by factors such as imaging conditions, production activities, and external environment, a feature extraction enhancement module based on spatiotemporal data at different scales is designed. This module fully decouples spatial and temporal information, and then processes and fuses them separately, which can effectively classify complex ground features.

[0019] Furthermore, one branch in the feature extraction enhancement module uses a point-by-point difference method to extract the difference features between the two temporal phases, expressed by the formula:

[0020] DF = Abs(F1-F2) (1)

[0021] Where Abs represents the absolute value of the pixel difference between time phase 1 and time phase 2; in order to effectively extract the true difference features in the image, a temporal branch is added by stacking the feature maps between the two time phases to generate a five-dimensional feature, and then using a three-dimensional convolutional block to learn the correlation information between the two time phases, expressed by the formula:

[0022] Input = Cat dim＝T (F1,F1+DF / 4,F1+DF / 2,F2) (2)

[0023] TF=(Res3DBlock(Input1),Res3DBlock(Input2)) (3)

[0024] The input part expands the four-dimensional tensor features of phase 1 and phase 2 into five dimensions, Cat dim＝T This indicates that tensors are concatenated in the time dimension; since 3D convolution has higher computational complexity than 2D convolution, this invention only adds one 3×3×3 convolution to reduce the amount of computation.

[0025] Furthermore, in step two, a multi-scale feature selective interactive aggregation module is designed to perform feature fusion using upstream features at different scales.

[0026] The feature-selective interactive aggregation module is divided into detail branches and context branches, where S represents shallow features, and M and D represent deeper features. S contains richer details and edge information, while M and D contain more accurate context information. To fully integrate multi-scale feature information and filter out semantic information at different stages of the same position, multi-scale feature cross-multiplication is first performed to obtain feature 1, feature 2, and feature 3. This process is represented as:

[0027] F1=S×M (4)

[0028] F2=M×D (5)

[0029] F3=S×D (6)

[0030] Features 1, 2, and 3 are compressed into features along the channel dimension. Figure 1 ,feature Figure 2 ,feature Figure 3 The channel dimension is reduced to 1, and then the three elements are concatenated along the channel dimension and summarized into a feature fusion result map using a 1×1 convolution, which serves as the weights for the global feature information extracted from S, M, and D. Next, the Sigmoid activation function is used to control the output result, as expressed by the following formula:

[0031] σ=Sigmoid(ww) (7)

[0032] DetailBranch=(1-σ)×S (8)

[0033] ContextBranch=σ×Cat(M,D) (9)

[0034] FF=DetailBranch+ContextBranch (10)

[0035] When σ > 0.5, the model output is more inclined towards the context branch; otherwise, it is more inclined towards the detail branch. Finally, a 3×3 convolution and self-attention module are used to further explore the feature relationships in the spatial and channel dimensions.

[0036] Furthermore, in step three, the twin convolutional neural network built in step two is used as the basic framework. That is, the neural network architecture with two or more identical sub-networks is used to train the model in the Windows system and PyTorch environment. During training, weight information is shared, requiring fewer parameters and making it less prone to overfitting.

[0037] Furthermore, in step four, the results of the model test are comprehensively and systematically evaluated, and the quantitative and qualitative results are analyzed and studied.

[0038] The beneficial effects of this invention are:

[0039] This invention uses the SECOND test set to test the model, which covers the following six object classes: water, ground, low vegetation, trees, buildings, and sports fields. Finally, this experiment uses mIoU, Sek, and OA as evaluation metrics to evaluate the results across all categories, and takes the average value as the final accuracy.

[0040] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a structural diagram of the model of the present invention;

[0043] Figure 2 This is a schematic diagram of the feature extraction module structure in this invention;

[0044] Figure 3 This is a schematic diagram of the multi-scale feature fusion module structure in this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention discloses a method for detecting semantic changes in remote sensing images based on a spatiotemporal multi-feature fusion deep learning model in the field of remote sensing information extraction technology. To overcome the limitations of existing methods, the inventors propose a novel Siamese convolutional network for remote sensing semantic change detection. First, to enhance the evaluation and perception capabilities of intra-class and inter-class differences, a difference feature enhancement module is introduced. This module comprehensively captures features from the temporal dimension. Next, to address the interrelationships between multi-temporal and multi-level features, the inventors studied feature selection interaction modules and modules, achieving multi-dimensional deep fusion and interaction of changing features. This enhances the information transmission and integration capabilities among features within multi-temporal remote sensing images. Finally, from the perspective of "perception-analysis-extraction," the inventors achieved semantic change detection in high-resolution remote sensing images, further improving the accuracy and practicality of remote sensing semantic change detection, and possessing significant research value.

[0047] Specific embodiments of the present invention are as follows:

[0048] like Figures 1-3 As shown, this embodiment provides a remote sensing semantic change detection method based on multi-feature fusion, including the following steps:

[0049] Step 1: Collect semantic change detection samples from remote sensing images

[0050] This embodiment creates a sample library based on remote sensing imagery. First, various publicly available remote sensing image datasets are downloaded from the internet. Second, the large amount of collected image data is organized, removing damaged or low-quality files. Then, data preprocessing is performed, including image cropping, scaling, and rotation to remove irrelevant content and adjust the images to appropriate sizes, thus completing the construction of the sample library.

[0051] Step 2: Construction of a Semantic Change Detection Model for Remote Sensing Images Based on Twin Spatiotemporal Multi-Feature Fusion

[0052] To overcome the limitations of existing methods, a novel Siamese convolutional network for semantic change detection is proposed. To enhance evaluation and perception capabilities related to intra- and inter-class differences, a differential feature enhancement module is introduced. This module comprehensively captures features from both spatiotemporal dimensions. To address the interrelationships between multi-temporal and multi-level features, this embodiment investigates a feature selection module and an interactive attention module, which achieve multi-dimensional deep fusion and interaction of changing features. This enhances the information transfer and integration capabilities among features within multi-temporal remote sensing images.

[0053] The main structure and components of the network model in this embodiment will be described in detail below. The model in this embodiment mainly includes: a feature extraction module and a multi-scale feature fusion module.

[0054] Feature extraction module: In response to the characteristics of pseudo-changes caused by factors such as imaging conditions, production activities, and external environment, as well as the complexity of multi-classification tasks and multiple change detection directions in semantic change detection, a feature extraction enhancement module based on spatiotemporal data at different scales is proposed. This module fully decouples spatial and temporal information, processes and fuses them separately, and can effectively classify complex ground features.

[0055] like Figure 2 As shown, one branch in the feature extraction enhancement module uses a point-by-point difference method to extract the difference features between the two temporal phases, which can be expressed by the formula:

[0056] DF = Abs(F1-F2) (1)

[0057] Where Abs represents the absolute value of the pixel difference between time phase 1 and time phase 2. To effectively extract the true difference features from the image, a temporal branch is added. This involves stacking the feature maps between the two time phases to generate a five-dimensional feature, and then using a three-dimensional convolutional block to learn the correlation information between the two time phases. This can be expressed by the formula:

[0058] Input = Cat dim＝T (F1, F1+DF / 4,F1+DF / 2,F2) (2)

[0059] TF = (Res3DBlock(Input1),Res3DBlock(Input2)) (3)

[0060] It should be noted that the input part expands the four-dimensional tensor features of temporal phase 1 and temporal phase 2 into five dimensions, Cat dim＝T This indicates that tensors are concatenated in the time dimension. Since 3D convolution has higher computational complexity than 2D convolution, this embodiment only adds one 3×3×3 convolution to reduce the amount of computation.

[0061] Multi-scale Feature Fusion Module: While deep features in the model are semantically accurate, they lose too much detail and boundary information. Utilizing multi-scale semantic information can better grasp deep and shallow features, preventing the loss of edge and detail information due to excessive network depth. As the network deepens, the differences between shallow and deep features often become too large, and directly fusing them inevitably generates a lot of noise, insufficient to effectively aggregate the feature information passed down from upstream. Based on this, a multi-scale feature selective interactive aggregation module is proposed. It utilizes upstream features at different scales for feature fusion.

[0062] like Figure 3 As shown, the module is divided into detail branches and context branches, where S represents shallow features, and M and D represent deeper features. S contains richer details and edge information; while M and D contain more accurate context information. To fully integrate multi-scale feature information and filter out semantic information at different stages of the same position, multi-scale feature cross-multiplication is first performed to obtain feature 1, feature 2, and feature 3. This process can be represented as:

[0063] F1=S×M (4)

[0064] F2=M×D (5)

[0065] F3=S×D (6)

[0066] Then, features 1, 2, and 3 are compressed into features along the channel dimension. Figure 1 ,feature Figure 2 ,feature Figure 3 The channel dimension is reduced to 1, and then the three elements are concatenated along the channel dimension and summarized using a 1×1 convolution to form a feature fusion result image, which serves as the weights for the global feature information extracted from S, M, and D. Next, the Sigmoid activation function is used to control the output result, which can be expressed by the following formula:

[0067] σ=Sigmoid(ww) (7)

[0068] DetailBranch=(1-σ)×S (8)

[0069] ContextBranch=σ×Cat(M,D) (9)

[0070] FF=DetailBranch+ContextBranch (10)

[0071] When σ > 0.5, the model output is more biased towards the context branch; otherwise, it is more biased towards the detail branch. Finally, a 3×3 convolution and a self-attention module are used to further explore the feature relationships in the spatial and channel dimensions.

[0072] Step 3: Model Implementation and Training

[0073] In terms of software environment configuration, all experiments in this embodiment were conducted on a Windows operating system. The network model used in this embodiment and the models used in the comparative experiments were both based on PyTorch as the deep learning framework. The deep learning environment for model training was configured using Anaconda, with Python version 3.7 used. The final development platform for the experiments was PyCharm. During the training and testing of the network model, NumPy, OpenCV, and other dependent libraries were mainly used. Furthermore, to improve the model's running speed, GPU acceleration technology was employed. The parallel programming model CUDA and the accelerated computing library cnDNN were used to accelerate parallel computing on the GPU, speeding up the processing of complex image processing tasks.

[0074] In terms of experimental hardware configuration, all experiments in this embodiment were run on a high-performance computer equipped with two Nvidia A6000 graphics cards for parallel acceleration, a 16-core Intel Core i7-13700F CPU, 128GB of Kingston DDR4 3600MHz memory, a Kingston 2TB solid-state drive, and a Seagate 4TB hard drive.

[0075] In the model training section, this embodiment trains the proposed model on the SECOND dataset using the SGDM optimizer. The initial learning rate is set to 0.02, and the learning rate decays exponentially over the set epochs. The weight decay is 0.0003, and the total batch size is 8. Data augmentation techniques, including horizontal / vertical flipping and random rotation, are used during training to increase the scale and diversity of the training data.

[0076] Step 4: Model Testing and Evaluation

[0077] This embodiment uses the SECOND test set to test the model, which covers the following six object classes: water, ground, low vegetation, trees, buildings, and sports fields. Finally, mIoU, Sek, and OA are used as evaluation metrics to assess the results across all categories, and the average value is taken as the final accuracy.

[0078] The first step involves collecting and processing remote sensing semantic change detection samples; the second step provides a specific construction method for a remote sensing image semantic change detection model based on spatiotemporal multi-feature fusion, which is the core focus of this embodiment; the third step mainly involves the implementation of the model and the training of the model algorithm in this embodiment; and the fourth step involves testing and evaluating the accuracy of the trained model.

[0079] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A remote sensing semantic change detection method based on multi-feature fusion, characterized in that, Includes the following steps: Step 1: Collect semantic change detection samples from remote sensing images; Step 2: Construction of a remote sensing image semantic change detection model based on twin spatiotemporal multi-feature fusion; Step 3: Model Implementation and Algorithm Training; Step 4: Model testing and evaluation; In step two, during the construction of the remote sensing image semantic change detection model, a feature extraction enhancement module is introduced into the Siamese convolutional network to enhance the evaluation and perception capabilities related to intra-class and inter-class differences; this feature extraction enhancement module can comprehensively capture features from the spatiotemporal dimensions. The design incorporates a feature selection module and an interactive attention module to achieve multi-dimensional deep fusion and interaction of changing features, thereby enhancing the information transmission and integration capabilities among features within multi-temporal remote sensing images. To address the challenges of pseudo-changes caused by imaging conditions, production activities, external environment, and factors, as well as the complexity of multi-classification tasks and the multiple directions of change detection in semantic change detection, a feature extraction enhancement module based on spatiotemporal data at different scales is designed. This module fully decouples spatial and temporal information, processes and fuses them separately, and can effectively classify complex ground features. One branch of the feature extraction enhancement module uses a point-by-point difference method to extract the difference features between the two temporal phases, which can be expressed by the following formula: （1） Where Abs represents the absolute value of the pixel difference between time phase 1 and time phase 2; in order to effectively extract the true difference features in the image, a temporal branch is added by stacking the feature maps between the two time phases to generate a five-dimensional feature, and then using a three-dimensional convolutional block to learn the correlation information between the two time phases, expressed by the formula: （2） （3） The input part expands the four-dimensional tensor features of phase 1 and phase 2 into five dimensions. This indicates that tensors are cascaded along the time dimension; In step two, a multi-scale feature selective interactive aggregation module is designed to perform feature fusion using upstream features at different scales. The feature-selective interactive aggregation module is divided into detail branches and context branches, where S represents shallow features, and M and D represent deeper features. S contains richer details and edge information, while M and D contain more accurate context information. To fully integrate multi-scale feature information and filter out semantic information at different stages of the same position, multi-scale feature cross-multiplication is first performed to obtain feature 1, feature 2, and feature 3. This process is represented as: （4） （5） （6） Features 1, 2, and 3 are compressed into feature maps 1, 2, and 3 respectively along the channel dimension, reducing the channel dimension to 1. These three features are then concatenated along the channel dimension and summed using a 1×1 convolution to form the feature fusion result map, which serves as the weights for the global feature information extracted from S, M, and D. The Sigmoid activation function is then used to control the output result, as expressed in the following formula: （7） （8） （9） （10） When σ > 0.5, the model output is more inclined towards the context branch; otherwise, it is more inclined towards the detail branch. Finally, a 3×3 convolution and self-attention module are used to further explore the feature relationships in the spatial and channel dimensions.

2. The remote sensing semantic change detection method based on multi-feature fusion according to claim 1, characterized in that, Step one includes the following steps: 1) Remote sensing image data collection and preprocessing; 2) Construction of semantic change detection sample set.

3. The remote sensing semantic change detection method based on multi-feature fusion according to claim 2, characterized in that, In step three, the twin convolutional neural network built in step two is used as the basic framework. That is, the neural network architecture with two or more identical subnetworks is used to train the model in the Windows system and PyTorch environment, and the weight information is shared during training.

4. The remote sensing semantic change detection method based on multi-feature fusion according to claim 3, characterized in that, In step four, the results of the model test are comprehensively and systematically evaluated, and the quantitative and qualitative results are analyzed and studied.

Images