Remote sensing image semantic change detection method and device based on Mamba model

Abstract

The invention belongs to the technical field of image processing, and relates to a remote sensing image semantic change detection method and device based on a Mama model, and the method comprises the steps: obtaining paired dual-time remote sensing images, and carrying out the preprocessing of the dual-time remote sensing images; inputting the preprocessed dual-time remote sensing image into a Mama encoder branch for feature extraction to obtain a multi-scale feature; processing the multi-scale features by adopting a difference feature extraction module to obtain multi-scale difference features; inputting the multi-scale features and the multi-scale difference features into a feature fusion enhancement module for fusion to obtain semantic features; respectively decoding the semantic features and the multi-scale difference features by adopting a Mamba decoder branch; inputting the decoded semantic features and the multi-scale difference features into a classifier, and carrying out masking operation to obtain a semantic change graph of the dual-temporal remote sensing image; according to the method, the changed area of the dual-time-phase remote sensing image and the specific change category of the changed area can be accurately detected.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and computer vision, and in particular relates to a remote sensing image semantic change detection method and device based on a Mamba model. Background Art

[0002] Semantic change detection is an important research direction in the field of computer vision and image processing. Its main goal is to identify changes in the categories of objects in multi-temporal images. With the advancement of remote sensing data acquisition technology, especially the development of high-resolution satellite images and drone technology, ground observations have become more and more sophisticated. This makes semantic change detection have broad application prospects in land cover monitoring, urban expansion tracking, disaster response, and resource management. Traditional change detection methods focus on binary detection of changed areas and cannot provide detailed category information before and after the change. Semantic change detection not only determines which areas have changed, but also further identifies the specific categories of the changes.

[0003] In the prior art, the changed area is detected by analyzing the change vector, and then the category classification is performed. With the rapid development of deep learning technology, end-to-end deep learning models are widely used in semantic change detection tasks. This type of model avoids the error accumulation problem that may be caused by step-by-step processing in traditional methods by processing multi-phase images simultaneously. Typical deep learning architectures such as fully convolutional networks (FCN) construct a three-branch FCN architecture, in which two branches are used to extract semantic information and the other branch extracts change information. However, the main problem with convolutional neural networks such as FCN is that their limited receptive field cannot capture long-distance dependencies between pixels, resulting in inaccurate detection. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention uses a remote sensing semantic change detection model based on Mamba to process remote sensing images, that is, time-varying parameters are introduced into the state space model, and global modeling with linear complexity is realized, so that the relationship between any area and the entire image can be effectively modeled, and the problem of insufficient receptive field in convolutional neural networks can be solved. For semantic change detection tasks, in complex areas containing different objects, the interaction between different feature layers is modeled from a global perspective, which can capture the complex relationship and long-distance semantic association between any area and the entire image. Through more comprehensive contextual information, the ability to recognize subtle changes is significantly improved, thereby accurately detecting change areas and categories, and improving the accuracy and robustness of semantic change detection in complex environments.

[0005] Specifically, the present invention provides a remote sensing image semantic change detection method based on the Mamba model, the method comprising: constructing a remote sensing semantic change detection model, the model comprising a Mamba encoder branch, a difference feature extraction module, a feature fusion enhancement module, a Mamba decoder branch and a classifier; using the remote sensing semantic change detection model to detect remote sensing images comprises:

[0006] Acquire paired dual-time remote sensing images and preprocess the dual-time remote sensing images;

[0007] The preprocessed dual-time remote sensing image is input into the Mamba encoder branch for feature extraction to obtain multi-scale features;

[0008] The multi-scale features are processed using a difference feature extraction module to obtain multi-scale difference features;

[0009] The multi-scale features and multi-scale difference features are input into the feature fusion enhancement module for fusion to obtain semantic features;

[0010] The Mamba decoder branch is used to decode semantic features and multi-scale difference features respectively;

[0011] The decoded semantic features and multi-scale difference features are input into the classifier and subjected to masking operation to obtain the semantic change map of the dual-temporal remote sensing image.

[0012] To achieve the above object, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned remote sensing image semantic change detection methods based on the Mamba model.

[0013] To achieve the above-mentioned purpose, the present invention also provides a remote sensing image semantic change detection device based on the Mamba model, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the remote sensing image semantic change detection device based on the Mamba model executes any of the above-mentioned remote sensing image semantic change detection methods based on the Mamba model.

[0014] Beneficial effects of the present invention:

[0015] The present invention proposes a remote sensing image semantic change detection method based on the Mamba model, which can accurately detect the changed area of ​​a dual-phase remote sensing image and the specific change category of the changed area; the present invention uses the Mamba encoder branch to extract image features; the difference feature extraction module expands and splices the image features in multiple directions, and then generates a feature sequence after processing using a state space model, which can more accurately identify the difference between the dual-phase images; the feature fusion enhancement module strengthens the category representation of the changed area, suppresses irrelevant background noise and subtle unchanged areas, and ensures the accuracy of the change feature in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the remote sensing image semantic change detection method based on the Mamba model of the present invention;

[0017] Figure 2 It is a structural diagram of the remote sensing semantic change detection model of the present invention;

[0018] Figure 3 It is a schematic diagram of the VSS block structure of the present invention;

[0019] Figure 4 It is a schematic diagram of the SS2D structure of the present invention;

[0020] Figure 5 This is a schematic diagram of the structure of the difference feature extraction module of the present invention;

[0021] Figure 6 It is a structural schematic diagram of the feature fusion enhancement module of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The invention provides a remote sensing image semantic change detection method based on a Mamba model. The method comprises: constructing a remote sensing semantic change detection model, wherein the model comprises a Mamba encoder branch, a difference feature extraction module, a feature fusion enhancement module, a Mamba decoder branch and a classifier; using the remote sensing semantic change detection model to detect remote sensing images comprises: acquiring paired dual-time remote sensing images, and preprocessing the dual-time remote sensing images; inputting the preprocessed dual-time remote sensing images into a Mamba encoder branch for feature extraction to obtain multi-scale features; using the difference feature extraction module to process the multi-scale features to obtain multi-scale difference features; inputting the multi-scale features and the multi-scale difference features into a feature fusion enhancement module for fusion to obtain semantic features; using a Mamba decoder branch to respectively decode the semantic features and the multi-scale difference features; inputting the decoded semantic features and the multi-scale difference features into a classifier and performing a masking operation to obtain a semantic change map of the dual-time phase remote sensing image.

[0024] The above remote sensing semantic change detection model is a trained remote sensing semantic change detection model.

[0025] The SECOND dataset is used when training the remote sensing semantic change detection model; the SECOND dataset collects 4662 pairs of aerial images with a size of 512×512 and a resolution of 0.5-3m / pixel; the dataset focuses on six major land cover categories: non-vegetated surface, trees, low vegetation, water bodies, buildings, and playgrounds. The dataset uses land cover map pairs and non-change masks to represent changes and distinguish between changed pixels and unchanged pixels in the same category.

[0026] A specific implementation method of a remote sensing image semantic change detection method based on the Mamba model is as follows: Figure 1 As shown, the method includes:

[0027] S1: Data preprocessing, performing preprocessing operations on the paired dual-temporal remote sensing images I1 and I2, including normalization, random inversion, etc.;

[0028] S2: Construct the Mamba encoder branch, difference feature extraction module, feature fusion enhancement module, Mamba decoder branch and classifier;

[0029] S3: Use the Mamba encoder to extract multi-scale image features of the preprocessed dual-temporal remote sensing images.

[0030] S4: The extracted multi-scale image features are input into the difference feature extraction module to extract the multi-scale difference features. The multi-scale image features and the multi-scale difference features are input into the feature fusion enhancement module to enhance the image features using the difference features to obtain the semantic features.

[0031] S5: The difference features are input into the Mamba decoder for decoding, and then the binary classifier is used to generate a binary change map. The semantic features are sent into the Mamba decoder for decoding, and then the semantic classifier is used to generate a semantic category map.

[0032] S6: Masking operation is performed on the binary change map and the semantic map, and finally the semantic change map of the dual-temporal remote sensing image is obtained.

[0033] In this embodiment, the image is preprocessed by simultaneously preprocessing the paired dual-phase remote sensing images I1 and I2; specifically, the images of the dual-phase high-resolution remote sensing image training set are subjected to data enhancement by random flipping, random blurring, random color adjustment, and normalization.

[0034] In this embodiment, the Mamba encoder branch adopts the VMamba backbone network. The Mamba encoder branch contains 5 stages. The first is the Patch Embedding operation to increase the image dimension to 96. The number of VSS blocks in the remaining four stages is [2, 2, 9, 2]. The difference feature extraction module expands and splices the feature maps of the two images in multiple directions, and then uses the state space model to generate a change feature sequence, which is then divided into two sequences and a weight matrix is ​​obtained by Sigmoid. The weight matrix is ​​multiplied and subtracted from the two image features to obtain the difference feature. The feature fusion enhancement module generates semantic features by fusing the difference features with the image features and weighting the image features in combination with the attention mechanism. Each block of the Mamba decoder is composed of VSS blocks, and then an upsampling module is used to double the size of the input feature.

[0035] Specifically, the two Mamba encoder branches are weight-shared, where the Mamba encoder branch uses the VMamba backbone network, which includes five stages. The first stage is Patch Embedding, and the second to fifth stages are Figure 2 The specific structure diagram of the VSS block is as follows: Figure 3 As shown in FIG. 1 , it includes a linear layer, a 3×3 depth-separable convolution, a SS2D, and a linear layer, where the SS2D that completes the feature extraction is shown in FIG. Figure 4 As shown, SS2D includes selective scanning and state space models. The second to fourth stages also include downsampling operations, which increase the channel dimension by two times and reduce the size by half. The image features of each stage will be used as a difference extraction module and a feature fusion enhancement module.

[0036] The difference feature extraction module is as follows Figure 5As shown in the figure, it consists of a selective scanning operation and an SSM module. First, the difference feature extraction module performs a scanning operation on the image features, and then the sequence splicing is input into the SSM module, and the corresponding addition and separation are performed to obtain the feature map. The corresponding attention weight is generated by the Sigmoid function, and then the attention weight is fused with the image features, and finally the results are subtracted to obtain the difference feature.

[0037] Feature fusion enhancement module such as Figure 6 As shown in the figure, it consists of a 3×3 convolution layer and a 1×1 convolution layer to form a convolution operation. Each convolution layer is followed by Batchnorm and ReLU, which are used to generate the fully connected layer 1 for scaling parameters and the fully connected layer 2 for offset parameters. First, the difference features are concatenated with the image features, and the features are extracted through the convolution operation. Then, the scaling parameters and offset parameters are obtained through the fully connected layer. Finally, the scaling parameters and offset parameters are fused with the image features to generate semantic features.

[0038] The Mamba decoder branches are two semantic Mamba decoders, which share weights, and one binary Mamba decoder. The semantic Mamba decoder consists of four stages, each of which is composed of VSS blocks. All decoders use bilinear interpolation for upsampling in each stage.

[0039] The classifiers are semantic classifier and binary classifier, both of which are composed of convolution layers with a convolution kernel of 1×1.

[0040] In this embodiment, the use of the Mamba encoder to extract multi-scale features includes: extracting features of the dual-phase images I1 and I2. The dual-phase image of size 512×512 is directly passed into the VMamba backbone network loaded with the pre-trained model. After Patch Emebedding in the Vmamba backbone network, the image dimension becomes 96, and the size becomes 1 / 4 of the original size, and then passes through 2, 2, 9, and 2 VSS blocks in sequence; four different directional feature sequences are obtained by selective scanning, and then the state space model processes the feature sequence by a method of globally processing context information, and finally the four different directional sequences are added, and the image features are obtained after passing through the linear layer. In the Mamba encoder branch, a total of four different scale feature maps are extracted, and their channel dimensions are 96, 192, 384, and 768, respectively.

[0041] In this embodiment, each VSS block includes a linear layer, a 3×3 depthwise separable convolution, an SS2D module, a linear layer, where the SS2D includes a 2D selective scan operation and a state space model (SSM) that relies on the intermediate implicit state h(t)∈R NMapping the one-dimensional input function x(t)∈R to the output y(t)∈R, the specific formula is:

[0042] h′(t)=Ah(t)+Bx(t)

[0043] y(t)=Ch(t)

[0044] where A∈R N×N represents the state matrix, B∈R N×1 , C∈R N×1 Represents the projection parameter. SS2D discretizes this continuous system, introduces the time scale parameter Δ, and uses a fixed discretization rule to convert A and B into discrete parameters and Usually, zero-order hold is used as the discretization rule, which can be defined as follows:

[0045]

[0046] After discretization, SSM can be linearly recursively expressed as:

[0047]

[0048] y(t)=Ch(t)

[0049] It can also be expressed as a global convolution:

[0050]

[0051] in Represents a structured convolution kernel, L represents the length of the input sequence x, and the first three units except the first unit also include a downsampling with a coefficient of 2, so that the output feature map is 1 / 2 of the input.

[0052] In this embodiment, four different scales of image features are input into the difference feature extraction module to obtain four different scales of difference features. The four different scales of difference features are used to perform feature fusion enhancement with the image features to obtain semantic features, which specifically includes the following steps:

[0053] Step 1: First, scan the image features to obtain a pair of feature sequences containing four directions. Then, splice the feature sequences in the same scanning direction together and input them into the SSM module, and add them up to obtain the change features. The change features are divided into two feature maps, and then Sigmoid is used to obtain the weight matrix, which is multiplied with the image features, and then the results are subtracted to obtain the difference features.

[0054] Step 2: Concatenate the difference features with the image features, and then perform a convolution operation. The number of channels is halved through a 3×3 convolution layer, and the features are further extracted through a 1×1 convolution layer. The difference features are input into the fully connected layer 1 and the fully connected layer 2 to obtain the scaling parameter and offset parameter. The scaling parameter is multiplied with the feature after the Sigmoid activation function, and then added to the offset parameter to generate the semantic feature.

[0055] The difference features of the four scales are input into the binary Mamba decoder for decoding, and the semantic features of the four scales are input into the semantic Mamba decoder for decoding. The binary Mamba decoder and the semantic Mamba decoder are Unet architectures, which are both composed of four blocks. The difference feature map and the semantic feature map are obtained through the VSS block and the upsampling module.

[0056] The difference feature map is subjected to a classifier to generate a binary change map, and the semantic feature map is subjected to a classifier to generate a final semantic category map. Specifically, for the generated difference feature map, a classifier is constructed to perform binary classification in the channel dimension to generate the final binary change map; for the semantic feature map, a classifier is constructed to perform multi-classification in the channel dimension to generate a semantic category map. The classifier is specifically implemented as follows: a 2D convolution is used to transform the channel dimension to generate the final binary change map and semantic category map. The obtained binary change map and I1, I2 semantic category maps are upsampled to the original size. The binary change map is masked with the I1, I2 semantic category maps, and finally the semantic change map of the dual-phase remote sensing image is obtained.

[0057] A remote sensing image semantic change detection system based on the Mamba model, the system comprising: a preprocessing module, a Mamba encoder branch, a difference feature extraction module, a feature fusion enhancement module, a Mamba decoder branch and a classifier;

[0058] The preprocessing module is used to perform image enhancement on the input dual-temporal remote sensing image;

[0059] The Mamba encoder branch is used to extract features of dual-phase remote sensing images;

[0060] The difference feature extraction module is used to extract difference features from image features;

[0061] The feature fusion enhancement module is used to perform feature fusion using difference features and image features;

[0062] The Mamba decoder branch is used to decode difference features and semantic features;

[0063] The classifier is used to obtain a binary change map from a difference feature map, obtain a semantic category map from a semantic feature map, judge whether a dual-temporal remote sensing image has changed according to the binary change map, and judge the semantic category of the dual-temporal remote sensing image according to the semantic category map.

[0064] The system implementation of the present invention is the same as the specific implementation of the method.

[0065] In one embodiment of the present invention, the present invention further includes a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the program implements any of the above-mentioned remote sensing image semantic change detection methods based on the Mamba model.

[0066] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0067] A remote sensing image semantic change detection device based on the Mamba model comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the remote sensing image semantic change detection device based on the Mamba model executes any of the remote sensing image semantic change detection methods based on the Mamba model.

[0068] Specifically, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0069] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0070] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image semantic change detection method based on the Mamba model, characterized in that: include: Construct a remote sensing semantic change detection model, which includes a Mamba encoder branch, a difference feature extraction module, a feature fusion enhancement module, a Mamba decoder branch, and a classifier; The remote sensing semantic change detection model is used to detect remote sensing images, including: Acquire paired dual-time remote sensing images and preprocess the dual-time remote sensing images; The preprocessed dual-time remote sensing image is input into the Mamba encoder branch for feature extraction to obtain multi-scale features; The multi-scale features are processed using a difference feature extraction module to obtain multi-scale difference features; The multi-scale features and multi-scale difference features are input into the feature fusion enhancement module for fusion to obtain semantic features; The Mamba decoder branch is used to decode semantic features and multi-scale difference features respectively; The decoded semantic features and multi-scale difference features are input into the classifier and subjected to masking operation to obtain the semantic change map of the dual-temporal remote sensing image.

2. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The preprocessing of the dual-time remote sensing image includes randomly flipping, randomly blurring and randomly adjusting the color of the remote sensing image to obtain an enhanced dual-time remote sensing image.

3. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The Mamba encoder branch consists of five cascade units, of which the first cascade unit is Patch Embedding, which consists of a 3×3 convolutional layer; the second to fifth cascade units include 2, 2, 9, and 2 VSS blocks respectively; each VSS block consists of a linear layer, a 3×3 depth-separable convolution, an SS2D module, and a linear layer; the SS2D module includes a 2D selective scanning operation and a state space model.

4. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The difference feature extraction module is used to process multi-scale features, including: passing the image features of the dual-phase remote sensing image through a linear layer and a 3×3 depth-separable convolution, and expanding the convolution results along four different directions; concatenating the two sequences expanded along the same direction and inputting them into the SSM module to generate a feature sequence; summing the feature sequences and outputting the features through a linear layer; separating the output features into two feature maps, and obtaining two attention weights after a Sigmoid activation function; using the two attention weights to perform dot product with the features of the dual-phase remote sensing image; subtracting the elements of the corresponding dot product results and taking the absolute value to obtain the difference features.

5. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The feature fusion enhancement module consists of a 3×3 convolutional layer, a 1×1 convolutional layer, two fully connected layers, and an activation function ReLU.

6. The remote sensing image semantic change detection method based on the Mamba model according to claim 5, characterized in that: The feature fusion enhancement module fuses multi-scale features and multi-scale difference features, including: splicing multi-scale features and multi-scale difference features; halving the number of channels of the spliced ​​features through a 3×3 convolution layer, and further extracting features through a 1×1 convolution layer; inputting the further feature-extracted difference features into the fully connected layer 1 to obtain scaling parameters, and inputting the difference features into the fully connected layer 2 to obtain offset parameters; the scaling parameters are multiplied by the image features after passing through the Sigmoid activation function, and then added to the offset parameters to generate semantic features.

7. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The Mamba decoder branch consists of four cascaded units, each unit includes 1 VSS block, and each VSS block includes a linear layer, a 3×3 depth-wise separable convolution, a SS2D module, and a linear layer.

8. The remote sensing image semantic change detection method based on the Mamba model according to claim 1, characterized in that: The classifier consists of a binary classifier and a semantic classifier, both of which are composed of a convolution layer with a convolution kernel of 1×1; the binary classifier performs binary classification processing on the decoded difference features to obtain a binary change map; the semantic classifier classifies the decoded semantic features to obtain a semantic category map of the dual-temporal remote sensing image; and a masking operation is performed on the binary change map and the semantic category map of the dual-temporal remote sensing image to obtain the semantic change map of the dual-temporal remote sensing image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the remote sensing image semantic change detection method based on the Mamba model in any one of claims 1 to 8.

10. A remote sensing image semantic change detection device based on the Mamba model, characterized in that: It comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the remote sensing image semantic change detection device based on the Mamba model executes any one of the remote sensing image semantic change detection methods based on the Mamba model in claims 1 to 8.

Images