Balanced global and local representation network and method for change detection in ultra-high resolution remote sensing images

Abstract

The present invention relates to the field of computer vision, and specifically to a balanced global and local representation network and method for change detection in ultra-high resolution remote sensing images. The network proposed in the present invention can retain more change features while effectively balancing global and local representations, and finally use deep semantic information to guide the generation of change maps. Specifically, the network proposed in the present invention uses a visual mamba feature encoder instead of ordinary convolutional neural network feature encoding, and uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method to process the features of the visual mamba feature encoder. It maintains more change features, while balancing global and local features, and uses deep semantic information features to suppress pseudo-change areas and enhance change areas. The present invention uses the above-mentioned network in conjunction with an ultra-high resolution remote sensing image change detection method to greatly improve the performance of ultra-high resolution remote sensing image change detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision applications, and in particular relates to a balanced global and local representation network and method for change detection in ultra-high resolution remote sensing images. Background Art

[0002] As a key tool in Earth observation, change detection (CD) compares remote sensing images of the same area taken over different time periods to identify differences. It is widely used in land use and land cover analysis, natural disaster identification, urban expansion research, and disaster assessment. In this context, obtaining detailed and accurate change detection maps is crucial.

[0003] At the same time, as the accuracy of optical remote sensing satellites continues to improve, the structural and texture features that can be captured on surface objects are becoming increasingly refined. While this increased spatial resolution provides more information, it also introduces more complex types of change, posing new challenges for change detection. Traditional methods are typically based on single-pixel change detection, such as image differencing, change vector analysis, and principal component analysis. Selecting appropriate features is crucial for these methods. However, these manually selected features struggle to adapt to high-resolution remote sensing imagery, resulting in poor detection results for traditional methods.

[0004] With the rapid development of deep learning technology, deep learning-based methods have been widely used in change detection, achieved remarkable success, and greatly improved the accuracy of CD tasks. Due to the powerful feature extraction capabilities of convolutional neural networks (CNNs), most CD-related methods are implemented based on CNNs. Researchers have proposed three fully convolutional neural network architectures that fully utilize the extracted features to interact and improve the performance of CD tasks. To further improve the performance of CD, researchers have made full use of various convolutional variants, feature pyramids, and multi-scale feature fusion methods. Although these methods can achieve good results, due to the inherent defects of the CNN model, CNN-based methods usually have difficulty capturing global context features.

[0005] In contrast, transformer-based methods use a self-attention mechanism that can effectively capture long-range dependencies in images to solve the problem of unclear segmentation boundaries. In addition, transformers can integrate information globally to better understand contextual information. However, models based on transformer architectures are sensitive to sequence length, and as the sequence length increases, the time increases quadratically. This leads to inefficiency in processing high-resolution remote sensing images. Some researchers use sparse attention mechanisms or dynamically adjust the number of queries to reduce the amount of computation and improve efficiency. Although these methods improve computational efficiency, they are not universal. Summary of the Invention

[0006] To address the low accuracy and time-consuming nature of existing change detection methods, this paper proposes a balanced global and local representation network and method for change detection in ultra-high-resolution remote sensing images. This method addresses these issues with high precision and speed when used with ultra-high-resolution remote sensing images. Furthermore, it demonstrates exceptionally high accuracy and robustness.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A balanced global and local representation network for change detection in ultra-high-resolution remote sensing images, including a visual Mamba feature encoder and a multi-scale feature balanced fusion decoder;

[0009] The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder, and the multi-scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method.

[0010] The visual mamba feature encoder uses a selection state space model instead of a common convolutional model as a feature encoder; the features of each level of the visual mamba feature encoder are represented as follows: Where i represents the features of different images, and j represents the features of different stages.

[0011] The gated fusion method in the multi-level scale feature balanced fusion decoder includes the following three steps:

[0012] Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f Represents a fixed number of channels; then a 3×3 depth convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ;

[0013] Step B: For the coarse-grained fusion feature f coarse , after two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , and obtain the fine-grained fusion feature f fine ;

[0014] Step C: For fine-grained fusion features P fine , after scaling each channel using element-by-element multiplication, the final output is ffuse .

[0015] The method for balancing global and local representations in the multi-scale feature balanced fusion decoder includes the following four steps:

[0016] Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; firstly, the global feature is modeled by the scanning mechanism of the selection state space model to obtain the global feature f global ;

[0017] Step B: In order to solve the problem of edge detection and small target detection, a local feature refinement branch is set up; the specific process is as follows: For the global feature f global First, it passes through 3×3 depth convolution, and then enters three branches. The first branch uses point-by-point convolution to reduce the channel dimension. After the activation layer, the feature is further dimensioned by point-by-point convolution to maintain the same channel dimension as the input feature. The second branch has the same process as the first branch, but uses a 3×3 convolution kernel. The third branch uses global average pooling and obtains the weight of each channel after two fully connected layers. After obtaining the features of the three branches, the features of each branch are added element by element to obtain the local features. In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then combined with Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ;

[0018] Step C: In order to coordinate the global and local feature representations, the global feature f obtained in step A is global And the local feature f obtained in step B local Scaling is performed so that it contributes differently at different layers of the decoder.

[0019] The deep semantic refinement method in the multi-level scale feature balanced fusion decoder includes the following three steps:

[0020] Step A: First, perform layer normalization on the output features Y1 and Y4 from different stages of the decoder, use a linear layer to map the features to a high-dimensional space, and use a 3×3 depthwise convolution to refine the fourth stage output feature Y4 of the decoder; where the subscripts 1 and 4 represent different stages of the decoder;

[0021] Step B: For the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, use the attention operation in the transformer to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as:

[0022]

[0023] Among them, Q shallow represents the query obtained by the decoder's fourth stage output feature Y4 after the linear layer, K deep , V deep , represents the key and value obtained by the linear layer after the output feature Y1 of the first stage of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow .

[0024] Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

[0025] A method for detecting changes in ultra-high-resolution remote sensing images is implemented using the balanced global and local representation network, comprising the following steps:

[0026] Step A: Given two images T of the same location at different time periods 1 , T 2 , the image size is 3×H×W, where 3 means the image has three RGB channels, and H and W represent the height and width of the image respectively;

[0027] Step B: T 1 , T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the area is a change area;

[0028] Step C: supervise the training of the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in the training process is L = λ1L CE +λ2L lovasz , where λ1 and λ2 represent different weight coefficients, which are set to 1 and 0.75 respectively, L CE represents the cross entropy loss function, L lovasz Represents the lovasz loss function.

[0029] Compared to existing technologies, this paper proposes a balanced global and local representation network and method for change detection in ultra-high-resolution remote sensing images. This solves the problem of traditional CD models' inability to effectively balance global and local information and alleviates the loss of semantic information during upsampling. It effectively maintains consistency within the detection area and the detection of edges and small objects. By leveraging deep semantic information, it alleviates the blurry and discontinuous edge detection issues of traditional methods. Specifically, it has the following advantages:

[0030] First: The present invention combines the advantages of the selective state space model Mamba, the convolutional neural network CNN, and the transformer architecture to greatly improve the accuracy of change detection. Traditional transformation methods have many problems when processing high-resolution remote sensing images. Simple methods cannot utilize spatial context information, cannot associate the relationship between surrounding pixels and the pixel, and cannot identify pseudo-changes caused by light and shadow changes, thus resulting in many false detections. Although the CNN-based method can effectively utilize the association of convolution with surrounding pixels to identify changes within small areas, the inductive bias of the convolution itself makes it impossible to identify long-range dependencies between images, resulting in a performance bottleneck. Although the transformer-based method can effectively solve the problem of long-range dependencies, the self-attention has a quadratic time complexity with the sequence length, which makes it time-consuming when applied to change detection methods. The present technical solution adopts the emerging selective state space model Mamba architecture, which not only has a linear growth relationship with the sequence length, but also can effectively identify long-range dependencies in the image. When applied to ultra-high-resolution remote sensing images, the accuracy and speed are greatly improved.

[0031] Second, this invention uses a gated fusion method, effectively utilizing the gating mechanism to retain more coarse-scale features, including more small objects and edge features. This not only improves the detection of small objects and edges, but also provides rich contextual information.

[0032] Third, this invention uses a balanced global and local representation method to effectively perform global and local modeling on the fused features, suppressing the false changes in the features fused by the gated fusion method and enhancing the true changes. This improves edge detection while maintaining internal consistency within the detection area.

[0033] Fourth: Different from traditional change detection models, this invention adopts a deep semantic refinement method to compensate for the loss of semantic information in the upsampling process, and uses deep semantic information to suppress pseudo-changes and enhance the changed areas.

[0034] Fifth: This paper conducts extensive experiments on three public change detection datasets: LEVIR-CD, WHU-CD, and CDD. It achieves state-of-the-art results in terms of F1 and IoU metrics, achieving 92.19% vs. 85.51%, 95.62% vs. 91.61%, and 98.36% vs. 96.78%, respectively. This represents a significant improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is an overall flow chart of the method of the present invention.

[0036] Figure 2 It is a detailed diagram of the internal network of the present invention.

[0037] Figure 3 Shows the details of the gated fusion method.

[0038] Figure 4 Detailed diagram showing the balanced global and local representation approach.

[0039] Figure 5 A detailed diagram showing the deep semantic refinement method.

[0040] Figure 6 The detection effect of the present invention is demonstrated on the LEVI-CD, WHU-CD and CDD datasets. DETAILED DESCRIPTION

[0041] The specific details of each step of the present invention will be described in detail below with reference to the drawings in the embodiments of the present invention.

[0042] The present invention provides a balanced global and local representation network for ultra-high resolution remote sensing image change detection, including a visual Mamba feature encoder and a multi-scale feature balanced fusion decoder;

[0043] The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder. The multi-scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method. The overall network structure is as follows Figure 2 shown.

[0044] A balanced global and local representation network for change detection in ultra-high-resolution remote sensing images. The visual Mamba feature encoder uses a selective state space model instead of a common convolutional model as a feature encoder. The feature representation of each level of the visual Mamba feature encoder is: Where i represents the features of different images, and j represents the features of different stages.

[0045] The gated fusion method in the multi-level scale feature balanced fusion decoder has the following process: Figure 3 As shown, it includes the following three steps:

[0046] Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f Represents a fixed number of channels; then a 3×3 depth convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ;

[0047] Step B: For the coarse-grained fusion feature f coarse , after two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , and obtain the fine-grained fusion feature f fine ;

[0048] Step C: For fine-grained fusion features f fine , after scaling each channel using element-by-element multiplication, the final output is P fuse .

[0049] The method for balancing global and local representations in the multi-scale feature balanced fusion decoder includes the following four steps:

[0050] Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; Figure 4 As shown on the right, the global feature is first modeled by the scanning mechanism of the selection state space model to obtain the global feature f global ;

[0051] Step B: In order to solve the problem of edge detection and small target detection, a local feature refinement branch is set up, such as Figure 4 As shown on the left; the specific process is as follows: For the global feature f global First, it passes through 3×3 depth convolution, and then enters three branches. The first branch uses point-by-point convolution to reduce the channel dimension. After the activation layer, the feature is further dimensioned by point-by-point convolution to maintain the same channel dimension as the input feature. The second branch has the same process as the first branch, but uses a 3×3 convolution kernel. The third branch uses global average pooling and obtains the weight of each channel after two fully connected layers. After obtaining the features of the three branches, the features of each branch are added element by element to obtain the local features. In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then combined with Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ;

[0052] Step C: In order to coordinate the global and local feature representations, the global feature f obtained in step A is global And the local feature f obtained in step B local Scaling is performed so that it contributes differently at different layers of the decoder.

[0053] The deep semantic refinement method in the multi-level scale feature balanced fusion decoder has the following overall process: Figure 5 As shown, it includes the following three steps:

[0054] Step A: First, perform layer normalization on the output features Y1 and Y4 from different stages of the decoder, use a linear layer to map the features to a high-dimensional space, and use a 3×3 depthwise convolution to refine the fourth stage output feature Y4 of the decoder; where the subscripts 1 and 4 represent different stages of the decoder;

[0055] Step B: If Figure 5 As shown on the right, for the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, the attention operation in the transformer is used to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as:

[0056]

[0057] Among them, Q shallow represents the query obtained by the decoder's fourth stage output feature Y4 after the linear layer, K deep , V deep , represents the key and value obtained by the linear layer after the output feature Y1 of the first stage of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow .

[0058] Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

[0059] A method for detecting changes in ultra-high resolution remote sensing images is implemented using the balanced global and local representation network, including the following steps. The entire processing flow is as follows: Figure 1 As shown:

[0060] Step A: Given two images T of the same location at different time periods 1 , T 2 , the image size is 3×H×W, where 3 means the image has three RGB channels, and H and W represent the height and width of the image respectively;

[0061] Step B: T 1 , T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the area is a change area;

[0062] Step C: supervise the training of the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in the training process is L = λ1L CE +λ2L lovasz , where λ1 and λ2 represent different weight coefficients, which are set to 1 and 0.75 respectively, L CE represents the cross entropy loss function, L lovasz Represents the lovaSz loss function.

[0063] Furthermore, in order to quantify the effectiveness of this method, the present invention measures the performance of the present invention through precision, recall, F1-score, and intersection-over-union (IoU). That is:

[0064]

[0065] Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of sample data that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

[0066] Comparative experiments and experimental evaluation

[0067] The quantitative results of the present invention are shown in Table 1, Table 2 and Table 3.

[0068] Table 1: Quantitative results of experiments on the LEVI R dataset

[0069]

[0070] Table 2: Quantitative results of experiments on the WHU dataset

[0071]

[0072] Table 3: Quantitative results of experiments on the CDD dataset

[0073]

[0074] The results in Tables 1, 2, and 3 show that the proposed method significantly outperforms other methods in terms of F1 and IoU. On the LEVIR dataset, it outperforms the current best model, ChangeMamba, by 0.58% and 0.92% in F1 and IoU, respectively. On the WHU dataset, it outperforms the current best method in terms of F1 and IoU by 1.43% and 2.59%. Similar improvements are also seen on the CDD dataset.

[0075] Figure 6 The comparison of some image detection results between the method proposed in this paper and the BIT method on the LEVIR, WHU and CDD datasets is shown. Figure 6 It can be seen that the area detected by the method proposed in the present invention is the most complete, the edges are continuous, and there is no false detection, while the BIT method has the problem of missed detection inside and false detection outside the area, which fully demonstrates the superior performance of the present invention.

[0076] In summary, this paper proposes a balanced global and local representation network and method for ultra-high-resolution remote sensing image change detection based on a gated fusion mechanism, a balanced global and local representation method, and a deep semantic refinement method. This significantly improves change detection performance, enhances the ability to detect changes at edges and small objects, emphasizes change regions, and suppresses spurious change regions. The effectiveness of the proposed CD model is demonstrated through extensive comparative and ablation experiments.

Claims

1. A balanced global and local representation network for change detection in ultra-high resolution remote sensing images, characterized by: Includes visual Mamba feature encoder and multi-scale feature balanced fusion decoder; The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder, and the multi-scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method; The gated fusion method in the multi-level scale feature balanced fusion decoder includes the following three steps: Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f Represents a fixed number of channels; then a 3×3 depth convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ; Step B: For the coarse-grained fusion feature f coarse , after two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , get the fine-grained fusion feature f fine ; Step C: For fine-grained fusion features f fine , after scaling each channel using element-by-element multiplication, the final output is f fuse ; The balanced global and local representation method in the multi-scale feature balanced fusion decoder includes the following four steps: Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; firstly, the global feature is modeled by the scanning mechanism of the selection state space model, and the global feature f is obtained global ; Step B: In order to solve the problems of edge detection and small target detection, a local feature refinement branch is set up; The specific process is as follows: For the global feature f global First, it passes through 3×3 depth convolution, and then enters three branches. The first branch uses point-by-point convolution to reduce the channel dimension. After the activation layer, the feature is further dimensioned by point-by-point convolution to maintain the same channel dimension as the input feature. The second branch has the same process as the first branch, but uses a 3×3 convolution kernel. The third branch uses global average pooling and obtains the weight of each channel after two fully connected layers. After obtaining the features of the three branches, the features of each branch are added element by element to obtain the local features In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then combined with Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ; Step C: In order to coordinate the global and local feature representations, the global feature f obtained in step A is global And the local feature f obtained in step B local Scaling to make different contributions at different decoder layers; The deep semantic refinement method in the multi-level scale feature balanced fusion decoder includes the following three steps: Step A: First, perform layer normalization on the output features Y1 and Y4 from different stages of the decoder, use a linear layer to map the features to a high-dimensional space, and use a 3×3 depthwise convolution to refine the fourth stage output feature Y4 of the decoder; where the subscripts 1 and 4 represent different stages of the decoder; Step B: For the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, use the attention operation in the transformer to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as: Among them, Q shallow represents the query obtained by the decoder's fourth stage output feature Y4 after the linear layer, K deep ,V deep , represents the key and value obtained by the linear layer after the output feature Y1 of the first stage of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow ; Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

2. The balanced global and local representation network for change detection in ultra-high resolution remote sensing images according to claim 1, characterized in that: The visual mamba feature encoder uses a selection state space model instead of a common convolutional model as a feature encoder; The features of each level of the visual Mamba feature encoder are represented as: Where i represents the features of different images, and j represents the features of different stages.

3. A method for detecting changes in ultra-high-resolution remote sensing images, characterized by: The method is implemented using the balanced global and local representation network according to claim 1 or 2, comprising the following steps: Step A: Given two images T of the same location at different time periods 1 ,T 2 ,The image size is 3×H×W, where 3 means the image has three RGB channels, and H and W represent the height and width of the image respectively; Step B: T 1 ,T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the position is a change area; Step C: supervise the training of the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in the training process is L = λ1L CE +λ2L lovasz , where λ1,λ2 represent different weight coefficients, L CE represents the cross entropy loss function, L lovasz Represents the lovasz loss function.

4. The method for detecting changes in ultra-high resolution remote sensing images according to claim 3, wherein: λ1 and λ2 are set to 1 and 0.75 respectively.

Images