Balanced global and local representation network and method for ultrahigh resolution remote sensing image change detection

Abstract

The invention relates to the field of computer vision, in particular to a balanced global and local representation network and method for ultrahigh-resolution remote sensing image change detection, which can keep more change features, effectively balance global and local representation and finally guide the generation of a change graph by using deep semantic information. Specifically, according to the network provided by the invention, a visual Mangbar feature encoder is used for replacing common convolutional neural network feature encoding, and features of the visual Mangbar feature encoder are processed by using a gating fusion method, a balanced global and local characterization method and a deep semantic refinement method. More change features are kept, global and local features are balanced, and a pseudo change region is suppressed and a change region is enhanced by using deep semantic information features. According to the invention, the network is matched with an ultrahigh-resolution remote sensing image change detection method, so that the performance of ultrahigh-resolution remote sensing image change detection is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision applications, and in particular relates to a balanced global and local representation network and method for ultra-high resolution remote sensing image change detection. Background Art

[0002] As one of the important means of earth observation, change detection (CD) compares remote sensing images of the same area at different time periods to identify the differences between them. It is widely used in land use and land cover analysis, natural disaster identification, urban expansion research, and disaster assessment. In this context, it is crucial to obtain detailed and accurate change detection maps.

[0003] At the same time, with the continuous improvement of the accuracy of optical remote sensing satellites, the structural and texture features that can obtain the characteristics of surface objects are becoming more and more refined. The improvement of spatial resolution can bring more information, but also more complex types of changes, which poses new challenges for change detection. Traditional methods are usually based on single-pixel change detection, such as image difference method, change vector analysis and principal component analysis. For these methods, the selection of appropriate features is crucial. However, these artificially selected features are difficult to adapt to high-resolution remote sensing images, making the detection effect of traditional methods poor.

[0004] With the rapid development of deep learning technology, deep learning-based methods have been widely used in change detection, achieved remarkable success, and greatly improved the accuracy of CD tasks. Due to the powerful feature extraction capabilities of convolutional neural networks (CNNs), most CD-related methods are implemented based on CNNs. Researchers have proposed three fully convolutional neural network architectures that make full use of the extracted features to interact and improve the performance of CD tasks. In order to continue to improve the performance of CD, researchers have made full use of various convolution variants, feature pyramids, and multi-scale feature fusion methods. Although these methods can achieve good results, due to the inherent defects of the CNN model, CNN-based methods are usually difficult to capture global context features.

[0005] In contrast, the transformer-based method uses a self-attention mechanism, which can effectively capture long-distance dependencies in the image to solve the problem of unclear segmentation boundaries. In addition, the transformer can integrate information globally to better understand contextual information. However, the model based on the transformer architecture is sensitive to the sequence length, and as the sequence length increases, it will cause a quadratic increase in time. This leads to inefficiency in processing high-resolution remote sensing images. Some researchers use sparse attention mechanisms or dynamically adjust the number of queries to reduce the amount of computation and improve efficiency. Although these methods improve computational efficiency, they are not universal. Summary of the invention

[0006] In view of the problems of low detection accuracy and long time consumption of existing change detection methods, the purpose of the present invention is to propose a balanced global and local representation network and method for change detection of ultra-high resolution remote sensing images, which can solve the above problems with a high-precision and high-speed method when facing ultra-high resolution remote sensing images. At the same time, the method has extremely high accuracy and robustness.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A balanced global and local representation network for change detection in ultra-high resolution remote sensing images, including a visual Mamba feature encoder and a multi-scale feature balanced fusion decoder;

[0009] The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder, and the multi-level scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method.

[0010] The visual mamba feature encoder uses a selection state space model instead of a common convolutional model as a feature encoder; the features of each level of the visual mamba feature encoder are represented as follows: Where i represents the features of different images, and j represents the features of different stages.

[0011] The gated fusion method in the multi-level scale feature balanced fusion decoder includes the following three steps:

[0012] Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f represents a fixed number of channels; then a 3×3 deep convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ;

[0013] Step B: For the coarse-grained fusion feature f coarse , through two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , and obtain the fine-grained fusion feature f fine ;

[0014] Step C: For the fine-grained fusion feature P fine , after scaling each channel using element-by-element multiplication, the final output is ffuse .

[0015] The method for balancing global and local representations in the multi-level scale feature balanced fusion decoder includes the following four steps:

[0016] Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; firstly, the global feature is modeled by the scanning mechanism of the selection state space model to obtain the global feature f global ;

[0017] Step B: In order to solve the edge detection and small target detection problems, a local feature refinement branch is set up; the specific process is as follows: For the global feature f global First, it passes through 3×3 depth convolution, and then enters three branches. The first branch uses point-by-point convolution to reduce the channel dimension. After the activation layer, the feature is further dimensioned through point-by-point convolution to maintain the same channel dimension as the input feature. The second branch has the same process as the first branch, but uses a 3×3 convolution kernel. The third branch uses global average pooling and obtains the weight of each channel after two fully connected layers. After obtaining the features of the three branches, the features of each branch are added element by element to obtain the local features. In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ;

[0018] Step C: In order to coordinate the global and local feature representations, the global features f obtained in step A are global And the local feature f obtained in step B local Scaling is performed so that it contributes differently at different layers of the decoder.

[0019] The deep semantic refinement method in the multi-level scale feature balanced fusion decoder includes the following three steps:

[0020] Step A: First, the output features Y1 and Y4 from different stages of the decoder are layer-normalized, the features are mapped to a high-dimensional space using a linear layer, and the fourth stage output feature Y4 of the decoder is refined using a 3×3 deep convolution; where the subscripts 1 and 4 represent different stages of the decoder;

[0021] Step B: For the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, use the attention operation in the transformer to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as:

[0022]

[0023] Among them, Q shallow represents the query obtained by the decoder's fourth-stage output feature Y4 after the linear layer, K deep , V deep , represents the key and value obtained by the linear layer after the first stage output feature Y1 of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow .

[0024] Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

[0025] A method for detecting changes in ultra-high resolution remote sensing images is implemented using the balanced global and local representation network, and includes the following steps:

[0026] Step A: Given two images T of the same location at different time periods 1 , T 2 , the image size is 3×H×W, where 3 means the image has three channels, RGB, and H and W represent the height and width of the image respectively;

[0027] Step B: T 1 , T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the area is a change area;

[0028] Step C: Perform supervised training on the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in this training process is L = λ1L CE +λ2L lovasz , where λ1 and λ2 represent different weight coefficients, which are set to 1 and 0.75 respectively, L CE represents the cross entropy loss function, L lovasz Represents the lovasz loss function.

[0029] Compared with the prior art, the present invention proposes a balanced global and local representation network and method for ultra-high resolution remote sensing image change detection, which solves the problem that the traditional CD model cannot effectively balance global and local information and alleviates the lack of semantic information in the upsampling process, effectively maintaining the consistency within the detection area and the detection effect of edges and small targets. By using deep semantic information, the problem of fuzzy and discontinuous edge detection in traditional methods is alleviated. It has the following advantages:

[0030] First: The present invention combines the advantages of the selection state space model Mamba, the convolutional neural network CNN and the transformer architecture to greatly improve the accuracy of change detection. Traditional transformation methods have many problems when processing high-resolution remote sensing images. Simple methods cannot utilize spatial context information, nor can they associate the connection between surrounding pixels and the pixel, and cannot identify pseudo changes caused by light and shadow changes, thus causing many false detections. Although the CNN-based method can effectively utilize the association of convolution with surrounding pixels to identify changes within small areas, due to the inductive bias of the convolution itself, it is impossible to identify long-range dependencies between images, resulting in performance bottlenecks. Although the transformer-based method can effectively solve the problem of long-range dependencies, since self-attention has a quadratic time complexity with the sequence length, it takes time when applied to change detection methods. The technical solution adopts the emerging selection state space model Mamba architecture, which not only has a linear growth relationship with the sequence length, but also can effectively identify long-range dependencies in images. When applied to ultra-high-resolution remote sensing images, the accuracy and speed are greatly improved.

[0031] Second: The present invention adopts a gated fusion method, which effectively uses the gated mechanism to retain more coarse change features, more small targets, and edge features. It not only improves the detection effect of small targets and edges, but also provides rich context information.

[0032] Third: The present invention adopts a balanced global and local representation method to effectively perform global and local modeling on the fused features, suppressing the pseudo-change features in the features fused by the gated fusion method and strengthening the real change features. While maintaining the internal consistency of the detection area, the edge detection effect is improved.

[0033] Fourth: Different from the traditional change detection model, the present invention adopts a deep semantic refinement method to compensate for the loss of semantic information in the upsampling process, and uses deep semantic information to suppress pseudo-changes and strengthen the changed area.

[0034] Fifth: The present invention has been extensively experimented on three public change detection datasets, LEVIR-CD, WHU-CD and CDD, and has achieved the most advanced results in F1 and IoU indicators, which are 92.19% and 85.51%, 95.62% and 91.61%, and 98.36% and 96.78%, respectively. This is a huge improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is an overall flow chart of the method of the present invention.

[0036] Figure 2 It is a detailed diagram of the internal network of the present invention.

[0037] Figure 3 A detailed diagram of the gated fusion method is shown.

[0038] Figure 4 Detailed diagram showing the balanced global and local characterization approach.

[0039] Figure 5 A detailed diagram showing the deep semantic refinement approach.

[0040] Figure 6 The detection effect of the present invention is demonstrated on the LEVI-CD, WHU-CD and CDD datasets. DETAILED DESCRIPTION

[0041] The specific details of each step of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention.

[0042] The present invention discloses a balanced global and local representation network for ultra-high resolution remote sensing image change detection, including a visual Mamba feature encoder and a multi-scale feature balanced fusion decoder;

[0043] The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder. The multi-scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method. The overall network structure is as follows Figure 2 shown.

[0044] A balanced global and local representation network for ultra-high resolution remote sensing image change detection, wherein the visual Mamba feature encoder uses a selection state space model instead of a common convolutional model as a feature encoder; the features of each level of the visual Mamba feature encoder are represented as: Where i represents the features of different images, and j represents the features of different stages.

[0045] The gated fusion method in the multi-level scale feature balanced fusion decoder has the following process: Figure 3 As shown, it includes the following three steps:

[0046] Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f represents a fixed number of channels; then a 3×3 deep convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ;

[0047] Step B: For the coarse-grained fusion feature f coarse , through two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , and obtain the fine-grained fusion feature f fine ;

[0048] Step C: For fine-grained fusion features f fine , after scaling each channel using element-by-element multiplication, the final output is P fuse .

[0049] The method for balancing global and local representations in the multi-level scale feature balanced fusion decoder includes the following four steps:

[0050] Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; Figure 4 As shown on the right, the global feature is first modeled by the scanning mechanism of the selection state space model to obtain the global feature f global ;

[0051] Step B: In order to solve the edge detection and small target detection problems, a local feature refinement branch is set up, such as Figure 4 As shown on the left; the specific process is as follows: For the global feature f global First, it passes through 3×3 depth convolution, and then enters three branches. The first branch uses point-by-point convolution to reduce the channel dimension. After the activation layer, the feature is further dimensioned through point-by-point convolution to maintain the same channel dimension as the input feature. The second branch has the same process as the first branch, but uses a 3×3 convolution kernel. The third branch uses global average pooling and obtains the weight of each channel after two fully connected layers. After obtaining the features of the three branches, the features of each branch are added element by element to obtain the local features. In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ;

[0052] Step C: In order to coordinate the global and local feature representations, the global features f obtained in step A are global And the local feature f obtained in step B local Scaling is performed so that it contributes differently at different layers of the decoder.

[0053] The deep semantic refinement method in the multi-level scale feature balanced fusion decoder has the following overall process: Figure 5 As shown, it includes the following three steps:

[0054] Step A: First, the output features Y1 and Y4 from different stages of the decoder are layer-normalized, the features are mapped to a high-dimensional space using a linear layer, and the fourth stage output feature Y4 of the decoder is refined using a 3×3 deep convolution; where the subscripts 1 and 4 represent different stages of the decoder;

[0055] Step B: Figure 5 As shown on the right, for the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, the attention operation in the transformer is used to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as:

[0056]

[0057] Among them, Q shallow represents the query obtained by the decoder's fourth-stage output feature Y4 after the linear layer, K deep , V deep , represents the key and value obtained by the linear layer after the first stage output feature Y1 of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow .

[0058] Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

[0059] A method for detecting changes in ultra-high resolution remote sensing images is implemented using the balanced global and local representation network, and includes the following steps. The entire processing flow is as follows: Figure 1 As shown:

[0060] Step A: Given two images T of the same location at different time periods 1 , T 2 , the image size is 3×H×W, where 3 means the image has three channels, RGB, and H and W represent the height and width of the image respectively;

[0061] Step B: T 1 , T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the area is a change area;

[0062] Step C: Perform supervised training on the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in this training process is L = λ1L CE +λ2L lovasz , where λ1 and λ2 represent different weight coefficients, which are set to 1 and 0.75 respectively, L CE represents the cross entropy loss function, L lovasz Represents the lovaSz loss function.

[0063] Furthermore, in order to quantify the effectiveness of the method, the present invention measures the performance of the present invention through precision, recall, F1-score, and intersection-over-union (IoU). That is:

[0064]

[0065] Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

[0066] Comparative experiments and experimental evaluation

[0067] The quantitative results of the present invention are shown in Table 1, Table 2 and Table 3.

[0068] Table 1: Quantitative results of experiments on the LEVI R dataset

[0069]

[0070] Table 2: Quantitative results of experiments on the WHU dataset

[0071]

[0072] Table 3: Quantitative results of experiments on CDD dataset

[0073]

[0074] From the results in Table 1, Table 2 and Table 3, it can be seen that the method proposed in this invention is much higher than other methods in terms of F1 and IoU. For the LEVIR dataset, it is 0.58% and 0.92% higher than the current best model changemamba in terms of F1 and IoU respectively. On the WHU dataset, it is 1.43% and 2.59% higher than the current best method in terms of F1 and IoU. The same effect improvement can also be seen on the CDD dataset.

[0075] Figure 6 The comparison of some image detection results between the method proposed in the present invention and the BIT method on the LEVIR, WHU and CDD datasets is shown. Figure 6 It can be seen that the area detected by the method proposed in the present invention is the most complete, the edge is continuous, and there is no false detection, while the BIT method has the problem of internal missed detection and external false detection of the area, which fully demonstrates the superiority of the performance of the present invention.

[0076] In summary, the present invention proposes a balanced global and local representation network and method, an ultra-high resolution remote sensing image change detection network based on a gated fusion mechanism, a balanced global and local representation method, and a deep semantic refinement method. The performance of change detection is greatly improved, the change detection capability of edges and small targets is enhanced, the change area is strengthened, and the pseudo-change area is suppressed. The effectiveness of the CD model proposed in the present invention is proved through a large number of comparative experiments and ablation experiments.

Claims

1. A balanced global and local representation network for change detection in ultra-high resolution remote sensing images, characterized by: Including visual Mamba feature encoder and multi-scale feature balanced fusion decoder; The visual Mamba feature encoder replaces the ordinary convolutional neural network feature encoder, and the multi-level scale feature balanced fusion decoder uses a gated fusion method, a balanced global and local representation method, and a deep semantic refinement method.

2. The balanced global and local representation network for ultra-high resolution remote sensing image change detection according to claim 1, characterized in that: The visual Mamba feature encoder uses a selection state space model instead of a common convolutional model as a feature encoder; The features of each level of the visual Mamba feature encoder are represented as: Where i represents the features of different images, and j represents the features of different stages.

3. The balanced global and local representation network for ultra-high resolution remote sensing image change detection according to claim 1, characterized in that: The gated fusion method in the multi-level scale feature balanced fusion decoder includes the following three steps: Step A: For each level of features obtained by the visual Mamba feature encoder After stacking on the channel dimension, a 1×1 convolution is used to change the feature dimension to C f , where C f represents a fixed number of channels; then a 3×3 deep convolution is used to refine the features of each channel to obtain a coarse-grained fusion feature f coarse ; Step B: For the coarse-grained fusion feature f coarse , through two linear layer branches, the feature dimensions are changed to 3C f , one of the branches is activated by SiLU activation function, and the other branch is not activated; after multiplying the results of the two branches, a linear layer is used to change the feature dimension to C f , and obtain the fine-grained fusion feature f fine ; Step C: For fine-grained fusion features f fine , after scaling each channel using element-by-element multiplication, the final output is f fuse .

4. The balanced global and local representation network for ultra-high resolution remote sensing image change detection according to claim 1, characterized in that: The balanced global and local representation method in the multi-scale feature balanced fusion decoder includes the following four steps: Step A: Assume the input feature dimension f∈R D×M×N , where D represents the number of channels of the feature, M and N represent the height and width of the feature respectively; firstly, the global feature is modeled by the scanning mechanism of the selection state space model to obtain the global feature f global ; Step B: In order to solve the edge detection and small target detection problems, a local feature refinement branch is set; The specific process is as follows: For the global feature f global , first through 3×3 depth convolution, then enter three branches, the first branch uses point-by-point convolution to reduce the channel dimension, and after the activation layer, the feature is further dimensioned through point-by-point convolution to maintain the same channel dimension as the input feature; the second branch has the same process as the first branch, but uses a 3×3 convolution kernel; the third branch uses global average pooling, and obtains the weight of each channel after two fully connected layers; After obtaining the features of the three branches, add the features of each branch element by element to obtain the local features In order to strengthen the boundary information, 2×2 maximum pooling is used to obtain the high-frequency information of local features; the pooled result is de-pooled and then Add together and get Afterwards The weight of each position is obtained through the sigmoid activation function, and then Multiply and finally output the local feature f local ; Step C: In order to coordinate the global and local feature representations, the global features f obtained in step A are respectively global And the local feature f obtained in step B local Scaling is performed so that it contributes differently at different layers of the decoder.

5. The balanced global and local representation network for ultra-high resolution remote sensing image change detection according to claim 1, characterized in that: The deep semantic refinement method in the multi-level scale feature balanced fusion decoder includes the following three steps: Step A: First, the output features Y1 and Y4 from different stages of the decoder are layer-normalized, the features are mapped to a high-dimensional space using a linear layer, and the fourth stage output feature Y4 of the decoder is refined using a 3×3 deep convolution; where the subscripts 1 and 4 represent different stages of the decoder; Step B: For the output feature Y4 of step A and the output feature Y1 of the first stage of the decoder, use the attention operation in the transformer to strengthen the changed area and suppress the pseudo-changed area; the formula is expressed as: Among them, Q shallow represents the query obtained by the decoder's fourth-stage output feature Y4 after the linear layer, K deep , V deep , represents the key and value obtained by the linear layer after the first stage output feature Y1 of the decoder; the superscript T represents the transpose of the matrix, Softmax represents the result after the softmax operation, and dim represents the dimension of the input feature; after the above formula transformation, the result Y shallow . Step C: For the result Y obtained in step B shallow In order to fuse the features of different channels, a multi-layer perceptron MLP is used to shallow Perform weighted summation in the channel dimension and finally get the final output Y 1 .

6. A method for detecting changes in ultra-high resolution remote sensing images, characterized in that: The method is implemented by using the balanced global and local representation network as described in any one of claims 1 to 5, comprising the following steps: Step A: Given two images T of the same location at different time periods 1 , T 2 , the image size is 3×H×W, where 3 means the image has three channels, RGB, and H and W represent the height and width of the image respectively; Step B: T 1 , T 2 Input into the balanced global and local representation network to obtain the prediction result image preds. At this time, the shape of preds is H×W, and the pixel at each position indicates whether the area is a change area; Step C: Perform supervised training on the prediction results preds obtained by balancing the global and local representation networks and the labels of the training set. The loss function in this training process is L = λ1L CE +λ2L lovasz , where λ1, λ2 represent different weight coefficients, L CE represents the cross entropy loss function, L lovasz Represents the lovasz loss function.

7. The ultra-high resolution remote sensing image change detection method according to claim 1, characterized in that: λ1, λ2 are set to 1 and 0.75 respectively.

Images