Differential feature fusion detection methods, devices, storage media, and electronic devices
By employing multi-level feature fusion and dynamic channel weight adjustment, the problem of limited feature interaction in existing technologies is solved, improving the accuracy and reliability of change detection, especially in complex scenarios.
Patent Information
- Application Number
- CN202410791375.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-19
AI Technical Summary
Existing technologies fail to effectively utilize the interactions between features at different levels in change detection, resulting in an inability to fully leverage the complementary information and contextual relationships of multi-level features, thus affecting detection accuracy and reliability.
A multi-level feature fusion strategy is adopted, in which fine-grained local features are extracted by ResNet18 and input into the Cross-Transformer module for learning. Combined with the dense connection mechanism and the ACEM module, the channel weights are dynamically adjusted to enhance the feature transfer and interaction capabilities.
It improves the accuracy and robustness of change detection methods, especially the ability to detect changed regions in complex scenes, and enhances the utilization of global contextual information.
Smart Images

Figure CN118675024B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, storage medium, and electronic device for differential feature fusion detection. Background Technology
[0002] With the rapid development of domestic and international sensors such as WorldView, Sentinel, GF, and ZY3, massive amounts of remote sensing data are accumulating, exhibiting characteristics such as multimodal, multi-angle, and multi-resolution. Effectively mining the knowledge contained in multi-source heterogeneous remote sensing data is of great significance. Change detection technology can extract surface change information from multi-temporal images covering the same area and is widely used in natural resource surveys, urban expansion monitoring, and ecological environment monitoring. Traditional change detection methods often rely on expert experience and knowledge, requiring manual feature design, resulting in problems such as time-consuming, labor-intensive, inefficient, and low-accuracy methods. They are also easily affected by seasonal changes, lighting conditions, satellite sensor parameters, and solar altitude angle, failing to meet the needs of change detection tasks in the context of big data. Therefore, effectively mining the potential of remote sensing big data is of significant practical importance for change detection tasks in remote sensing images.
[0003] Based on the application improvement needs of the aforementioned background technologies, deep learning has demonstrated tremendous potential in the field of change detection in recent years, achieving advantages unmatched by traditional methods, thanks to its superior feature representation and nonlinear modeling capabilities. Daudt et al. first applied the UNet structure to change detection tasks and proposed three different fully convolutional change detection networks, including FC-EF, FC-Siam-conc, and FC-Siam-diff. However, convolutional networks are limited by their receptive field and cannot take into account global contextual information, resulting in poor detection performance in complex scenes. Chen et al. proposed STANet, which captures long-range contextual relationships by introducing a pyramid spatiotemporal attention module. Dong et al. designed a multi-scale context aggregation network, MCANet, which aggregates multi-scale contextual information through a scale-aware feature pyramid module and a channel-space attention module. However, with the increasing demand for long-range dependencies and global contextual information, change detection methods incorporating attention mechanisms have begun to exhibit problems of insufficient information transmission or excessive ambiguity. To further improve the performance of change detection networks, Chen et al. proposed a change detection network, BIT, combining CNN and Transformer. The CNN network extracts multi-temporal image depth features, which are then used to construct semantic tokens. A Transformer encoder learns the contextual relationships between these semantic tokens, and finally, a Transformer decoder remaps the semantic tokens back to the original feature space to enhance the original feature representation. Bandara et al. proposed a Transformer-based Siamese network, ChangeFormer. By combining a hierarchical Transformer encoder with a multilayer perceptron decoder in the Siamese network structure, it can efficiently extract multi-scale global contextual information, thereby improving the accuracy and reliability of change detection.
[0004] However, the aforementioned existing technologies all ignore the interaction between features at different levels and cannot make full use of the complementary information and contextual association of multi-level features, thus resulting in a lack of understanding of the overall features and semantics of the changing target.
[0005] Therefore, there is an urgent need for a new image detection method to further address the problems existing in current technologies. Summary of the Invention
[0006] Exemplary embodiments of this application provide a differential feature fusion detection method, apparatus, storage medium, and electronic device to at least achieve the technical effect of improving the accuracy and reliability of change detection.
[0007] This application provides an exemplary embodiment of a differential feature fusion detection method, the method comprising the following steps:
[0008] S1. Construct a change detection dataset. Crops the large-format remote sensing images and their corresponding change label maps according to a uniform scale. Then, divide the cropped T1 time-phase images, T2 time-phase images, and their corresponding change label maps into training set, validation set, and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0009] S2. Read the change detection dataset and perform data augmentation preprocessing;
[0010] S3. Construct a change detection network, including feature extraction, fusing global contextual information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic and localization differences, and outputting the changes.
[0011] S4. Input the preprocessed data from S2 into the change detection network, calculate the loss function between the network prediction and the actual change label, and iterate through backpropagation to gradually optimize the network parameters until the network converges.
[0012] S5. Read the test set constructed in S1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network to obtain the predicted probability map, and finally convert it into the change detection result.
[0013] In another aspect of this application, a differential feature fusion detection device is also proposed, the device comprising:
[0014] The change detection dataset construction module is configured to construct a change detection dataset by cropping large-format remote sensing images and corresponding change label maps according to a uniform scale, and then dividing the cropped T1 time-phase images, T2 time-phase images and corresponding change label maps into training set, validation set and test set according to a preset ratio; wherein, the training set is used for network training, the validation set is used to check the convergence of the model during the training process and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0015] The data preprocessing module is configured to read the change detection dataset and perform data augmentation preprocessing.
[0016] The change detection network construction module is configured to construct a change detection network, including feature extraction, fusing global context information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic differences and localization differences, and outputting the changes.
[0017] The network optimization module is configured to input the preprocessed data into the change detection network, calculate the loss function between the network prediction and the actual change label, and gradually optimize the network parameters through continuous backpropagation iterations until the network converges.
[0018] The result prediction module is configured to read the constructed test set, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network, obtain a prediction probability map, and finally convert it into a change detection result.
[0019] In another aspect of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps of any of the above method embodiments at runtime.
[0020] In another aspect of this application, an electronic device is also proposed, comprising a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0021] Beneficial effects:
[0022] 1. In the encoding part, a multi-level feature fusion strategy is adopted to input the fine-grained local features extracted by ResNet18 into the Cross-Transformer module to learn the differential feature representation in remote sensing images, and to perform global modeling and multi-level feature interaction to improve the network's ability to perceive changing areas in complex scenes.
[0023] 2. In the decoding part, a dense connection mechanism is introduced to further enhance the feature transmission capability. Through a series of upsampling operations, the outputs of deep and shallow layers are connected, realizing the multi-path propagation of features and the full utilization of information.
[0024] 3. The ACEM module adaptively adjusts channel weights to achieve feature representation of key information. This method enhances the interaction and information transmission capabilities between features across time phases through the Cross-Transformer module, improving the limitations of interaction between multi-level features and the weak ability to handle long-distance dependencies, thereby improving the accuracy and robustness of the change detection method. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of the process of the present invention;
[0026] Figure 2 This is a schematic diagram of the overall network structure of the present invention;
[0027] Figure 3This is a schematic diagram of the feature extraction network structure of the present invention;
[0028] Figure 4 This is a schematic diagram of the Cross-Transformer module structure of the present invention;
[0029] Figure 5 This is a schematic diagram of the adaptive channel enhancement module structure of the present invention;
[0030] Figure 6 This is a schematic diagram of some detection results of the present invention on the LEVIR-CD dataset;
[0031] Figure 7 This is a schematic diagram of an optional electronic device structure according to this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0033] In one embodiment, such as Figures 1 to 2 As shown, this invention proposes a differential feature fusion detection method, the method comprising:
[0034] S1. Construct a change detection dataset. Crops the large-format remote sensing images and their corresponding change label maps according to a uniform scale. Then, divide the cropped T1 time-phase images, T2 time-phase images, and their corresponding change label maps into training set, validation set, and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0035] S2. Read the change detection dataset and perform data augmentation preprocessing;
[0036] S3. Construct a change detection network, including feature extraction, fusing global contextual information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic and localization differences, and outputting the changes.
[0037] S4. Input the preprocessed data from S2 into the change detection network, calculate the loss function between the network prediction and the actual change label, and iterate through backpropagation to gradually optimize the network parameters until the network converges.
[0038] S5. Read the test set constructed in S1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network to obtain the predicted probability map, and finally convert it into the change detection result.
[0039] It should be noted that the preset ratio is set according to the situation. After a large number of experiments, this example uses a ratio of 7:1:2 to divide the dataset into training set, validation set and test set, which is a preferred ratio. Other ratios that fluctuate from this are also within the protection scope of this invention.
[0040] The network parameters are gradually optimized, specifically by using the binary cross-entropy loss function.
[0041] In one embodiment, the preprocessing step of constructing the change detection dataset in S1, and performing data augmentation, specifically includes:
[0042] Read the image data from the training set and the validation set, wherein only the training set data is subjected to data augmentation operations. Both the training set data and the validation set data are normalized and converted into tensor form. The data augmentation includes, but is not limited to, horizontal flipping, vertical flipping, random scale cropping, Gaussian blurring, and color transformation.
[0043] In one embodiment, the step of constructing the change detection network in S3 specifically includes:
[0044] S3.1 Construct a feature extraction network. The feature extraction network is a Siamese network structure. The two branches of the Siamese network have identical structures and share weights. The Siamese network uses a ResNet18 residual network to extract multi-level deep features of the T1 and T2 time phase images respectively. Each residual block consists of two 3×3 convolutional layers, a skip connection and a ReLU activation function. The residual connection directly adds the output of the previous layer to the input of the current layer, so that the network can directly learn the residual instead of the complete feature map.
[0045] S3.2, input the dual-temporal images f1, f2 ∈ R C×H×W Divide the image into two groups of N×N blocks of the same size. i Each image patch is flattened into a one-dimensional vector, resulting in H×W / N. 2 Each dimension is C×P 2 The image sequence is used as the input encoded representation;
[0046] The input sequence is positionally encoded so that the network can perceive the relative distance and order of different positions in the image. The dimensions of the input sequence are reshaped by layer normalization.
[0047] The input feature vector is embedded with positional encoding and fed into a multi-head cross-attention module. The input bi-temporal feature vector X is then processed through a set of linear projection layers. i Mapping to an embedded representation using a linear transformation matrix The query vector Q is calculated. i Key vector K i Sum vector V i ;
[0048] Q is obtained for each set of input feature vectors. i K i V i It interacts with the other party's key vector K, and then performs scaling and 1×1 convolution operations to model the interaction information between different features;
[0049] The result is normalized to a probability distribution using the Softmax function, cross-attention weights are generated, and multiplied by the corresponding Vi vector to obtain a weighted cross-attention representation;
[0050] A dual-branch fusion feedforward neural network is introduced to model local features by leveraging the dependency between each token and its nearest neighbor image features. The input sequence Z is divided into two subsequences Z1 and Z2, and each is transformed using feature transformation matrices W1 and W2 of two linear layers. Next, a depthwise separable convolution operation is performed on subsequence Z1, and the output subsequence Z1 is activated and multiplied with subsequence Z2. After a third linear layer feature transformation, the final output sequence Z is obtained. FFN
[0051] S3.3, for the five output features {D1, F} of the decoding part 1,i The concatenation of features X is performed on the two parallel features, i∈{1,2,3,4}}. Then, the concatenated features X are input into parallel 3×3 and 5×5 convolutional layers for operation. After that, adaptive global average pooling is performed on the two parallel features, and then the two parallel outputs are added element by element to obtain a feature tensor S of size C×1×1.
[0052] The pooled output features are passed through two 1×1 convolutional layers and a ReLU activation function. After channel dimension compression, they are remapped back to the output of size C×1×1. Then, the branch channel weights S' are obtained through the Softmax activation function.
[0053] Inspired by self-attention, a 1×1 convolution operation is performed on the concatenated feature X, and the result is multiplied by the branch channel weight S'.
[0054] Next, the final channel feature weights W are obtained through layer normalization and the Sigmoid activation function, and then multiplied with the original features to obtain the final output features.
[0055] According to another aspect of the embodiments of this application, reference is made to Figures 1-4 Furthermore, a differential feature fusion detection device is provided, the device comprising:
[0056] The change detection dataset construction module is configured to construct a change detection dataset by cropping large-format remote sensing images and corresponding change label maps according to a uniform scale, and then dividing the cropped T1 time-phase images, T2 time-phase images and corresponding change label maps into training set, validation set and test set according to a preset ratio; wherein, the training set is used for network training, the validation set is used to check the convergence of the model during the training process and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0057] The data preprocessing module is configured to read the change detection dataset and perform data augmentation preprocessing.
[0058] The change detection network construction module is configured to construct a change detection network, including feature extraction, fusing global context information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic differences and localization differences, and outputting the changes.
[0059] The network optimization module is configured to input the preprocessed data into the change detection network, calculate the loss function between the network prediction and the actual change label, and gradually optimize the network parameters through continuous backpropagation iterations until the network converges.
[0060] The result prediction module is configured to read the constructed test set, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network, obtain a prediction probability map, and finally convert it into a change detection result.
[0061] In one embodiment, reference Figures 2-4 The change detection network construction module specifically includes:
[0062] The feature extraction network module is configured as a Siamese network structure. The two branches of the Siamese network have identical structures and share weights. The Siamese network uses a ResNet18 residual network to extract multi-level deep features from the T1 and T2 time phase images respectively. Each residual block consists of two 3×3 convolutional layers, a skip connection, and a ReLU activation function. The residual connection directly adds the output of the previous layer to the input of the current layer, so that the network can directly learn the residual rather than the complete feature map.
[0063] The Cross-Transformer module, consisting of position encoding, a multi-head cross-attention module, and a feedforward neural network, is configured to process the input bi-temporal images f1, f2 ∈ R. C×H×W Divide the image into two groups of N×N blocks of the same size. i Each image patch is flattened into a one-dimensional vector, resulting in H×W / N. 2 Each dimension is C×P 2 The image sequence is used as the input encoded representation; the input sequence is positionally encoded so that the network can perceive the relative distance and order of different positions in the image, and the dimension of the input sequence is reshaped after layer normalization; the input feature vector is embedded with position encoding and input into a multi-head cross-attention module, and the input bi-temporal feature vector X is transformed through a set of linear projection layers. i Mapping to an embedded representation using a linear transformation matrix The query vector Q is calculated. i Key vector K i Sum vector V i Q is obtained for each set of input feature vectors. i K i V i The vector is interacted with the key vector K of the other party, followed by scaling and 1×1 convolution to model the interaction information between different features; the result is normalized to a probability distribution using the Softmax function to generate cross-attention weights, which are then multiplied by the corresponding V. i The vector is represented by a weighted cross-attention representation;
[0064] A dual-branch fusion feedforward neural network is introduced to model local features by leveraging the dependency between each token and its nearest neighbor image features. The input sequence Z is divided into two subsequences Z1 and Z2, and each is transformed using feature transformation matrices W1 and W2 of two linear layers. Next, a depthwise separable convolution operation is performed on subsequence Z1, and the output subsequence Z1 is activated and multiplied with subsequence Z2. After a third linear layer feature transformation, the final output sequence Z is obtained. FFN ;
[0065] An adaptive channel enhancement module is configured to enhance the five output features {D1, F} of the decoding section. 1,iThe features X are concatenated, i∈{1,2,3,4}}. Then, the concatenated features X are input into parallel 3×3 and 5×5 convolutional layers for processing. After that, adaptive global average pooling is performed on the two parallel features. Then, the two parallel outputs are added element by element to obtain a feature tensor S of size C×1×1. The pooled output features are passed through two 1×1 convolutional layers and a ReLU activation function. After channel dimension compression, they are remapped back to the output of size C×1×1. Then, the branch channel weights S' are obtained by passing the Softmax activation function.
[0066] Inspired by self-attention, a 1×1 convolution operation is performed on the concatenated feature X, and the result is multiplied by the branch channel weight S'.
[0067] Next, the final channel feature weights W are obtained through layer normalization and the Sigmoid activation function, and then multiplied with the original features to obtain the final output features.
[0068] The following is an example to illustrate this point.
[0069] Step 1: Construct a change detection dataset.
[0070] First, the collected large-scale change detection data (including T1 time-phase images, T2 time-phase images, and real change label images) were cropped into 256×256 images. The dataset was then divided into training, validation, and test sets in a 7:1:2 ratio. The training set was used for network training, the validation set was used to check the convergence of the model during training and did not participate in network training, and the test set was used to test the model and evaluate its generalization ability after training.
[0071] Step 2: Read the constructed change detection dataset and perform data augmentation.
[0072] During the network training phase, training and validation set data are read, and data augmentation operations such as horizontal flipping, vertical flipping, random scaling, Gaussian blurring, and color transformation are used to process the training set data to improve its diversity and enhance the network's generalization ability. Since the validation set, used to verify model convergence, is not involved in the training process, no data augmentation operations are performed on it. However, both the training and validation set data are normalized and converted to tensor form.
[0073] Step 3: Construct a change detection network and train the network.
[0074] The change detection network structure diagram described in this invention is as follows: Figure 2 As shown. It includes the following steps:
[0075] Step 3.1: Construct the feature extraction network. Because ResNet-18 has fewer parameters and a relatively shallow network structure, while possessing strong local feature representation capabilities and high training efficiency, it is used as the backbone network for the change detection task. Specifically, ResNet-18 employs a residual block structure to address the vanishing and exploding gradient problems in deep networks. Each residual block consists of two 3×3 convolutional layers, a skip connection, and a ReLU activation function, as shown below. Figure 3 As shown, residual connections directly add the output of the previous layer to the input of the current layer, allowing the network to learn residuals instead of complete feature maps. This makes the network easier to optimize and train, while also improving feature transfer efficiency.
[0076] Step 3.2: Construct the Cross-Transformer module to fuse global contextual information of multi-level features. By introducing cross-coding and interaction mechanisms, it better captures the correlation within and between sequences, realizes information interaction between features of dual-temporal images, and promotes information transmission and sharing between different regions.
[0077] First, the input biphase images f1, f2 ∈ R are... C×H×W Divide the image into two groups of N×N blocks of the same size. i Each image patch is flattened into a one-dimensional vector, resulting in H×W / N. 2 Each dimension is C×P 2 The image sequence is used as the input encoded representation. Next, positional encoding is performed on the input sequence so that the network can perceive the relative distances and order of different locations in the image. Then, layer normalization is performed to reshape the dimensions of the input sequence to facilitate subsequent global attention calculations across different spatial dimensions of the image. This process can be represented as:
[0078] X i =LN(Flatten(Z) i ))+E pos #(1)
[0079] In the formula, X i Represents the encoded feature vector; LN(.) represents the layer normalization operation; Flatten(.) represents the flattening operation; E pos This indicates the embedding location encoding.
[0080] The input feature vector is embedded with positional encoding and then fed into the multi-head cross-attention module, such as... Figure 4 As shown in (a), the input biphase feature vector X is first processed through a set of linear projection layers. i Mapping to an embedded representation using a linear transformation matrix The query vector Q is calculated. iKey vector K i Sum vector V i :
[0081]
[0082] Q is obtained for each set of input feature vectors. i K i V i This involves interacting with the other party's key vector K, using a dot product operation to calculate the cross-attention weights. Specifically, by... i and Multiplying them together gives us Q. i The similarity between each of the K vectors, divided by Perform a normalization operation, where d k Let V be the dimension of the K vector. Then, these similarities are scaled and subjected to 1×1 convolutions to model the interaction information between different features. The results are then normalized to a probability distribution using the Softmax function to generate cross-attention weights, which are multiplied by the corresponding V. i The vector is then represented by a weighted cross-attention representation. This process can be expressed as:
[0083]
[0084] In the formula, IN(.) represents the instance normalization operation.
[0085] Furthermore, multiple cross-attention heads are introduced, and the cross-attention mechanism is applied to different projection matrices to obtain multiple attention representations. The attention weights of each cross-attention head are concatenated and linearly projected to obtain the final output. This process can be represented as:
[0086]
[0087] MHCA(X i )=Concat(h ead1,h ead2,…,h ead n W O #(8)
[0088] In the formula, Let W represent the Q, K, and V linear transformation matrices of the i-th cross-attention head, respectively; O represents the trainable linear transformation matrix; n represents the total number of cross attention heads.
[0089] Finally, a dual-branch fusion feedforward neural network is used to learn the dependency relationship between each token and its nearest neighbor image features to achieve local feature modeling and enhance the network's ability to represent local detail information, such as... Figure 4As shown in (b). Specifically, the input sequence Z is divided into two subsequences Z1 and Z2, and feature transformation is performed on each subsequence using feature transformation matrices W1 and W2 of two linear layers. This process can be represented as:
[0090] Z′1=Z1W1#(9)
[0091] Z′2=Z2W2#(10)
[0092] Secondly, a depthwise separable convolution operation is performed on the subsequence Z1, and an activation operation is performed on the output subsequence Z1. This is then multiplied with the subsequence Z2, and the final output sequence Z is obtained after feature transformation in a third linear layer. FFN :
[0093] Z FFN =δ(DWConv(Z′1))Z′2W3#(11)
[0094] In the formula, Z'1 and Z'2 are the subsequences after linear transformation; δ represents the GELU activation function; DWConv(.) represents the depthwise separable convolution operation; and W3 is the feature transformation matrix of the third linear layer.
[0095] Step 3.3: Fusing features from sub-decoders (3.2). The adaptive channel enhancement module aims to dynamically adjust channel weights, capture feature information from the outputs of different sub-decoders, and suppress semantic and localization differences, such as... Figure 5 As shown. First, the five output features {D1, F} of the decoding part are analyzed. 1,i The features X, i∈{1,2,3,4}, are concatenated. The concatenated features X are then input into parallel 3×3 and 5×5 convolutional layers. Adaptive global average pooling is then performed on the two parallel features, and the two parallel outputs are summed element-wise to obtain a feature tensor S of size C×1×1. To reduce the loss of positional information, the pooled output features are passed through two 1×1 convolutional layers and a ReLU activation function. After channel dimension compression, they are remapped back to a C×1×1 output, and then the branch channel weights S' are obtained through a Softmax activation function. This process can be represented as:
[0096] S = GAP(Conv) 3×3 (X))+GAP(Conv 5×5 (X))#(12)
[0097]
[0098] In the formula, GAP(.) represents the adaptive global average pooling operation. This represents the ReLU activation function.
[0099] Inspired by self-attention, a 1×1 convolution operation is performed on the concatenated features X, and the result is multiplied by the branch channel weights S'. Finally, layer normalization and the sigmoid activation function are applied to obtain the final channel feature weights W, which are then multiplied by the original features to obtain the final output features. This process can be represented as:
[0100] WE = Sigmoid(LN(S′⊙Conv) 1×1 (X)))#(14)
[0101] Finally, ACEM was used to extract the most representative features from different semantic levels and then used for final change detection.
[0102] Step 4: After the network is built, data can be input into the network for training. During the training phase, the binary cross-entropy loss function is used to calculate the difference between the actual change and the predicted change.
[0103]
[0104] In the formula, N is the number of samples in the image; y i Represents a binary tag; h(y) i ) indicates that the label y belongs to the network after prediction. i The probability value.
[0105] Step 5: Read the test set constructed in Step 1, normalize and transform it, and then use the trained model to predict the change probability map. Determine the category of each pixel based on its probability value to generate the final change map.
[0106] The superiority of the method of this invention is verified below with specific experiments. This experiment uses the LEVIR-CD dataset as the experimental data. After fixed-scale cropping, the training set, validation set, and test set contain 7120, 1024, and 2048 images of size 256×256, respectively. This experiment is conducted in the PyTorch deep learning framework, using the AdamW optimizer to optimize the training process. It iterates for 100 epochs, with an initial learning rate set to 0.0001, decaying by 0.000001 in each epoch, and a batch size of 4.
[0107] To verify the superiority of the method of this invention, it was compared with two other change detection networks that also incorporate the Transformer structure, namely BIT and ChangeFormer. The accuracy evaluation metrics for the detection results were F1 score, intersection-over-union ratio (IoU), and overall accuracy (OA). The specific detection results are shown in Table 1.
[0108] Table 1. Performance Comparison of Different Change Detection Methods
[0109]
[0110] As can be seen from Table 1, compared with other methods, the method of the present invention achieves the best results in all three accuracy indicators, proving the effectiveness and superiority of the method of the present invention.
[0111] Figure 6 The image shows the detection results of different methods on the LEVIR-CD dataset. As can be seen from the image, the method of this invention achieves the best visual results, accurately detecting building boundaries and irregular structures, avoiding the adhesion of detection results for dense buildings, and effectively suppressing interference from spectral variations.
[0112] According to another aspect of the embodiments of this application, a differential feature fusion detection electronic device is also provided, which can be, but is not limited to, applied in a server. For example... Figure 7 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps of any of the above method embodiments via the computer program.
[0113] Optionally, in this embodiment, the electronic device may be located in at least one of a plurality of network devices in a computer network.
[0114] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:
[0115] S1. Construct a change detection dataset. Crops the large-format remote sensing images and their corresponding change label maps according to a uniform scale. Then, divide the cropped T1 time-phase images, T2 time-phase images, and their corresponding change label maps into training set, validation set, and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0116] S2. Read the change detection dataset and perform data augmentation preprocessing;
[0117] S3. Construct a change detection network, including feature extraction, fusing global contextual information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic and localization differences, and outputting the changes.
[0118] S4. Input the preprocessed data from S2 into the change detection network, calculate the loss function between the network prediction and the actual change label, and iterate through backpropagation to gradually optimize the network parameters until the network converges.
[0119] S5. Read the test set constructed in S1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network to obtain the predicted probability map, and finally convert it into the change detection result.
[0120] Alternatively, as those skilled in the art will understand, Figure 7 The structure shown is for illustrative purposes only; the electronic device may also be a mobile internet device (MID), a PAD, or other terminal device. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 6 The different configurations shown.
[0121] The memory 902 may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include memory remotely located relative to the processor 904, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof. As an example, such as... Figure 7 As shown, the memory 902 may include, but is not limited to, a model training module, etc., which will not be described in detail in this example.
[0122] Optionally, the transmission device 906 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 906 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 906 is a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0123] In addition, the aforementioned electronic device also includes: a display 908 for displaying the current simulation results; and a connection bus 910 for connecting the various module components in the aforementioned electronic device.
[0124] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above method embodiments when run.
[0125] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:
[0126] S1. Construct a change detection dataset. Crops the large-format remote sensing images and their corresponding change label maps according to a uniform scale. Then, divide the cropped T1 time-phase images, T2 time-phase images, and their corresponding change label maps into training set, validation set, and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training.
[0127] S2. Read the change detection dataset and perform data augmentation preprocessing;
[0128] S3. Construct a change detection network, including feature extraction, fusing global contextual information of multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic and localization differences, and outputting the changes.
[0129] S4. Input the preprocessed data from S2 into the change detection network, calculate the loss function between the network prediction and the actual change label, and iterate through backpropagation to gradually optimize the network parameters until the network converges.
[0130] S5. Read the test set constructed in S1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network to obtain the predicted probability map, and finally convert it into the change detection result.
[0131] Optionally, the storage medium is also configured to store a computer program for performing the steps included in the methods of the above embodiments, which will not be described again in this embodiment.
[0132] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0133] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0134] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0135] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0136] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.
[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0139] The above description is merely a preferred embodiment of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
[0140] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications based on these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of this application.
[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
[0142] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A differential feature fusion detection method, characterized in that, The method includes: S1. Construct a change detection dataset by cropping large-format remote sensing images and their corresponding change label maps at a uniform scale, and then... Phase image, The temporal images and their corresponding change label images are divided into training set, validation set and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training. S2. Read the change detection dataset and perform data augmentation preprocessing; S3. Construct a change detection network, including feature extraction, fusing global contextual information from multi-level features, dynamically adjusting channel weights, capturing feature information from different sub-decoder outputs, suppressing semantic and localization differences, and outputting the result; specifically including: A feature extraction network is constructed, which is a Siamese network structure. The two branches of the Siamese network have identical structures and share weights. The Siamese network uses a ResNet18 residual network to extract features. Phase and The multi-layered deep features of temporal images consist of two 3×3 convolutional layers, a skip connection, and a ReLU activation function in each residual block. The residual connection adds the output of the previous layer directly to the input of the current layer, allowing the network to learn the residual instead of the complete feature map. Input two-temporal images Divide into two groups of the same size Image Patch Each image patch is flattened into a one-dimensional vector, resulting in Each dimension is The image sequence is used as the input encoded representation; The input sequence is positionally encoded so that the network can perceive the relative distance and order of different positions in the image. The dimensions of the input sequence are reshaped by layer normalization. The input feature vector is embedded with positional encoding and fed into a multi-head cross-attention module. The input bi-temporal feature vector is then processed through a set of linear projection layers. Mapping to an embedded representation using a linear transformation matrix , , Calculate the query vector Key vector Sum value vector ; For each set of input feature vectors, the following is obtained Compare it with the other party's key vector. The interaction is performed, followed by scaling and 1×1 convolution operations to model the interaction information between different features; The result is normalized to a probability distribution using the Softmax function, cross-attention weights are generated, and then multiplied by the corresponding... The vector is represented by a weighted cross-attention representation; A dual-branch fusion feedforward neural network is introduced to model local features by leveraging the dependency between each token and its nearest neighbor image features. The input sequence is... Divided into two subsequences The feature transformation matrices of the two linear layers are respectively processed. Perform feature transformation; then process the subsequence Perform depthwise separable convolution operations and process the output subsequence. Perform activation operations and combine them with subsequences. After multiplication and feature transformation in the third linear layer, the final output sequence is obtained. ; S4. Input the preprocessed data from S2 into the change detection network, calculate the loss function between the network prediction and the actual change label, and iterate through backpropagation to gradually optimize the network parameters until the network converges. S5. Read the test set constructed in S1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network to obtain the predicted probability map, and finally convert it into the change detection result.
2. The differential feature fusion detection method according to claim 1, characterized in that, The preprocessing steps for constructing the change detection dataset and performing data augmentation in step S1 specifically include: Read the image data from the training set and the validation set, wherein only the training set data is subjected to data augmentation operations. Both the training set data and the validation set data are normalized and converted into tensor form. The data augmentation includes, but is not limited to, horizontal flipping, vertical flipping, random scale cropping, Gaussian blurring, and color transformation.
3. The differential feature fusion detection method according to claim 2, characterized in that, The specific steps of constructing the change detection network in step S3, dynamically adjusting channel weights, capturing feature information from different sub-decoders, suppressing semantic and localization differences, and outputting the result are as follows: Five output features of the decoding section Perform splicing, then combine the spliced features The input is processed by parallel 3×3 and 5×5 convolutional layers. Then, adaptive global average pooling is applied to the two parallel features. Finally, the two parallel outputs are summed element-wise to obtain a result of size [size missing]. Feature tensor ; The pooled output features are processed through two... A convolutional layer and a ReLU activation function, after channel-dimensional compression, are remapped back to a size of [size missing]. The output is then used to obtain the branch channel weights through the Softmax activation function. ; Inspired by self-attention, the spliced features conduct Convolution operation, and its result is combined with the branch channel weights. Perform dot product; Then, the final channel feature weights are obtained through layer normalization and the Sigmoid activation function. Then multiply it by the original feature to obtain the final output feature.
4. The differential feature fusion detection method according to claim 3, characterized in that, In step S4, the network parameters are optimized step by step, specifically by using the binary cross-entropy loss function to optimize the network parameters.
5. A differential feature fusion detection device, characterized in that, The device includes: The change detection dataset construction module is configured to build a change detection dataset by cropping large-format remote sensing images and their corresponding change label maps at a uniform scale, and then... Phase image, The temporal images and their corresponding change label images are divided into training set, validation set and test set according to a preset ratio. The training set is used for network training, the validation set is used to check the convergence of the model during training and does not participate in network training, and the test set is used to test the model and evaluate its generalization ability after training. The data preprocessing module is configured to read the change detection dataset and perform data augmentation preprocessing. The change detection network construction module is configured to construct a change detection network, including feature extraction, fusing global contextual information from multi-level features, dynamically adjusting channel weights, capturing feature information output by different sub-decoders, suppressing semantic and localization differences, and outputting the changes; specifically including: A feature extraction network is constructed, which is a Siamese network structure. The two branches of the Siamese network have identical structures and share weights. The Siamese network uses a ResNet18 residual network to extract features. Phase and The multi-layered deep features of temporal images consist of two 3×3 convolutional layers, a skip connection, and a ReLU activation function in each residual block. The residual connection adds the output of the previous layer directly to the input of the current layer, allowing the network to learn the residual instead of the complete feature map. Input two-temporal images Divide into two groups of the same size Image Patch Each image patch is flattened into a one-dimensional vector, resulting in Each dimension is The image sequence is used as the input encoded representation; The input sequence is positionally encoded so that the network can perceive the relative distance and order of different positions in the image. The dimensions of the input sequence are reshaped by layer normalization. The input feature vector is embedded with positional encoding and fed into a multi-head cross-attention module. The input bi-temporal feature vector is then processed through a set of linear projection layers. Mapping to an embedded representation using a linear transformation matrix , , Calculate the query vector Key vector Sum value vector ; For each set of input feature vectors, the following is obtained Compare it with the other party's key vector. The interaction is performed, followed by scaling and 1×1 convolution operations to model the interaction information between different features; The result is normalized to a probability distribution using the Softmax function, cross-attention weights are generated, and then multiplied by the corresponding... The vector is represented by a weighted cross-attention representation; A dual-branch fusion feedforward neural network is introduced to model local features by leveraging the dependency between each token and its nearest neighbor image features. The input sequence is... Divided into two subsequences The feature transformation matrices of the two linear layers are respectively processed. Perform feature transformation; then process the subsequence Perform depthwise separable convolution operations and process the output subsequence. Perform activation operations and combine them with subsequences. After multiplication and feature transformation in the third linear layer, the final output sequence is obtained. ; The network optimization module is configured to input the preprocessed data into the change detection network, calculate the loss function between the network prediction and the actual change label, and gradually optimize the network parameters through continuous backpropagation iterations until the network converges. The result prediction module is configured to read the constructed test set, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network, obtain the prediction probability map, and finally convert it into change detection results.
6. The differential feature fusion detection device according to claim 5, characterized in that, The change detection network construction module specifically includes: The feature extraction network module is configured as a Siamese network, where the two branches are identical and share weights. The Siamese network uses a ResNet18 residual network to extract features. Phase and The multi-layered deep features of the temporal image consist of two 3×3 convolutional layers, a skip connection and a ReLU activation function in each residual block. The residual connection adds the output of the previous layer directly to the input of the current layer, so that the network can learn the residual instead of the complete feature map directly. The Cross-Transformer module, consisting of position encoding, a multi-head cross-attention module, and a feedforward neural network, is configured to process the input bi-temporal images. Divide into two groups of the same size Image Patch Each image patch is flattened into a one-dimensional vector, resulting in Each dimension is The image sequence is used as the input encoded representation; the input sequence is positionally encoded so that the network can perceive the relative distance and order of different positions in the image, and the dimension of the input sequence is reshaped after layer normalization; the input feature vector is embedded with position encoding and input into a multi-head cross-attention module, and the input bi-temporal feature vector is processed through a set of linear projection layers. Mapping to an embedded representation using a linear transformation matrix , , Calculate the query vector Key vector Sum value vector For each set of input feature vectors, the following is obtained: Compare it with the other party's key vector. The interaction is performed, followed by scaling and 1×1 convolution operations to model the interaction information between different features; the result is normalized to a probability distribution using the Softmax function to generate cross-attention weights, which are then multiplied by the corresponding... The vector is represented by a weighted cross-attention representation; A dual-branch fusion feedforward neural network is introduced to model local features by leveraging the dependency between each token and its nearest neighbor image features. The input sequence is... Divided into two subsequences The feature transformation matrices of the two linear layers are respectively processed. Perform feature transformation; then process the subsequence Perform depthwise separable convolution operations and process the output subsequence. Perform activation operations and combine them with subsequences. After multiplication and feature transformation in the third linear layer, the final output sequence is obtained. ; An adaptive channel enhancement module is configured to enhance the five output features of the decoding section. Perform splicing, then combine the spliced features The input is processed by parallel 3×3 and 5×5 convolutional layers. Then, adaptive global average pooling is applied to the two parallel features. Finally, the two parallel outputs are summed element-wise to obtain a result of size [size missing]. Feature tensor The pooled output features are processed through two... A convolutional layer and a ReLU activation function, after channel-dimensional compression, are remapped back to a size of [size missing]. The output is then used to obtain the branch channel weights through the Softmax activation function. ; Inspired by self-attention, the spliced features conduct Convolution operation, and its result is combined with the branch channel weights. Perform dot product; Then, the final channel feature weights are obtained through layer normalization and the Sigmoid activation function. Then multiply it by the original feature to obtain the final output feature.
7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 4 when it is run.
8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Remote sensing image change detection method based on semantic guidance and SAM optimization
CN120451664A
