Remote sensing image change detection method and device based on feature enhancement and fusion

Abstract

A remote sensing image change detection method and device based on feature enhancement and fusion belongs to the field of remote sensing image change detection technology, and particularly relates to real-time processing of remote sensing images on the ground, on satellites, or on drones. It solves the problems of low accuracy, poor feature representation ability, insufficient category balance ability, and inability to have high precision and efficiency while maintaining low computational complexity in existing change detection technologies. The detection method comprises: inputting registered dual-phase remote sensing image data of the same area to be detected, and extracting three levels of features of the data respectively, and then performing global and local feature enhancement to obtain three corresponding difference features; further, performing fusion processing to obtain aggregated features; and finally, obtaining remote sensing image change detection results based on the features. The present invention is suitable for deployment on the ground, on drones, or on satellites for change detection of remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image change detection, and in particular relates to real-time processing of remote sensing images on the ground, on a satellite or on an unmanned aerial vehicle. Background Art

[0002] Change detection quantitatively analyzes and identifies the characteristics and processes of surface change from remote sensing data over time. Change detection in remote sensing images essentially detects surface changes over time. my country's various satellites have returned a vast amount of remote sensing imagery. Using change detection algorithms to analyze this data is of great significance, primarily in mining and extracting valuable information over time. Currently, change detection applications primarily include observing landforms and topographic changes. It is crucial for obtaining dynamic information about key areas of interest and adjusting response plans in a timely manner. Furthermore, when a satellite is in a fixed-view mode (i.e., continuously capturing images of a target area from the same location without changing its camera's viewing angle), on-orbit change detection of the image sequence can eliminate duplicate images and extract effective information, significantly reducing the amount of data required for downlink and alleviating bandwidth pressure.

[0003] With the widespread adoption of high-resolution remote sensing imagery, change detection has become one of the most researched topics in remote sensing. Traditional image change detection methods involve image processing, difference map generation, and difference map analysis. Mainstream methods in the difference map generation stage include difference methods, ratio methods, wavelet fusion, log-ratio methods, and related improved operators. Most of these methods require domain-specific prior knowledge and then manually design and extract image features such as texture, morphology, and neighborhoods for change detection. The extracted image features are relatively shallow, making it difficult to fully model the image, and are time-consuming and labor-intensive. Traditional image change detection methods rely on manual design and lack the ability to represent features. They can only achieve good results in relatively simple scenes, but often exhibit poor accuracy in complex scenes.

[0004] In recent years, deep learning-based change detection methods have developed and gradually become the preferred tool for remote sensing image analysis. The application of deep learning in remote sensing image change detection has shown promising results in overcoming the limitations of image processing algorithms, improving change detection accuracy, and automating the process. Building on deep learning methods, designing a more intelligent change detection process for planetary remote sensing imagery by building networks and defining optimization rules has important research value and theoretical significance for further improving change detection accuracy and fully realizing the potential application value of deep learning models. In recent years, deep learning-based change detection methods, such as convolutional neural networks and Transformers, have shown promising results in terms of change detection accuracy. However, these methods are often complex, making them difficult to deploy on drones or satellites.

[0005] In addition, since the areas of change in remote sensing images are often much smaller than the areas that do not change, there will be a phenomenon of class imbalance, which in turn leads to poor change detection performance; existing change detection methods cannot solve this problem well.

[0006] Therefore, it is extremely necessary to study lightweight change detection methods, models or devices. Summary of the Invention

[0007] The present invention proposes a remote sensing image change detection method and device based on feature enhancement and fusion, which solves the problems of low accuracy, poor feature representation ability, and insufficient category balance ability existing in existing change detection technologies; and the problem that existing change detection technologies cannot maintain low computational complexity while having high accuracy and efficiency, making it impossible to apply and deploy change detection technologies with high accuracy and efficiency on drones or satellites.

[0008] The remote sensing image change detection method based on feature enhancement and fusion described in the present invention has the following technical solutions:

[0009] The method comprises:

[0010] S1, input the registered dual-phase remote sensing image data of the same area to be detected, and extract three levels of features of the dual-phase remote sensing image data respectively; wherein the dual-phase remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2;

[0011] S2. performing global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features;

[0012] S3, fusing the three difference features that enhance the global and local features to obtain aggregated features;

[0013] S4. Obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface topography of the same area to be detected at time T2 relative to time T1.

[0014] The present invention also proposes a remote sensing image change detection device based on feature enhancement and fusion, and its technical solution is as follows:

[0015] The device includes a dual-phase image feature extraction module, a global and local feature enhancement module, a progressive phase feature fusion module, and a classification detection module; wherein:

[0016] The dual-temporal image feature extraction module is used to input the dual-temporal remote sensing image data after registration of the same area to be detected, and respectively extract three levels of features of the dual-temporal remote sensing image data; wherein the dual-temporal remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2;

[0017] The global and local feature enhancement module is used to perform global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features;

[0018] The progressive temporal feature fusion module is used to fuse the three difference features that enhance the global and local features to obtain aggregated features;

[0019] The classification detection module is used to obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface terrain of the same area to be detected at time T2 relative to time T1.

[0020] Furthermore, a preferred embodiment is provided: the dual-temporal image feature extraction module uses a shufflenet network as a feature extractor;

[0021] The feature extractor is used to extract the three-level features of the dual-phase remote sensing image data; the feature extractor includes a neck and three-stage feature output position

[0022] The neck portion is used to downsample the input dual-temporal remote sensing image data to a predetermined size;

[0023] The three stage feature output positions are respectively a first stage feature output position, a second stage feature output position and a third stage feature output position; the three stage feature output positions are used to extract and output three levels of features based on the downsampled dual-temporal remote sensing image data;

[0024] The first stage feature output position is used to extract and output the first level features, namely the first level features Sgl_T1 of the remote sensing image data at time T1, and the first level features Sg2_T1 of the remote sensing image data at time T2;

[0025] The second stage feature output position is used to extract and output the second level features, namely the second level features Sgl_T2 of the remote sensing image data at time T1, and the second level features Sg2_T2 of the remote sensing image data at time T2;

[0026] The third stage feature output position is used to extract and output the third level features, namely the third level features Sgl_T3 of the remote sensing image data at time T1 and the third level features Sg2_T3 of the remote sensing image data at time T2.

[0027] Furthermore, a preferred embodiment is provided: the global and local feature enhancement module includes a global branch, a local branch, a skip connection branch, a summing unit, and a difference unit;

[0028] The global branch is used to receive the three levels of features output by the dual-temporal image feature extraction module, and enhance the global features among the three levels of features to obtain the three levels of features with enhanced global features;

[0029] The local branch is used to receive the three-level features output by the dual-temporal image feature extraction module, and enhance the local features in the three-level features to obtain the three-level features with enhanced local features;

[0030] The jump connection branch is used to receive the three levels of features output by the dual-temporal image feature extraction module and obtain correction parameters, wherein the correction parameters are used to correct the output results of the global branch and the local branch;

[0031] The adding unit is used to add the three levels of features that enhance the global features output by the global branch, the three levels of features that enhance the local features output by the local branch, and the correction parameters output by the jump connection branch to obtain the three levels of features Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2 that enhance the global and local features;

[0032] The difference unit is used to calculate the feature map tensor differences between Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2, respectively, to obtain three difference features that enhance global and local features; the three difference features are the first difference feature Df3, the second difference feature Df4, and the third difference feature Df5.

[0033] Furthermore, a preferred embodiment is provided: the global branch includes two separable self-attention modules; the two separable self-attention modules have global feature extraction capabilities.

[0034] Furthermore, a preferred embodiment is provided: the progressive temporal feature fusion module includes three upsampling units, three convolution blocks with different dilation rates, a first combination unit, and a second combination unit; the three upsampling units are respectively a first upsampling unit, a second upsampling unit, and a third upsampling unit; the three convolution blocks with different dilation rates are respectively a convolution block with a dilation rate of 1, a convolution block with a dilation rate of 3, and a convolution block with a dilation rate of 5;

[0035] The convolution block with a void rate of 1 is used to perform a convolution operation on the third difference feature Df5 to obtain the convolved third difference feature Df5;

[0036] The first upsampling unit is configured to perform an upsampling operation on the convolved third difference feature Df5 to obtain the upsampled third difference feature Df5;

[0037] The first combining unit is configured to fuse the upsampled third difference feature Df5 with the second difference feature Df4 to obtain a first fused feature;

[0038] The convolution block with a dilation rate of 3 is used to perform a convolution operation on the first fused feature to obtain the first fused feature after convolution;

[0039] The second upsampling unit is configured to perform an upsampling operation on the convolved first fused feature to obtain the upsampled first fused feature;

[0040] The second combining unit is configured to fuse the upsampled first fused feature with the first difference feature Df3 to obtain a second fused feature;

[0041] The convolution block with a void rate of 5 is used to perform a convolution operation on the second fused feature to obtain the convolved second fused feature;

[0042] The third upsampling unit is used to perform an upsampling operation on the second fused features after convolution to obtain aggregated features.

[0043] Furthermore, a preferred embodiment is provided: the classification detection module is a convolution classification head;

[0044] The convolution classification head includes a convolution unit at the front end and a classification unit at the back end;

[0045] The convolution unit of the front end is used to perform a convolution operation on the aggregated features to obtain the convolved aggregated features;

[0046] The classification unit of the backend is used to classify the aggregated features after convolution to obtain remote sensing image change detection results.

[0047] Furthermore, a preferred embodiment is provided: the device is constructed based on deep learning training; when the device performs deep learning training, the loss function used is a hybrid loss function L total (pre,gt);

[0048] The hybrid loss function L total (pre,gt), by the BEC loss function L bce (pre,gt) and Dice loss function L dice (pre,gt) composition:

[0049] L total (pre,gt)=L bce (pre,gt)+L dice (pre,gt);

[0050] L bce (pre,gt)=gt·log(pre)+(1-gt)·log(1-pre);

[0051]

[0052] Wherein, gt represents the ground truth map of the changed area, in which the non-changed area is marked as 0 and the changed area is marked as 1; pre represents the remote sensing image change detection result.

[0053] The present invention also proposes an electronic device, the technical solution of which is as follows:

[0054] An electronic device includes: a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to perform the above-mentioned remote sensing image change detection method based on feature enhancement and fusion by executing the executable instructions.

[0055] The present invention also proposes a computer storage medium, the technical solution of which is as follows:

[0056] A computer storage medium stores a computer program, which, when running, executes the above-mentioned remote sensing image change detection method based on feature enhancement and fusion.

[0057] The present invention has the following beneficial effects:

[0058] 1. The remote sensing image change detection device based on feature enhancement and fusion described in the present invention adopts global and local feature enhancement modules. While ensuring low computational complexity, it enhances the global and local features in the remote sensing image data, improves the feature representation capability of the device, and further improves the precision and accuracy of change detection.

[0059] 2. The remote sensing image change detection device based on feature enhancement and fusion described in the present invention adopts a progressive temporal feature fusion module to effectively fuse the difference features of different levels, thereby improving the feature representation capability of the device, that is, enhancing the device's ability to discriminate difference change information in dual-temporal remote sensing image data, and further improving the precision and accuracy of change detection.

[0060] 3. The remote sensing image change detection device based on feature enhancement and fusion described in the present invention not only has high accuracy but also has sufficiently low computational complexity. The device has a parameter count of 1.76M and a computational complexity of 1.91G FLOPs. It is an extremely lightweight change detection device that can be deployed not only on ground equipment but also in the hardware environments of drones or satellites.

[0061] The remote sensing image change detection method and device based on feature enhancement and fusion described in the present invention are suitable for deployment on the ground, on an unmanned aerial vehicle (UAV) or on a satellite to detect changes in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 This is a structural diagram of a remote sensing image change detection device based on feature enhancement and fusion in an embodiment of the present invention;

[0064] Figure 2 This is a structural diagram of the global and local feature enhancement modules in an embodiment of the present invention;

[0065] Figure 3A visual comparison diagram of change detection results obtained by performing change detection using multiple remote sensing image change detection methods in an embodiment of the present invention;

[0066] The above-mentioned Figure 3 middle:

[0067] (a): Remote sensing image data at time T1;

[0068] (b): Remote sensing image data at time T2;

[0069] (c) The actual result of the change of the surface topography of the same area to be detected at time T2 compared with time T1 in the dual-temporal remote sensing image data;

[0070] (d) The change detection results of the surface terrain at time T2 compared with time T1 for the same area to be detected are obtained by using the SNUNet model for change detection.

[0071] (e) The change detection results of the surface terrain at time T2 compared with time T1 in the same area to be detected are obtained by using the ICIFNet model for change detection.

[0072] (f) Change detection results of the surface terrain at time T2 compared to time T1 in the same area to be detected using the BIT model;

[0073] (g) The change detection results of the surface terrain at time T2 compared with time T1 for the same area to be detected are obtained by using the MSPSNet model for change detection.

[0074] (h): Change detection results of the surface terrain at time T2 compared with time T1 for the same area to be detected when using the remote sensing image change detection method based on feature enhancement and fusion;

[0075] In Figures (b)-(h), the white part represents the changed area, and the black part represents the unchanged area. DETAILED DESCRIPTION

[0076] To make the technical solutions and advantages of the present invention more clearly described, the specific implementation methods of the embodiments of the present invention will be further described in detail and completely with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] Implementation Method 1: Combination Figure 1-3 Description of this embodiment: This embodiment provides a remote sensing image change detection method based on feature enhancement and fusion. The specific implementation contents are as follows:

[0078] A remote sensing image change detection method based on feature enhancement and fusion, the method comprising:

[0079] S1, input the registered dual-phase remote sensing image data of the same area to be detected, and extract three levels of features of the dual-phase remote sensing image data respectively; wherein the dual-phase remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2;

[0080] S2. performing global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features;

[0081] S3, fusing the three difference features that enhance the global and local features to obtain aggregated features;

[0082] S4. Obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface topography of the same area to be detected at time T2 relative to time T1.

[0083] Implementation Method 2: Combination Figure 1 -3 Description of this embodiment: This embodiment provides a remote sensing image change detection device based on feature enhancement and fusion. The specific implementation contents are as follows:

[0084] A remote sensing image change detection device based on feature enhancement and fusion, comprising a dual-temporal image feature extraction module, a global and local feature enhancement module, a progressive temporal feature fusion module, and a classification detection module; wherein:

[0085] The dual-temporal image feature extraction module is used to input the dual-temporal remote sensing image data after registration of the same area to be detected, and respectively extract three levels of features of the dual-temporal remote sensing image data; wherein the dual-temporal remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2;

[0086] The global and local feature enhancement module is used to perform global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features;

[0087] The progressive temporal feature fusion module is used to fuse the three difference features that enhance the global and local features to obtain aggregated features;

[0088] The classification detection module is used to obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface terrain of the same area to be detected at time T2 relative to time T1.

[0089] In this embodiment, the remote sensing image change detection device based on feature enhancement and fusion is used to implement the remote sensing image change detection method based on feature enhancement and fusion described in the first embodiment.

[0090] In this embodiment, the remote sensing image change detection device based on feature enhancement and fusion is constructed based on a deep learning network;

[0091] In this embodiment, the types of remote sensing image data include visible light images and multispectral images.

[0092] Implementation Method 3: Combination Figure 1 -3. Description of this embodiment. This embodiment further defines the dual-temporal image feature extraction module in the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 2. Specific implementation details are as follows:

[0093] The dual-phase image feature extraction module uses the shufflenet network as a feature extractor;

[0094] The feature extractor is used to extract the three-level features of the dual-phase remote sensing image data; the feature extractor includes a neck and three-stage feature output position

[0095] The neck portion is used to downsample the input dual-temporal remote sensing image data to a predetermined size;

[0096] The three stage feature output positions are respectively a first stage feature output position, a second stage feature output position and a third stage feature output position; the three stage feature output positions are used to extract and output three levels of features based on the downsampled dual-temporal remote sensing image data;

[0097] The first stage feature output position is used to extract and output the first level features, namely the first level features Sgl_T1 of the remote sensing image data at time T1, and the first level features Sg2_T1 of the remote sensing image data at time T2;

[0098] The second stage feature output position is used to extract and output the second level features, namely the second level features Sgl_T2 of the remote sensing image data at time T1, and the second level features Sg2_T2 of the remote sensing image data at time T2;

[0099] The third stage feature output position is used to extract and output the third level features, namely the third level features Sgl_T3 of the remote sensing image data at time T1 and the third level features Sg2_T3 of the remote sensing image data at time T2.

[0100] In this embodiment, the ShuffleNet network is a lightweight neural network. The purpose of this embodiment is to provide an apparatus, method, or system that can be deployed on a drone or satellite. If other non-lightweight neural networks are used, it will be difficult to effectively operate in the hardware environment of drones or satellites.

[0101] In this embodiment, the neck includes a convolution block with a stride of 2 and a kernel size of 2 and a maximum pooling layer.

[0102] In this embodiment, each of the three stage feature output positions uses a shufflenet basic unit and a shufflenet downsampling unit. The shufflenet basic unit and the shufflenet downsampling unit mainly include channel separation, 1×1 convolution, depthwise convolution, and channel shuffling operations.

[0103] In this embodiment, it is assumed that the image height of the dual-phase remote sensing image data of the area to be detected is H, and the image width is W: then the feature map space size of the first-level feature is H / 8 in height and W / 8 in width; the feature map space size of the second-level feature is H / 16 in height and W / 16 in width; the feature map space size of the third-level feature is H / 32 in height and W / 32 in width; that is, for the three-stage feature output positions, the height and width of the feature map space output by the feature output position of the latter stage are respectively half of the height and width of the feature map space output by the feature output position of the previous stage.

[0104] Implementation Method 4: Combination Figure 1 -3. Description of this embodiment. This embodiment further defines the global and local feature enhancement modules in the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 3. Specific implementation details are as follows:

[0105] The global and local feature enhancement module includes a global branch, a local branch, a jump connection branch, a summing unit, and a difference unit;

[0106] The global branch is used to receive the three levels of features output by the dual-temporal image feature extraction module, and enhance the global features among the three levels of features to obtain the three levels of features with enhanced global features;

[0107] The local branch is used to receive the three-level features output by the dual-temporal image feature extraction module, and enhance the local features in the three-level features to obtain the three-level features with enhanced local features;

[0108] The jump connection branch is used to receive the three levels of features output by the dual-temporal image feature extraction module and obtain correction parameters, wherein the correction parameters are used to correct the output results of the global branch and the local branch;

[0109] The adding unit is used to add the three levels of features that enhance the global features output by the global branch, the three levels of features that enhance the local features output by the local branch, and the correction parameters output by the jump connection branch to obtain the three levels of features Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2 that enhance the global and local features;

[0110] The difference unit is used to calculate the feature map tensor differences between Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2, respectively, to obtain three difference features that enhance global and local features; the three difference features are the first difference feature Df3, the second difference feature Df4, and the third difference feature Df5.

[0111] In this embodiment, the three difference features, namely the first difference feature Df3, the second difference feature Df4 and the third difference feature Df5, are expressed by the formulas:

[0112]

[0113] In this embodiment, the local branch includes a 1×1 convolutional layer and two 3×3 convolutional layers.

[0114] In this implementation, the skip connection branch includes a 1×1 convolutional layer.

[0115] In this embodiment, global information is crucial for remote sensing image change detection, while the operations employed by the dual-temporal image feature extraction module are all local operations. Therefore, the multi-level (three-level) features obtained by the dual-temporal image feature extraction module lack global information. This embodiment enhances global and local features by inputting the different levels of features output by the dual-temporal image feature extraction module into the global and local feature enhancement module. In addition to its global and local feature extraction capabilities, the global and local feature enhancement module itself is a lightweight module.

[0116] In this embodiment, the output result of the global and local feature enhancement module is the sum of the output results of the global branch, the local branch, and the jump connection branch, which can be expressed as follows:

[0117] output=local(input)+global(input)+skip(input);

[0118] Among them, output represents the output of the global and local feature enhancement module; input represents the input of each branch; local (·) represents the local branch; global (·) represents the global branch; skip (·) represents the skip connection branch.

[0119] Implementation Method 5: Combination Figure 1 -3. This embodiment further defines the global branch in the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 4. The specific implementation details are as follows:

[0120] The global branch includes two separable self-attention modules; the two separable self-attention modules have global feature extraction capabilities.

[0121] In this implementation, the global branch further includes a 1×1 convolutional layer.

[0122] In this embodiment, the separable self-attention module with global feature extraction capability has lower complexity than the Transformer structure with the same global feature extraction capability, and can capture global information at a very low computational complexity. The complexity of the Transformer structure is O(n 2 ), while the complexity of the separable self-attention module used in this embodiment is O(n). The purpose of this embodiment is to provide a model, method, or system that can be deployed on drones or satellites. If the existing Transformer structure is used to extract global information, its complexity is too high to be effectively implemented in hardware environments such as drones or satellites.

[0123] The data types of the input and output of the separable self-attention module are both feature map tensors;

[0124] The input is X and the output is Y, where

[0125] Its output Y can be expressed as:

[0126]

[0127]

[0128] Cs =softmax(XW I );

[0129] W I ∈R d ,

[0130] W K ∈R d×d ,

[0131] W O ∈R d×d ,

[0132] Among them, X is the input of the separable self-attention module, W I 、W K and W O is the weight parameter, R d represents the d-dimensional real number set, R d represents a d×d dimensional real number set, C V is the context vector, C S is the context score;

[0133] Combining the first three formulas, we get:

[0134]

[0135] Implementation Method 6: Combination Figure 1 -3 Description of this embodiment. This embodiment further defines the progressive temporal feature fusion module in the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 3. Specific implementation details are as follows:

[0136] The progressive temporal feature fusion module includes three upsampling units, three convolution blocks with different hole rates, a first combination unit, and a second combination unit; the three upsampling units are respectively a first upsampling unit, a second upsampling unit, and a third upsampling unit; the three convolution blocks with different hole rates are respectively a convolution block with a hole rate of 1, a convolution block with a hole rate of 3, and a convolution block with a hole rate of 5;

[0137] The convolution block with a void rate of 1 is used to perform a convolution operation on the third difference feature Df5 to obtain the convolved third difference feature Df5;

[0138] The first upsampling unit is configured to perform an upsampling operation on the convolved third difference feature Df5 to obtain the upsampled third difference feature Df5;

[0139] The first combining unit is configured to fuse the upsampled third difference feature Df5 with the second difference feature Df4 to obtain a first fused feature;

[0140] The convolution block with a dilation rate of 3 is used to perform a convolution operation on the first fused feature to obtain the first fused feature after convolution;

[0141] The second upsampling unit is configured to perform an upsampling operation on the convolved first fused feature to obtain the upsampled first fused feature;

[0142] The second combining unit is configured to fuse the upsampled first fused feature with the first difference feature Df3 to obtain a second fused feature;

[0143] The convolution block with a void rate of 5 is used to perform a convolution operation on the second fused feature to obtain the convolved second fused feature;

[0144] The third upsampling unit is used to perform an upsampling operation on the second fused features after convolution to obtain aggregated features.

[0145] In this implementation, the three upsampling units are configured to perform upsampling using a bilinear interpolation method.

[0146] In this implementation, a progressive temporal feature fusion module is proposed to better capture temporal variation information. In this progressive temporal feature fusion module, convolution blocks with high dilation rates act on larger feature maps, while convolution blocks with low dilation rates act on smaller feature maps. This allows for maintaining a relatively large receptive field from beginning to end and ensuring that multi-level variation features can be effectively combined.

[0147] Implementation Method VII: Combination Figure 1 -3 Description of this embodiment. This embodiment further defines the classification detection module in the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 2. The specific implementation contents are as follows:

[0148] The classification detection module is a convolution classification head;

[0149] The convolution classification head includes a convolution unit at the front end and a classification unit at the back end;

[0150] The convolution unit of the front end is used to perform a convolution operation on the aggregated features to obtain the convolved aggregated features;

[0151] The classification unit of the backend is used to classify the aggregated features after convolution to obtain remote sensing image change detection results.

[0152] In this embodiment, the convolution unit of the front end includes two convolution blocks and one convolution.

[0153] Implementation Method 8: Combination Figure 1-3 Description of this embodiment. This embodiment further defines the remote sensing image change detection device based on feature enhancement and fusion described in Embodiment 2. The specific implementation contents are as follows:

[0154] The device is constructed based on deep learning training; when the device performs deep learning training, the loss function used is a hybrid loss function L total (pre,gt);

[0155] The hybrid loss function L tital (pre,gt), by the BEC loss function L bce (pre,gt) and Dice loss function L dice (pre,gt) composition:

[0156] L total (pre,gt)=L bce (pre,gt)+L dice (pre,gt);

[0157] L bce (pre,gt)=gt·log(pre)+(1-gt)·log(1-pre);

[0158]

[0159] Wherein, gt represents the ground truth map of the changed area, in which the non-changed area is marked as 0 and the changed area is marked as 1; pre represents the remote sensing image change detection result.

[0160] In this embodiment, the remote sensing image change detection device based on feature enhancement and fusion uses a hybrid loss function L when performing deep learning training. total (pre,gt), can reduce the category imbalance phenomenon and thus improve the performance of change detection.

[0161] In this embodiment, considering that in the change detection task, the changed area in the dual-temporal remote sensing image is often much smaller than the unchanged area, there will be a class imbalance phenomenon, which further affects the detection performance of the change detection method. In order to reduce the class imbalance phenomenon, in this embodiment, the remote sensing image change detection device adopts a hybrid loss function L when performing deep learning training. total (pre,gt). The hybrid loss function L total (pre,gt), which is different from the cross entropy loss function commonly used in the industry, can reduce the phenomenon of class imbalance and improve the performance of change detection.

[0162] The hybrid loss function Ltotal (pre,gt), by the BEC loss function L bce (pre,gt) and Dice loss function L dice (pre, gt); BEC stands for Binary Cross Entropy. The Dice loss function is a region-dependent loss function that excels at mining positive samples. It focuses on mining foreground regions, thereby reducing the problem of poor change detection performance caused by class imbalance.

[0163] Implementation Method 9: Combination Figure 3 To illustrate this embodiment, in order to detect the change detection performance of the remote sensing image change detection method and device based on feature enhancement and fusion, this embodiment provides a comparative experiment; the comparative experiment uses multiple remote sensing image change detection methods to perform change detection on the same dual-phase remote sensing image data, and compares the change detection results of each method.

[0164] As attached Figure 3 As shown in the figure, it shows the various change detection results obtained when using multiple remote sensing image change detection methods to detect changes in the same area to be detected; in the attached Figure 3 In the present invention, the multiple remote sensing image change detection methods include: a method for change detection using the SNUNet model, a method for change detection using the ICIFNet model, a method for change detection using the BIT model, a method for change detection using the MSPSNet model, and the remote sensing image change detection method based on feature enhancement and fusion described in the above embodiment.

[0165] As attached Figure 3As shown, (a) is the remote sensing image data at time T1; (b) is the remote sensing image data at time T2; (c) is the actual result of the change of the surface terrain of the same area to be detected at time T2 compared with time T1 in the dual-phase remote sensing image data; (d) is the change detection result of the surface terrain of the same area to be detected at time T2 compared with time T1 obtained by using the SNUNet (Densely Connected Twin Change Detection Network) model for change detection; (e) is the change detection result of the surface terrain of the same area to be detected at time T2 compared with time T1 obtained by using the ICIFNet (Intra-Scale Cross Interaction and Inter-Scale Feature Fusion Network) model for change detection; (f ) Figure 1 shows the change detection results of the surface terrain of the same area to be detected at time T2 compared with time T1, obtained by using the BIT (Bi-temporal Image Converter) model for change detection; (g) Figure 2 shows the change detection results of the surface terrain of the same area to be detected at time T2 compared with time T1, obtained by using the MSPSNet (Multi-scale Twin Network with Parallel Convolution and Self-Attention) model for change detection; (h) Figure 3 shows the change detection results of the surface terrain of the same area to be detected at time T2 compared with time T1, obtained by using the remote sensing image change detection method based on feature enhancement and fusion; Among them, in Figures (b)-(h), the white part represents the changed area, and the black part represents the unchanged area.

[0166] From the attached Figure 3 It can be seen that the method for change detection using the remote sensing image change detection device based on feature enhancement and fusion, that is, the remote sensing image change detection method based on feature enhancement and fusion, has better change detection performance than other change detection methods. Specifically:

[0167] First, the remote sensing image change detection method based on feature enhancement and fusion has a higher degree of completeness in detecting changed areas, the results are closest to the true values, and there are no holes in the changed areas. However, the change detection methods using models such as SNUNet, ICIFNet and MSPSNet all have missed detections of changed areas within their change detection results, resulting in the destruction of the integrity of the changed areas.

[0168] Secondly, the remote sensing image change detection method based on feature enhancement and fusion can more accurately detect the edges of changed areas and unchanged areas, and the detected edges are more consistent; when other models are used for change detection, more or less convex or concave phenomena appear at the edges.

[0169] Finally, the remote sensing image change detection method based on feature enhancement and fusion can more accurately identify non-changing areas and will not identify areas outside the changing area as changing areas.

[0170] Implementation Method 10: Combination Figure 3 To illustrate this embodiment, in order to further test the change detection performance of the remote sensing image change detection method and device based on feature enhancement and fusion, this embodiment provides another comparative experiment; this comparative experiment still compares the change detection performance of the above-mentioned multiple remote sensing image change detection methods; however, this comparative experiment uses indicator comparison to conduct evaluation.

[0171] The experimental datasets used in this comparative experiment are the LEVIR-CD and SYSU-CD datasets. The LEVIR-CD dataset is derived from images collected from 20 regions in the United States. It contains 637 pairs of 1024×1024 registered remote sensing images with a resolution of 0.5m and exhibits significant seasonal and illumination variations. This dataset focuses on building changes. Following the official segmentation method, the data was split into a 7:1:2 ratio for training, validation, and test sets. These images were then sliced ​​into 256×256 tiles, resulting in 7,120, 1,024, and 2,048 image pairs for training, validation, and testing, respectively. The SYSU-CD dataset contains 20,000 pairs of registered remote sensing images with a resolution of 0.5m. This dataset focuses on various types of changes, including changes in buildings, roads, water bodies, and ships. The number of image pairs in the training, validation, and test sets is 12,000, 4,000, and 4,000, respectively.

[0172] In this comparative experiment, a remote sensing image change detection device based on feature enhancement and fusion for change detection was implemented using the Pytorch framework and trained and inferred using an NVIDIA RTX A6000 GPU. During training, the backbone (the backbone refers to the weight-sharing information extraction backbone, i.e., the dual-temporal image feature extraction module) was initialized using pre-trained parameters from ImageNet. We used the Adam optimizer to optimize the parameters of our device, with an initial learning rate set to 0.0005 and a Poly learning rate descent algorithm to dynamically update the learning rate during training. The batch size and maximum number of iterations during training were set to 32 and 80,000, respectively. In addition, random temporal transformation, random mirroring, and random cropping were used for data augmentation.

[0173] In this comparative experiment, the evaluation indicators used are four accuracy evaluation indicators and three complexity evaluation indicators, as follows:

[0174] The four accuracy evaluation indicators include: Precision (P), Recall (R), F1 score (F1), and Intersection over Union (IoU), which are expressed as follows:

[0175]

[0176]

[0177]

[0178]

[0179] Among them, TP, FP, TN and FN represent true positives, false positives, true negatives and false negatives, respectively.

[0180] The three complexity evaluation indicators include: parameters (Params), floating point operations (FLOPs), and frames per second (FPS).

[0181] In this comparative experiment, the various remote sensing image change detection methods used for comparison are still: the method of change detection using the SNUNet model, the method of change detection using the ICIFNet model, the method of change detection using the BIT model, the method of change detection using the MSPSNet model, and the remote sensing image change detection method based on feature enhancement and fusion (referred to as this method in this comparative experiment).

[0182] This comparative experiment finally obtained the comparison results, as shown in the following table:

[0183]

[0184]

[0185] In the table above, the floating-point operations (FLOPs) are calculated based on a 256×256 input bi-temporal image. The frames per second (FPS) are calculated based on an RTX A6000 processor and a batch size of 16.

[0186] From the above table, we can draw the following conclusions:

[0187] In terms of accuracy comparison:

[0188] On the LEVIR-CD dataset: the P, R, F1 and IoU of this method are 91.03%, 89.91%, 90.47% and 82.60% respectively; the R, F1 and IoU of this method are 0.54%, 0.51% and 0.85% higher than those of the second best method.

[0189] On the SYSU-CD dataset: the P, R, F1 and IoU of this method are 84.16%, 79.04%, 81.52% and 68.80% respectively; the P, R, F1 and IoU of this method are 2.14%, 1.25%, 2.83% and 3.93% higher than those of the second-best method.

[0190] It can be seen that on the LEVIR-CD dataset, among the four accuracy evaluation indicators of this method, the best evaluation results were obtained in recall rate R, F1 score and intersection over union (IoU), and only the second-best evaluation result was obtained in precision P; on the SYSU-CD dataset, the four accuracy evaluation indicators of this method all achieved the best evaluation results.

[0191] In terms of complexity comparison:

[0192] In terms of the complexity evaluation index of floating-point operations (FLOPs(G),) the evaluation result of this method is 1.91, while the evaluation results of other methods are all in double digits. Compared with other methods, this method reduces the number of floating-point operations by an order of magnitude.

[0193] At the same time, the evaluation result of this method on the complexity evaluation index of frames per second (FPS) is 914, while the evaluation results of other methods are less than 200, indicating that the inference speed of this method is more than 5 times that of other methods; this shows that the complexity of the remote sensing image change detection device based on feature enhancement and fusion adopted by this method has extremely excellent performance compared with the models adopted by other methods.

[0194] In summary, compared to the models used in other methods, the remote sensing image change detection device based on feature enhancement and fusion employed in this method has the fewest parameters, the lowest computational complexity, and the fastest inference speed. Specifically, the remote sensing image change detection device based on feature enhancement and fusion used in this comparative experiment has only 1.75M parameters, a computational complexity of only 1.91GFLOPs, and an inference speed of 897FPS.

[0195] The above further describes the technical solution provided by the present invention in detail through several specific embodiments in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific embodiments described above are not intended to limit the present invention. Any reasonable changes and improvements to the present invention, reasonable combinations of implementation methods and equivalent replacements based on the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote sensing image change detection method based on feature enhancement and fusion, characterized in that: The method comprises: S1, input the registered dual-phase remote sensing image data of the same area to be detected, and extract three levels of features of the dual-phase remote sensing image data respectively; wherein the dual-phase remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2; S2. performing global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features; S3, fusing the three difference features that enhance the global and local features to obtain aggregated features; S4. Obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface topography of the same area to be detected at time T2 relative to time T1; S2 is implemented by using global branches, local branches, skip connection branches, summing units and difference units; The global branch is used to receive the three-level features output by S1 and enhance the global features among the three-level features to obtain the three-level features with enhanced global features; The local branch is used to receive the three-level features output by S1 and enhance the local features in the three-level features to obtain the three-level features with enhanced local features; The skip connection branch is used to receive the three-level features output by S1 and obtain correction parameters, and the correction parameters are used to correct the output results of the global branch and the local branch; The adding unit is used to add the three levels of features that enhance the global features output by the global branch, the three levels of features that enhance the local features output by the local branch, and the correction parameters output by the jump connection branch to obtain the three levels of features Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2 that enhance the global and local features; The difference calculation unit is used to calculate the feature map tensor differences between Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2, respectively, to obtain three difference features that enhance global and local features; the three difference features are respectively a first difference feature Df3, a second difference feature Df4, and a third difference feature Df5; The global branch includes two separable self-attention modules and a 1×1 convolutional layer; The local branch includes a 1×1 convolutional layer and two 3×3 convolutional layers; The skip connection branch includes a 1×1 convolutional layer.

2. A remote sensing image change detection device based on feature enhancement and fusion, characterized in that: The device includes a dual-phase image feature extraction module, a global and local feature enhancement module, a progressive phase feature fusion module, and a classification detection module; wherein: The dual-temporal image feature extraction module is used to input the dual-temporal remote sensing image data after registration of the same area to be detected, and respectively extract three levels of features of the dual-temporal remote sensing image data; wherein the dual-temporal remote sensing image data are remote sensing image data at time T1 and remote sensing image data at time T2; The global and local feature enhancement module is used to perform global and local feature enhancement on the extracted features of the three levels to obtain three differential features with enhanced global and local features; The progressive temporal feature fusion module is used to fuse the three difference features that enhance the global and local features to obtain aggregated features; The classification detection module is used to obtain a remote sensing image change detection result based on the aggregated features; the remote sensing image change detection result is the changed area and the non-changed area of ​​the surface topography of the same area to be detected at time T2 relative to time T1; The global and local feature enhancement module includes a global branch, a local branch, a jump connection branch, a summing unit, and a difference unit; The global branch is used to receive the three levels of features output by the dual-temporal image feature extraction module, and enhance the global features among the three levels of features to obtain the three levels of features with enhanced global features; The local branch is used to receive the three-level features output by the dual-temporal image feature extraction module, and enhance the local features in the three-level features to obtain the three-level features with enhanced local features; The jump connection branch is used to receive the three levels of features output by the dual-temporal image feature extraction module and obtain correction parameters, wherein the correction parameters are used to correct the output results of the global branch and the local branch; The adding unit is used to add the three levels of features that enhance the global features output by the global branch, the three levels of features that enhance the local features output by the local branch, and the correction parameters output by the jump connection branch to obtain the three levels of features Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2 that enhance the global and local features; The difference calculation unit is used to calculate the feature map tensor differences between Sg1_T1 and Sg1_T2, Sg2_T1 and Sg2_T2, and Sg3_T1 and Sg3_T2, respectively, to obtain three difference features that enhance global and local features; the three difference features are respectively a first difference feature Df3, a second difference feature Df4, and a third difference feature Df5; The global branch includes two separable self-attention modules and a 1×1 convolutional layer; The local branch includes a 1×1 convolutional layer and two 3×3 convolutional layers; The skip connection branch includes a 1×1 convolutional layer.

3. The remote sensing image change detection device based on feature enhancement and fusion according to claim 2, characterized in that: The dual-phase image feature extraction module uses the shufflenet network as a feature extractor; The feature extractor is used to extract the three-level features of the dual-phase remote sensing image data; the feature extractor includes a neck and three-stage feature output position The neck portion is used to downsample the input dual-temporal remote sensing image data to a predetermined size; The three stage feature output positions are respectively a first stage feature output position, a second stage feature output position and a third stage feature output position; the three stage feature output positions are used to extract and output three levels of features based on the downsampled dual-temporal remote sensing image data; The first stage feature output position is used to extract and output the first level features, namely the first level features Sgl_T1 of the remote sensing image data at time T1, and the first level features Sg2_T1 of the remote sensing image data at time T2; The second stage feature output position is used to extract and output the second level features, namely the second level features Sgl_T2 of the remote sensing image data at time T1, and the second level features Sg2_T2 of the remote sensing image data at time T2; The third stage feature output position is used to extract and output the third level features, namely the third level features Sgl_T3 of the remote sensing image data at time T1 and the third level features Sg2_T3 of the remote sensing image data at time T2.

4. The remote sensing image change detection device based on feature enhancement and fusion according to claim 3 is characterized in that: The progressive temporal feature fusion module includes three upsampling units, three convolution blocks with different hole rates, a first combination unit, and a second combination unit; the three upsampling units are respectively a first upsampling unit, a second upsampling unit, and a third upsampling unit; the three convolution blocks with different hole rates are respectively a convolution block with a hole rate of 1, a convolution block with a hole rate of 3, and a convolution block with a hole rate of 5; The convolution block with a void rate of 1 is used to perform a convolution operation on the third difference feature Df5 to obtain the convolved third difference feature Df5; The first upsampling unit is configured to perform an upsampling operation on the convolved third difference feature Df5 to obtain the upsampled third difference feature Df5; The first combining unit is configured to fuse the upsampled third difference feature Df5 with the second difference feature Df4 to obtain a first fused feature; The convolution block with a dilation rate of 3 is used to perform a convolution operation on the first fused feature to obtain the first fused feature after convolution; The second upsampling unit is configured to perform an upsampling operation on the convolved first fused feature to obtain the upsampled first fused feature; The second combining unit is configured to fuse the upsampled first fused feature with the first difference feature Df3 to obtain a second fused feature; The convolution block with a void rate of 5 is used to perform a convolution operation on the second fused feature to obtain the convolved second fused feature; The third upsampling unit is used to perform an upsampling operation on the second fused features after convolution to obtain aggregated features.

5. The remote sensing image change detection device based on feature enhancement and fusion according to claim 2, characterized in that: The classification detection module is a convolution classification head; The convolution classification head includes a convolution unit at the front end and a classification unit at the back end; The convolution unit of the front end is used to perform a convolution operation on the aggregated features to obtain the convolved aggregated features; The classification unit of the backend is used to classify the aggregated features after convolution to obtain remote sensing image change detection results.

6. The remote sensing image change detection device based on feature enhancement and fusion according to claim 2, characterized in that: The device is constructed based on deep learning training; when the device performs deep learning training, the loss function used is a hybrid loss function ; The hybrid loss function , by the BEC loss function and Dice loss function composition: + ； + ； ； in, A truth map representing a changed region, wherein a non-changed region in the truth map is marked as 0 and a changed region is marked as 1; Represents the remote sensing image change detection results.

7. An electronic device comprising: A processor and a memory, characterized in that the memory is used to store executable instructions of the processor, and the processor is configured to execute the remote sensing image change detection method based on feature enhancement and fusion as described in claim 1 by executing the executable instructions.

8. A computer storage medium, characterized in that The storage medium stores a computer program, and when the computer program is run, the remote sensing image change detection method based on feature enhancement and fusion according to claim 1 is executed.

Images