Heterogenous remote sensing image building change detection method based on mutual distillation and difference evolution

Abstract

The invention particularly relates to a heterogeneous remote sensing image building change detection method based on mutual distillation and difference evolution. The method comprises the following steps: acquiring visible light and remote sensing images in the same region at different time; establishing a pseudo twin encoder change detection network for independently extracting different-source image features; a structure perception mutual distillation module is established, knowledge mutual distillation is carried out on structural representation of modal difference robustness in the heterogenous features, cross-modal alignment of the heterogenous features is realized, and the feature extraction performance of an encoder is enhanced; a difference dynamic evolution extraction module is established, a mapping relation between different-source features is modeled as a dynamic evolution process on a time domain, and on the basis of distinguishing characteristics of modal difference of an invariant region and difference of a change region in the dynamic evolution process, characterization related to the change is accurately extracted, the accuracy degree of prediction of the change region is effectively enhanced, and the prediction accuracy of the change region is improved. And missing detection can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of change detection technology, and in particular to a heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution, which realizes accurate change area detection of building areas in heterogeneous remote sensing image data of synthetic aperture radar (SAR) and visible light images. Background Art

[0002] Change detection is of great significance for disaster assessment, urban planning, environmental protection, etc. It can detect changes in objects of concern and provide comprehensive action references for planners or leaders. Most existing change detection methods use homologous data obtained by the same type of sensors for change detection. When faced with tasks such as disaster assessment that require extremely high timeliness and reliability, and the same modality data may not be obtained in a short period of time, homologous change detection methods are difficult to respond effectively. Therefore, it is of utmost importance to study heterogeneous change detection methods to perform change detection immediately regardless of the data modality after a disaster occurs, so as to provide accurate and reliable data support for disaster assessment and subsequent rescue missions.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present invention provides a method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution, which is used to solve the defect of existing change detection methods that they are deeply dependent on homologous data and achieve accurate detection of building change areas in heterogeneous remote sensing image data.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to a first aspect of the present invention, a method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution is provided, the method comprising:

[0007] Obtain visible light and SAR remote sensing images of the same area at different times, mark the areas where building changes occur, establish a heterogeneous remote sensing image change detection dataset, and divide the dataset into a training set and a test set;

[0008] A pre-trained heterogeneous change detection network is constructed, wherein the heterogeneous change detection network includes a pseudo-twin encoder and a decoder. The two branches of the pseudo-twin encoder have the same structure but do not share parameters, respectively extracting features of heterogeneous images and gradually reducing their dimensions; each layer of the pseudo-twin encoder includes a structured mutual distillation module for realizing cross-modal feature alignment, and a differential dynamic evolution extraction module for discriminatively extracting relevant representations of real change regions; the differential dynamic evolution extraction module inputs the extracted differential representations related to the real change regions into the layers of corresponding sizes in the decoder through skip connections, so as to guide the decoder to generate more accurate and fine-grained prediction masks of change regions;

[0009] The heterogeneous change detection network is trained using the training set, and the trained heterogeneous change detection network is tested using the test set to obtain the building change detection results of heterogeneous image data.

[0010] In some exemplary embodiments, the method further comprises: performing data augmentation on the training set, wherein the data augmentation comprises cropping, flipping and shifting.

[0011] In some exemplary embodiments, the structured inter-distillation module specifically includes the following processing steps:

[0012] The heterogeneous features output by the pseudo-twin encoder are cut into blocks, and the cut heterogeneous features are independently Fourier transformed and high-pass filtered in the image blocks. Then, the inverse Fourier transform is performed and the image blocks are spliced ​​back to the original input feature distribution to output the independent structured representation of the heterogeneous features.

[0013] Each element of the structured representation is mapped to a node, and the positional relationship between different elements is mapped to the edges connecting the nodes, a grid graph is constructed, and the structured representation is smoothed through a two-layer graph convolutional neural network;

[0014] The smoothed structured representation is masked using the mutual attention mechanism of transposed matrix multiplication; knowledge mutual distillation is performed on the masked structured representation.

[0015] In some exemplary embodiments, a loss function combining weighting and a weight increment factor is used to perform knowledge mutual distillation on the masked structured representation.

[0016] In some exemplary embodiments, the differential dynamic evolution extraction module specifically includes the following processing steps:

[0017] The intermediate transition feature from visible light feature to SAR feature is obtained by convolving the absolute value of the difference between the visible light feature and the SAR feature with the visible light feature, feature difference, and SAR feature in order;

[0018] The visible light features, intermediate transition features, and SAR features are reorganized according to the time series conversion angle to obtain the time domain evolution features;

[0019] The 3D convolution is used to capture the overall difference representation of the time domain evolution features in the time domain, including the modal differences in the unchanged area and the change differences in the changed area;

[0020] Perform time-domain average pooling on the time-domain evolution features, and effectively distinguish them based on the distinguishing features of modal differences and change differences in the time-domain pooling process;

[0021] The convolution kernel of the used 3D convolution is reconstructed, and the 3D convolution after convolution kernel reconstruction is used to perform difference extraction on the time domain pooling features to obtain the modal difference representation;

[0022] The overall difference representation is subtracted from the modal difference representation to obtain the difference representation related only to the changed area, and the 3D convolution is used to further refine and optimize the difference representation related to the changed area.

[0023] In some exemplary embodiments, the use of the training set to train the heterogeneous change detection network specifically includes: selecting AdamW as an optimizer, and using structured inter-distillation loss and change loss to optimize the network model.

[0024] In some exemplary embodiments, the optimizing the network model by using structured inter-distillation loss and variation loss includes:

[0025] Construct MSE loss as the loss function to supervise the full interaction of structured representation knowledge of each layer of encoder, and fuse the mutual distillation loss L corresponding to each layer of encoder in a weighted manner smd :

[0026]

[0027] Among them, y i and Represent the true value and predicted value of the i-th pixel respectively, N is the total number of encoder layers, and λ w is the weight factor of each layer of knowledge distillation loss;

[0028] Combining DICE loss and BCE loss to construct the change loss L cd , to solve the class imbalance problem between the changing and unchanged regions in heterogeneous data:

[0029]

[0030]

[0031] L cd =Ldice +λ1*L bce

[0032] Among them, λ1 is a hyperparameter;

[0033] By combining structural inter-distillation loss and change loss, the network model is guided to accurately predict the change area.

[0034] In some exemplary embodiments, according to the second aspect of the present invention, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution as described in the first aspect above is implemented.

[0035] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution as described in the first aspect is implemented.

[0036] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:

[0037] A processor; and a memory for storing executable instructions of the processor;

[0038] Among them, the processor is configured to implement the heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution described in the first aspect by executing the executable instructions.

[0039] The heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution provided by the embodiment of the present invention first constructs a feature representation and knowledge interaction mechanism that is robust to modal differences through a pseudo-twin encoder and a structured mutual distillation module, solves the interference of modal differences between SAR and visible light images on change detection, and improves the cross-modal alignment capability. At the same time, by designing a differential dynamic evolution extraction module, the relationship between heterogeneous features is modeled as a time domain evolution process, which can effectively distinguish between real changes and modal pseudo changes, extract accurate change area features from complex difference information, and significantly improve detection accuracy. In addition, a joint optimization strategy of structured mutual distillation loss and ratio change loss is adopted to guide the model to fine-tune cross-modal alignment in heterogeneous data change detection tasks, thereby improving the accuracy of change area prediction. Compared with existing homologous methods, this method significantly reduces its dependence on data modality, has stronger generalization ability, and is suitable for disaster assessment, urban planning and other scenarios. The method of the present invention has a clear process, efficient detection network structure design, low computing cost and high operating efficiency, is easy to deploy and promote, and provides reliable technical support for practical engineering applications.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0042] Figure 1 is a flow chart of a method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution in an exemplary embodiment of the present invention;

[0043] Figure 2 is a network framework diagram of heterogeneous change detection in an exemplary embodiment of the present invention;

[0044] Figure 3 is a structural mutual distillation module diagram in an exemplary embodiment of the present invention;

[0045] Figure 4 A diagram of a differential dynamic evolution extraction module in an exemplary embodiment of the present invention;

[0046] Figure 5 A heterogeneous building change detection map generated by a method according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] In view of the shortcomings and deficiencies of the prior art, a method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution is provided in this example implementation. The method first establishes a pseudo-twin encoder change detection network for independently extracting heterogeneous image features. Secondly, a structure-aware mutual distillation module is established to achieve cross-modal alignment of heterogeneous features and enhance the feature extraction performance of the encoder by performing knowledge mutual distillation on the structured representations that are robust to modal differences in heterogeneous features. In addition, a differential dynamic evolution extraction module is designed to model the mapping relationship between heterogeneous features as a dynamic evolution process in the time domain, and based on the distinguishing characteristics of the modal differences in the unchanged areas and the differences in the changing areas during the dynamic evolution process, accurately extract the representations related to the changes, effectively enhance the accuracy of the prediction of the changing areas, and help reduce missed detections.

[0050] The specific steps may include:

[0051] Step 1: Obtain visible light and SAR remote sensing images of the same area at different times, mark the areas where building changes occur, establish a heterogeneous remote sensing image change detection dataset, and divide the dataset into a training set and a test set;

[0052] Step 2: Perform data augmentation on the heterogeneous image building change detection training set, such as cropping, flipping and shifting, to generate training samples with dimensions of H×W×3, where H and W are length and width respectively;

[0053] Step 3: Input the training samples into the heterogeneous change detection network framework, and use the pseudo-twin encoder to capture the independent representations of the heterogeneous features. Each layer of the encoder is connected with a structured mutual distillation module and a differential dynamic evolution extraction module to achieve cross-modal alignment of heterogeneous features and discriminative and accurate extraction of change areas.

[0054] Specifically, the heterogeneous change detection network framework includes a pair of pseudo-twin encoders and a decoder. The two branches of the pseudo-twin encoder have the same structure but do not share parameters, respectively extracting features of heterogeneous images and gradually reducing their dimensions; each layer of the pseudo-twin encoder includes a structured inter-distillation module for achieving cross-modal feature alignment, and a difference dynamic evolution extraction module for discriminatively extracting relevant representations of the real change region, which is used to enhance the network's robustness to modal differences and improve the accuracy of change detection; the difference dynamic evolution extraction module inputs the extracted difference representations related to the real change region into the decoder layer of the corresponding size through jump connections, which is used to guide the decoder to generate more accurate and fine-grained change region prediction masks.

[0055] Step 4: Input the output of each layer of the pseudo-twin encoder in step 3 into the structured mutual distillation module of the current layer one by one. By performing knowledge mutual distillation on the structured representation of heterogeneous features, the features of the two source inputs are aligned, thereby guiding the encoder to overcome the interference of modal differences in extracting the representation of building area changes, so as to improve the robustness and discrimination ability of the pseudo-twin decoder.

[0056] Specifically, the structured inter-distillation module includes the following processing steps:

[0057] Step 4-1: Cut the heterogeneous features output by the pseudo-twin encoder into blocks, select different block sizes according to the different feature sizes output by the encoder, and independently perform Fourier transform on the heterogeneous features after cutting in the image block. After centering the change results, perform high-pass filtering to filter out the low-frequency information in the heterogeneous features, and only retain the transformation and texture information with drastic gradient changes in the reaction features. Then perform inverse Fourier transform and splice the image blocks back to the original input feature distribution, and output the independent structured representation of the heterogeneous features;

[0058] Step 4-2: Build a graph network, map each element of the heterogeneous features to a node, and use edges to represent the distribution between different nodes, that is, each node is only connected to four adjacent nodes, and embed the original structured representation into the corresponding node. The structured representation is processed through a two-layer graph convolutional neural network to achieve smoothing of the structured representation, so as to avoid the original extracted structured representation being too discrete, resulting in the knowledge mutual distillation process not being able to play a positive role;

[0059] Step 4-3: The mutual attention mechanism implemented by transposed matrix multiplication focuses on the unchanged areas in the heterogeneous features. Based on this attention, the structured representation is masked to shield the areas where real changes occur as much as possible, so as to prevent the subsequent knowledge mutual distillation process from affecting the representation of the changed areas and affecting the accuracy of the final change detection;

[0060] Step 4-4: Perform knowledge mutual distillation on the masked structured representation to achieve full knowledge interaction of heterogeneous features, and then realize cross-modal alignment of features, thereby improving the robustness of the pseudo-twin encoder to modal differences and the feature extraction capability of real change differences.

[0061] Step 5: The two outputs of each layer of the pseudo-twin encoder in step 3 are input into the difference dynamic evolution extraction module. In the module, the conversion correlation between heterogeneous features is modeled as a time domain evolution process, and then the difference representation related to the real change is accurately extracted based on the distinguishing characteristics of different difference representations in the time domain evolution process;

[0062] Specifically, the differential dynamic evolution extraction module includes the following processing steps:

[0063] Step 5-1: Convolve the original feature with the absolute value of the difference between the visible light feature and the SAR feature to obtain the intermediate transition feature from the visible light feature to the SAR feature;

[0064] Step 5-2: Reorganize the visible light features, intermediate transition features, and SAR features according to the perspective of time series conversion, and construct three independent 4-dimensional features into a 5-dimensional time domain evolution feature;

[0065] Step 5-3: Use 3D convolution to capture the overall difference representation of the time domain evolution characteristics of step 5-2 in the time domain, including the modal difference of the unchanged area and the change difference of the changed area, which are complexly coupled;

[0066] Step 5-4: Perform time-domain 3D average pooling on the time-domain evolution features described in step 5-2, and effectively distinguish them based on the distinguishing features of the above-mentioned modal differences and change differences shown in the time-domain pooling process;

[0067] Step 5-5: Reconstruct the 3D convolution used in step 5-3, and use the reconstructed 3D convolution to perform difference extraction on the time domain pooling features obtained in step 5-4 to maintain the modal difference representation with the same difference distribution and inductive bias as the overall difference;

[0068] Step 5-6: Subtract the overall difference representation obtained in step 5-3 from the modal difference representation obtained in step 5-5 to obtain a difference representation related only to the changed area.

[0069] Step 6: Input the difference representation of the real change area extracted layer by layer by the difference dynamic evolution extraction module on the pseudo twin encoder side in step 5 into the decoder of the corresponding size and fuse it with the decoder information to guide the decoder to generate a more refined change area mask;

[0070] Step 7: Select AdamW as the optimizer, and use structured inter-distillation loss and change loss to optimize the model to achieve cross-modal alignment of heterogeneous features, enhance the robustness of the encoder to modal differences, and improve the accuracy of prediction of change areas.

[0071] Below, each step of the heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0072] refer to Figure 1-5 The specific steps of a heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution provided by an embodiment of the present invention are as follows:

[0073] Figure 1The flowchart of the method of the embodiment of the present invention includes the following process: obtaining heterogeneous building change data samples and using Label_cd software to annotate the data, performing data augmentation on the data, generating training sets and test sets; constructing a heterogeneous change detection network framework, and constructing a structured mutual distillation module and a differential dynamic evolution extraction module thereon, and using the training set to train the heterogeneous change detection network; inputting the test data into the trained heterogeneous change detection network for testing, and obtaining heterogeneous image data building change detection results. The method specifically includes the following steps:

[0074] Step 1: Obtain SAR and visible light large-scale remote sensing image data from remote sensing satellites taken at different time points in the same area, select the area where the building has changed for processing and segmentation, use Labelme_cd to mark the area where the building has changed, and segment the large-scale remote sensing image. The image data is constructed into a standard format of 512×512 datasets by 1 / 2 overlapping segmentation, and the above samples are divided into training and test sets of heterogeneous image building change detection datasets in a ratio of 8:2;

[0075] Step 2: Perform data augmentation on the heterogeneous image building change detection training set, such as cropping, flipping, and shifting, to generate training samples with a dimension of 512×512×3;

[0076] Step 3: Input the training samples obtained in step 2 into the heterogeneous change detection network framework, and use the pseudo-twin encoder to capture the independent representations of the heterogeneous features respectively. Each layer of the encoder is connected with a structured mutual distillation module and a differential dynamic evolution extraction module to achieve cross-modal alignment of heterogeneous features and discriminative and accurate extraction of change areas;

[0077] Exemplary, reference Figure 2 ,The heterogeneous change detection network framework includes the following steps:

[0078] Step 3-1: The encoder of the heterogeneous change detection network framework is designed based on ResNet18 and adopts a pseudo-twin structure, that is, the two encoder branches have the same network structure but do not share weights with each other, and process the features from the optical image and the SAR image respectively;

[0079] Step 3-2: The encoder gradually downsamples the image features to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original through four encoder layers, and obtains high-level discriminative semantic features; each encoder layer is connected to a structured mutual distillation module and a difference dynamic evolution extraction module;

[0080] Step 3-3: The two heterogeneous high-level discriminative row semantic features output by the pseudo-twin encoder are input into the decoder through channel splicing. The corresponding decoder contains a four-layer structure. Each layer contains an upsampling module and a deconvolution module. The real change area representation output by the difference dynamic evolution extraction module connected to the encoder on the same layer is fused layer by layer. The final output is output through the change detection prediction head composed of two layers of full convolutional network FCN to output the change area mask.

[0081] Step 4: Input the two heterogeneous feature outputs of each layer of the pseudo twin encoder described in step 3 into the structured mutual distillation module of the current layer respectively, extract the structured representations of the heterogeneous features respectively, smooth the structured representations using a graph convolutional neural network, calculate the mutual attention of the original input to shield the structured representations of the changed area, and then perform knowledge mutual distillation on the remaining structured representations to achieve cross-modal alignment of features, thereby enhancing the encoder's robustness to modal differences and its ability to discriminate against modal difference interference;

[0082] Exemplary, reference Figure 3 , the structured mutual distillation module includes the following steps:

[0083] Step 4-1: According to the size of the input layer, the heterogeneous features are cut into blocks of 16×16, 8×8, and 4×4, and independent Fourier high-pass filtering is performed in each image block, retaining only 10% of the high-frequency information for processing. This information removes the low-frequency features related to the modal difference and retains only the structured representation. The image blocks are then reassembled and spliced. The block Fourier structure-aware filtering helps capture more detailed structural representations;

[0084] Step 4-2: Map each element of the structured representation to a node, and map the positional relationship between different elements to the edges connecting the nodes, build a grid graph, and use a two-layer graph convolutional neural network to smooth the structured representation, reduce discrete features, and effectively suppress noise to improve the performance of subsequent knowledge mutual distillation;

[0085] Step 4-3: Based on the heterogeneous features before filtering, the mutual attention mechanism of transposed matrix multiplication is used to obtain the attention representation of the area that has not actually changed, and use it to mask the above smoothed structured representation to ensure that during the mutual distillation process, the network focuses on achieving cross-modal alignment of the unchanged area without affecting the detection and extraction of the changed area;

[0086] Step 4-4: Use a loss function that combines weighting and weight increasing factors to perform knowledge mutual distillation on the masked heterogeneous structured representation. The superposition of multiple layers of encoders gradually realizes cross-modal alignment of heterogeneous features and improves the robustness of the encoder to modal differences.

[0087] Step 5: Input the two outputs of each layer of the pseudo-twin encoder in step 3 into the difference dynamic evolution extraction module. Convolute the two with their absolute interpolation and combine them with the original features to construct the time domain evolution features. And accurately distinguish the real change area differences and modal differences through 3D convolution reconstruction and time domain average pooling.

[0088] Exemplary, reference Figure 4 ,The differential dynamic evolution extraction module includes the following steps:

[0089] Step 5-1: Obtain the absolute value of the difference between the visible light feature and the SAR feature, concatenate the channels with the original feature, and use convolution to fuse the features, reducing the number of channels to the original input size as the intermediate state of the conversion from the visible light feature to the SAR feature in the time domain perspective;

[0090] Step 5-2: Consider the relationship between visible light features and SAR features as an evolutionary process in the time domain, and expand the original features into the time dimension, that is, from the size of [B, C, H, W] to [B, C, T, H, W]. Stack the visible light features, conversion intermediate state features, and SAR features in the time dimension in sequence to construct the transformation process from visible light features to SAR features.

[0091] Step 5-3: Use 3D convolution to extract the time-domain evolution features from the time, length, and width dimensions to capture the difference representations between heterogeneous features;

[0092] Step 5-4: Perform time-domain 3D average pooling on the time-domain evolution features. The regions with only modal differences but no changes are enhanced in the average pooling process, while the difference representation of the regions with real changes is suppressed in the time-domain average pooling process due to their spatial discontinuity.

[0093] Step 5-5: Reconstruct the 3D convolution used in step 5-3, and only use the convolution kernel weight layer acting on visible light features and SAR features for fusion, and use the same inductive bias to extract modal difference representation from the evolution features after time domain average pooling;

[0094] Step 5-6: Subtract the overall difference representation obtained in step 5-3 from the modal difference representation obtained in step 5-5 to obtain a difference representation related only to the changed area.

[0095] Step 6: Input the difference representation of the real change area extracted layer by layer by the difference dynamic evolution extraction module on the pseudo twin encoder side in step 5 into the decoder of the corresponding size and fuse it with the decoder information to guide the decoder to generate a more refined change area mask;

[0096] Step 7: Select AdamW as the optimizer and use the heterogeneous change detection network framework to optimize the loss function, namely the structured mutual distillation loss L smd With the change loss L cd The model is optimized to achieve cross-modal alignment of heterogeneous features, enhance the encoder's robustness to modal differences, and improve the accuracy of prediction of changed areas:

[0097] L=L smd +L cd

[0098] Exemplarily, the optimization loss function of the heterogeneous change detection network framework includes the following steps:

[0099] Step 7-1: Construct MSE loss as the loss function to supervise the full interaction of structured representation knowledge of each layer of encoder, and fuse the Hu distillation loss corresponding to each layer of encoder in a weighted manner:

[0100]

[0101] Among them, y i and Represent the true value and predicted value of the i-th pixel respectively, N is the total number of encoder layers, and λ w It is the weight factor of each layer of knowledge distillation loss. It provides tolerance for knowledge mutual distillation in the early stage of training when the encoder feature extraction is not accurate enough, and enhances the strength of knowledge mutual distillation in the later stage of training to guide the refined cross-modal alignment of the model:

[0102]

[0103] Among them, t and T are the current training round and the total training round respectively;

[0104] Step 7-2: Combine DICE loss and BCE loss to construct the change loss L cd , to solve the class imbalance problem between the changing and unchanged regions in heterogeneous data:

[0105]

[0106] L cd =L dice +λ1*L bce

[0107] Among them, λ1 is a hyperparameter used to balance the DICE and BCE losses so that DICE does not over-focus on the change area and increase the misjudgment;

[0108] Step 7-3: Through the effective combination of change loss and structural inter-distillation loss, guide the model to accurately predict the change area:

[0109] L=L cd +L smd

[0110] The above-mentioned heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution is used to realize change detection of SAR and visible light heterogeneous remote sensing images.

[0111] The present invention provides a method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution, which can realize accurate detection of change areas of building areas in heterogeneous remote sensing image data of synthetic aperture radar (SAR) and optical images. It includes: obtaining visible light and remote sensing images of the same area at different times, using the Labelme_cd tool to mark the areas where building changes have occurred, establishing a heterogeneous remote sensing image change detection dataset, performing data augmentation on the dataset, and generating training and test sets; constructing a heterogeneous change detection network, using a pseudo-twin encoder to independently extract the unique representations of the two source images, and designing a single decoder with layer-by-layer residual connections to fuse the two source representations to generate a refined change area prediction mask; constructing a structured mutual distillation module, performing knowledge mutual distillation on the structured representations of the two source images that are robust to modal differences, realizing cross-modal alignment of SAR and visible light heterogeneous images, solving the interference of modal differences on the change detection process, and improving the change detection accuracy; constructing a difference dynamics and evolution extraction module, modeling the mutual transformation correlation between visible light and SAR image features as a time domain evolution process, based on the distinguishing features of the pseudo changes introduced by the modality and the real change areas represented in the time domain evolution process, effectively extracting the change-related difference feature information from various types of complex coupled difference information, and guiding the network to reconstruct a more accurate change area mask. The present invention can adapt to the huge modal differences of heterogeneous remote sensing data and realize accurate detection of building change areas. Compared with the method that requires homologous data, it has stronger generalization ability and is easy to promote and use.

[0112] refer to Figure 5 The method of the present invention can effectively realize the cross-modal alignment of heterogeneous features, and accurately extract the difference representation related to the real change from the complex coupled modal differences and change differences, so as to realize the accurate detection of building changes. The effect of the method of the present invention is further illustrated by the following simulation experiments.

[0113] 1. Simulation conditions.

[0114] The method of the present invention is a simulation performed by using Anaconda software on a central processing unit of Intel Core i7-9750H CPU, a memory of 32G, a graphics card of Nvidia RTX3090, and a WINDOWS10 operating system.

[0115] 2. Simulation content.

[0116] The data used in the simulation are 2360 SAR and optical heterogeneous remote sensing images with a size of 52\times512, describing the architectural changes in the shooting area.

[0117] In order to prove the effectiveness of the method of the invention, the method of the invention is executed on the above image dataset and compared with 14 different current most advanced methods. The comparison results are measured by using 4 different evaluation indicators. The results are shown in Table 1.

[0118] As can be seen from Table 1, the damage assessment results of the present invention can achieve the best performance. Applying the present invention to fields such as disaster assessment will effectively improve execution efficiency and have more accurate detection results.

[0119] Table 1

[0120]

[0121] It should be noted that, as another aspect, the present application also provides a storage medium, which may be included in an electronic device; or may exist independently without being assembled into the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0122] In one embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0123] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0124] Other embodiments of the invention will readily occur to those skilled in the art after considering the specification and practicing the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0125] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution, characterized in that: The method comprises: Obtain visible light and SAR remote sensing images of the same area at different times, mark the areas where building changes occur, establish a heterogeneous remote sensing image change detection dataset, and divide the dataset into a training set and a test set; A pre-trained heterogeneous change detection network is constructed, wherein the heterogeneous change detection network includes a pseudo-twin encoder and a decoder. The two branches of the pseudo-twin encoder have the same structure but do not share parameters, respectively extracting features of heterogeneous images and gradually reducing their dimensions; each layer of the pseudo-twin encoder includes a structured mutual distillation module for realizing cross-modal feature alignment, and a differential dynamic evolution extraction module for discriminatively extracting relevant representations of real change regions; the differential dynamic evolution extraction module inputs the extracted differential representations related to the real change regions into the layers of corresponding sizes in the decoder through skip connections, so as to guide the decoder to generate more accurate and fine-grained prediction masks of change regions; The heterogeneous change detection network is trained using the training set, and the trained heterogeneous change detection network is tested using the test set to obtain the heterogeneous image data building change detection results.

2. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 1, characterized in that: The method further includes: performing data augmentation on the training set, wherein the data augmentation includes cropping, flipping and shifting.

3. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 1 or 2, characterized in that: The structured mutual distillation module specifically includes the following processing steps: The heterogeneous features output by the pseudo-twin encoder are cut into blocks, and the cut heterogeneous features are independently Fourier transformed and high-pass filtered in the image blocks. Then, the inverse Fourier transform is performed and the image blocks are spliced ​​back to the original input feature distribution to output the independent structured representation of the heterogeneous features. Each element of the structured representation is mapped to a node, and the positional relationship between different elements is mapped to the edges connecting the nodes, a grid graph is constructed, and the structured representation is smoothed through a two-layer graph convolutional neural network; The smoothed structured representation is masked using a mutual attention mechanism with transposed matrix multiplication. Perform knowledge mutual distillation on the masked structured representation.

4. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 3 is characterized in that: A loss function combining weighted and weight-increasing factors is used to perform knowledge mutual distillation on the masked structured representation.

5. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 1, characterized in that: The differential dynamic evolution extraction module specifically includes the following processing steps: The intermediate transition feature from visible light feature to SAR feature is obtained by convolving the absolute value of the difference between the visible light feature and the SAR feature with the visible light feature, feature difference, and SAR feature in order; The visible light features, intermediate transition features, and SAR features are reorganized according to the time series conversion angle to obtain the time domain evolution features; The 3D convolution is used to capture the overall difference representation of the time domain evolution features in the time domain, including the modal differences in the unchanged area and the change differences in the changed area; Perform time-domain average pooling on the time-domain evolution features, and effectively distinguish them based on the distinguishing features of modal differences and change differences in the time-domain pooling process; The convolution kernel of the used 3D convolution is reconstructed, and the 3D convolution after convolution kernel reconstruction is used to perform difference extraction on the time domain pooling features to obtain the modal difference representation; The overall difference representation is subtracted from the modal difference representation to obtain the difference representation related only to the changed area, and the 3D convolution is used to further refine and optimize the difference representation related to the changed area.

6. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 1, characterized in that: The use of the training set to train the heterogeneous change detection network specifically includes: selecting AdamW as an optimizer, and using structured mutual distillation loss and change loss to optimize the network model.

7. The method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to claim 6, characterized in that: The network model is optimized by using structured mutual distillation loss and variation loss, including: Construct MSE loss as the loss function to supervise the full interaction of structured representation knowledge of each layer of encoder, and fuse the mutual distillation loss L corresponding to each layer of encoder in a weighted manner smd : Among them, y i and Represent the true value and predicted value of the i-th pixel respectively, N is the total number of encoder layers, and λ w is the weight factor of each layer of knowledge distillation loss; Combining DICE loss and BCE loss to construct the change loss L cd , to solve the class imbalance problem between the changing and unchanged regions in heterogeneous data: L cd =L dice +λ1*L bce Among them, λ1 is a hyperparameter; By combining structural inter-distillation loss and change loss, the network model is guided to accurately predict the change area.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution as described in any one of claims 1 to 7 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting building changes in heterogeneous remote sensing images based on mutual distillation and differential evolution according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions for the processor; Wherein, the processor is configured to execute the heterogeneous remote sensing image building change detection method based on mutual distillation and differential evolution as described in any one of claims 1 to 7 by executing the executable instructions.

Images