Change detection method and device based on SAR (Synthetic Aperture Radar) and optical image collaborative learning

Abstract

The invention provides a change detection method and device based on SAR and optical image collaborative learning, and relates to the technical field of remote sensing image processing. The method comprises the following steps: acquiring a pair of SAR images of front and back time phases and a pair of optical images of front and back time phases; constructing a change detection network based on SAR and optical image collaborative learning; performing feature extraction on the SAR image and the optical image based on a CNN feature extraction encoder to obtain SAR image features and optical image features; the SAR image features and the optical image features are enhanced by a feature enhancement module based on Transform, and the enhanced SAR image features and the enhanced optical image features are obtained; fusing the two enhanced features to obtain a fused feature; calculating a total loss function of the network through changing a truth value label and the obtained probability distribution after feature decoding; and training according to the total loss function to obtain a trained change detection network. According to the invention, the change detection interpretation effect can be enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a change detection method and device based on collaborative learning of SAR and optical images. Background Art

[0002] Remote sensing change detection technology is widely used in disaster loss assessment, urban dynamic monitoring, illegal construction screening, and surface cover analysis. Due to differences in the imaging mechanisms of different remote sensing data, each type of data has different characteristics and has its own advantages and disadvantages in image interpretation. For example, optical sensors have the advantage of high spatial resolution, but due to the influence of "same spectrum, different objects" or "same object, different spectrum", two different objects may be identified as the same in optical images. SAR imagery, through the microwave backscattering mechanism, has the ability to resolve spectral response confusion in optical images. However, compared to optical images, SAR images have less spectral information and are more susceptible to coherent speckle noise. Therefore, integrating the complementary characteristics of SAR and optical multi-source remote sensing data, breaking through the detection limitations of a single data type, and combining the detail recognition capabilities of optical images with the resolution capabilities of SAR for objects with spectral response confusion can enhance the effectiveness of change detection and interpretation.

[0003] However, in practical applications, optical imaging conditions are limited by cloud cover and darkness, meaning effective data can only be acquired during clear daylight hours. This limitation restricts the breadth of optical imaging applications. In particular, extreme weather conditions hinder timely observation of ground targets. SAR imaging, on the other hand, is unaffected by weather and darkness and can acquire data around the clock. Traditional explicit fusion methods require complete bimodal data for both training and inference, limiting their scope of application. Summary of the Invention

[0004] To address the technical issues in existing technologies that are limited by the availability of optical images under weather and time conditions, the present invention provides a change detection method and device based on collaborative learning of SAR and optical images. The technical solution is as follows:

[0005] In one aspect, a change detection method based on collaborative learning of SAR and optical images is provided. The method is implemented by a change detection device based on collaborative learning of SAR and optical images. The method includes:

[0006] S1. Acquire a pair of SAR images in the preceding and following time phases and a pair of optical images in the preceding and following time phases; construct a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0007] S2. Inputting the SAR image into a first CNN-based feature extraction encoder to perform feature extraction to obtain SAR image features; inputting the optical image into a second CNN-based feature extraction encoder to perform feature extraction to obtain optical image features;

[0008] S3. Inputting the SAR image features into a first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; inputting the enhanced SAR image features into a first decoder for decoding to obtain a SAR image change detection prediction probability map; inputting the optical image features into a second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; inputting the enhanced optical image features into a second decoder for decoding to obtain an optical image change detection prediction probability map;

[0009] S4, inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features; inputting the fusion features into a third decoder for decoding to obtain a change detection prediction probability map of the fusion information;

[0010] S5. Construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map; train the network based on the total loss function of the network to obtain a trained change detection network; obtain a remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results for multiple types of land objects in the remote sensing image.

[0011] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features includes:

[0012] Inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module to fuse them by feature cascade or addition to obtain fusion features; or

[0013] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused based on the cross-attention method to obtain the fused features.

[0014] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion, and fusing them based on a cross-attention method to obtain fusion features, includes:

[0015] Using the optical image feature as a query and the SAR image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a first linear transformation result; using the SAR image feature as a query and the optical image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a second linear transformation result;

[0016] Calculating a cross-attention result of the optical modality to the SAR modality based on the first linear transformation result; and calculating a cross-attention result of the SAR modality to the optical modality based on the second linear transformation result;

[0017] The cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality are fused to obtain the fusion features.

[0018] Optionally, the step of calculating the total loss function of the network according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map includes:

[0019] S51, comparing the optical image change detection prediction probability map with the true value label of the optical image change detection, and calculating the supervision loss function of the optical modality;

[0020] Among them, the supervision loss function of the optical modality is expressed by the following formula (1):

[0021] (1)

[0022] in, Represents the supervised loss function of the optical modality; Indicates the Predicting the probability of optical image changes of the sample; Indicates the The changing ground truth labels for the sample pairs; Represents the number of sample pairs in the training batch;

[0023] S52, comparing the SAR image change detection prediction probability map with the SAR image change detection true value label, and calculating the supervision loss function of the SAR modality;

[0024] Among them, the supervision loss function of the SAR modality is expressed by the following formula (2):

[0025] (2)

[0026] in, Represents the supervised loss function of the optical modality; Indicates the Prediction probability of SAR image changes for samples;

[0027] S53, comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the SAR modality relative to the optical modality; comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the optical modality relative to the SAR modality;

[0028] S54, comparing the optical image change detection prediction probability map with the fusion information change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the optical modality relative to the fusion information; comparing the SAR image change detection prediction probability map with the fusion information change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the SAR modality relative to the fusion information;

[0029] S55. Constructing a total loss function of the optical modality according to the supervision loss function of the optical modality, the collaborative loss function of the optical modality relative to the SAR modality, and the collaborative loss function of the optical modality relative to the fusion information;

[0030] S56. Construct a total loss function of the SAR modality based on the supervision loss function of the SAR modality, the collaborative loss function of the SAR modality relative to the optical modality, and the collaborative loss function of the SAR modality relative to the fusion information.

[0031] Optionally, the synergy loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0032] (3)

[0033] in, represents the collaborative loss function of the SAR mode relative to the optical mode;

[0034] The synergy loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0035] (4)

[0036] in, represents the synergy loss function of the optical mode relative to the SAR mode.

[0037] Optionally, the collaborative loss function of the optical modality relative fusion information is expressed by the following formula (5):

[0038] (5)

[0039] in, A collaborative loss function representing the relative fusion information of optical modalities; Indicates the Predict the probability of changes in sample fusion branches;

[0040] The collaborative loss function of the relative fusion information of the SAR modalities is expressed by the following formula (6):

[0041] (6)

[0042] in, The collaborative loss function representing the relative fusion information of SAR modalities.

[0043] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0044] (7)

[0045] in, represents the total loss function of the optical mode; represents the first weight coefficient of the optical mode; The second weight coefficient representing the optical mode; represents the third weight coefficient of the optical mode;

[0046] The total loss function of the SAR modality is expressed by the following formula (8):

[0047] (8)

[0048] in, represents the total loss function of the SAR modality; Represents the first weight coefficient of the SAR mode; Represents the second weight coefficient of the SAR mode; Represents the third weight coefficient of the SAR mode.

[0049] On the other hand, a change detection device based on collaborative learning of SAR and optical images is provided. The device is applied to a change detection method based on collaborative learning of SAR and optical images. The device includes:

[0050] An acquisition and construction unit is configured to acquire a pair of SAR images in the preceding and following time phases and a pair of optical images in the preceding and following time phases; and to construct a change detection network based on collaborative learning of SAR and optical images; the network comprises: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0051] A decoding unit is configured to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features;

[0052] The enhancement unit is configured to input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map;

[0053] A fusion unit is configured to input the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features; and input the fusion features into a third decoder for decoding to obtain a change detection prediction probability map of the fusion information;

[0054] The training unit is used to construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fusion information change detection prediction probability map and the obtained change true value label; train the network according to the total loss function of the network to obtain a trained change detection network; obtain a remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results for multiple types of ground objects in the remote sensing image.

[0055] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features includes:

[0056] Inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module to fuse them by feature cascade or addition to obtain fusion features; or

[0057] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused based on the cross-attention method to obtain the fused features.

[0058] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion, and fusing them based on a cross-attention method to obtain fusion features, includes:

[0059] Using the optical image feature as a query and the SAR image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a first linear transformation result; using the SAR image feature as a query and the optical image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a second linear transformation result;

[0060] Calculating a cross-attention result of the optical modality to the SAR modality based on the first linear transformation result; and calculating a cross-attention result of the SAR modality to the optical modality based on the second linear transformation result;

[0061] The cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality are fused to obtain the fusion features.

[0062] Optionally, the training unit is used to:

[0063] Compare the predicted probability map of optical image change detection with the true value label of optical image change detection to calculate the supervised loss function of the optical modality;

[0064] Among them, the supervision loss function of the optical modality is expressed by the following formula (1):

[0065] (1)

[0066] in, Represents the supervised loss function of the optical modality; Indicates the Predicting the probability of optical image changes of the sample; Indicates the The changing ground truth labels for the sample pairs; Represents the number of sample pairs in the training batch;

[0067] Compare the SAR image change detection prediction probability map with the SAR image change detection ground truth label and calculate the supervised loss function of the SAR modality;

[0068] Among them, the supervision loss function of the SAR modality is expressed by the following formula (2):

[0069] (2)

[0070] in, Represents the supervised loss function of the optical modality; Indicates the Prediction probability of SAR image changes for samples;

[0071] The predicted probability map of optical image change detection is compared with the predicted probability map of SAR image change detection, and the KL divergence is used to calculate the collaborative loss function of the SAR modality relative to the optical modality; the predicted probability map of optical image change detection is compared with the predicted probability map of SAR image change detection, and the KL divergence is used to calculate the collaborative loss function of the optical modality relative to the SAR modality;

[0072] The optical image change detection prediction probability map is compared with the change detection prediction probability map of the fusion information, and the KL divergence is used to calculate the collaborative loss function of the optical modality relative to the fusion information; the SAR image change detection prediction probability map is compared with the change detection prediction probability map of the fusion information, and the KL divergence is used to calculate the collaborative loss function of the SAR modality relative to the fusion information;

[0073] The total loss function of the optical modality is constructed based on the supervised loss function of the optical modality, the collaborative loss function of the optical modality relative to the SAR modality, and the collaborative loss function of the optical modality relative to the fusion information.

[0074] According to the supervision loss function of the SAR modality, the collaborative loss function of the SAR modality relative to the optical modality, and the collaborative loss function of the SAR modality relative to the fusion information, the total loss function of the SAR modality is constructed.

[0075] Optionally, the synergy loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0076] (3)

[0077] in, represents the collaborative loss function of the SAR mode relative to the optical mode;

[0078] The synergy loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0079] (4)

[0080] in, represents the synergy loss function of the optical mode relative to the SAR mode.

[0081] Optionally, the collaborative loss function of the optical modality relative fusion information is expressed by the following formula (5):

[0082] (5)

[0083] in, A collaborative loss function representing the relative fusion information of optical modalities; Indicates the Predict the probability of changes in sample fusion branches;

[0084] The collaborative loss function of the relative fusion information of the SAR modalities is expressed by the following formula (6):

[0085] (6)

[0086] in, The collaborative loss function representing the relative fusion information of SAR modalities.

[0087] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0088] (7)

[0089] in, represents the total loss function of the optical mode; represents the first weight coefficient of the optical mode; The second weight coefficient representing the optical mode; represents the third weight coefficient of the optical mode;

[0090] The total loss function of the SAR modality is expressed by the following formula (8):

[0091] (8)

[0092] in, represents the total loss function of the SAR modality; Represents the first weight coefficient of the SAR mode; Represents the second weight coefficient of the SAR mode; Represents the third weight coefficient of the SAR mode.

[0093] On the other hand, a change detection device based on collaborative learning of SAR and optical images is provided, and the change detection device based on collaborative learning of SAR and optical images includes: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned change detection methods based on collaborative learning of SAR and optical images is implemented.

[0094] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned change detection methods based on collaborative learning of SAR and optical images.

[0095] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0096] The embodiment of the present invention first obtains a pair of SAR images of the preceding and following time phases and a pair of optical images of the preceding and following time phases; constructs a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module and three decoders; the SAR image is input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical image is input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module ... The enhanced optical image features are enhanced in the feature enhancement module of the ansformer to obtain enhanced optical image features; the enhanced optical image features are input into the second decoder for decoding to obtain an optical image change detection prediction probability map; secondly, the enhanced SAR image features and the enhanced optical image features are input into the feature fusion module for fusion to obtain fused features; the fused features are input into the third decoder for decoding to obtain a change detection prediction probability map of the fused information; according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map and the change detection prediction probability map of the fused information, a total loss function of the network is constructed; the network is trained according to the total loss function of the network to obtain a trained change detection network; finally, the remote sensing image to be detected is obtained; the remote sensing image to be detected is input into the trained change detection network to obtain change detection results of various types of land objects in the remote sensing image.

[0097] The embodiments of the present invention can be used on single SAR modality data and are no longer restricted by the availability of optical images under weather and time conditions. The use of the embodiments of the present invention can enhance the change detection interpretation effect and improve the accuracy of the SAR change detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0099] Figure 1 This is a flow chart of a change detection method based on collaborative learning of SAR and optical images provided by an embodiment of the present invention;

[0100] Figure 2Schematic diagram of a change detection method based on collaborative learning of SAR and optical images provided by an embodiment of the present invention;

[0101] Figure 3 This is a block diagram of a change detection device based on collaborative learning of SAR and optical images provided by an embodiment of the present invention;

[0102] Figure 4 This is a structural diagram of a change detection device based on collaborative learning of SAR and optical images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0103] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0104] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0105] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0106] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0107] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0108] The embodiment of the present invention provides a change detection method based on collaborative learning of SAR and optical images. The method can be implemented by a change detection device based on collaborative learning of SAR and optical images. The change detection device based on collaborative learning of SAR and optical images can be a terminal or a server. Figure 1 The flowchart of the change detection method based on collaborative learning of SAR and optical images is shown. The processing flow of this method may include the following steps:

[0109] S1. Acquire a pair of SAR images and a pair of optical images in the preceding and following time phases; construct a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders.

[0110] S2. Input the SAR image into a first CNN-based feature extraction encoder to perform feature extraction to obtain SAR image features; input the optical image into a second CNN-based feature extraction encoder to perform feature extraction to obtain optical image features.

[0111] S3. Input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map.

[0112] S4. Input the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fusion features; input the fusion features into the third decoder for decoding to obtain a change detection prediction probability map of the fusion information.

[0113] Optionally, the enhanced SAR image features and the enhanced optical image features are input into a feature fusion module for fusion to obtain fusion features, including:

[0114] Inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module to fuse them by feature cascade or addition to obtain fusion features; or

[0115] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused based on the cross-attention method to obtain the fused features.

[0116] The implementation process of feature cascade is expressed by the following formula (1):

[0117] (1)

[0118] in, represents fusion features; represents the enhanced optical image features; Indicates the enhanced SAR image features;

[0119] The fusion process by adding features is expressed by the following formula (2):

[0120] (2)

[0121] Optionally, the enhanced SAR image features and the enhanced optical image features are input into a feature fusion module for fusion, and fusion is performed based on a cross-attention method to obtain fusion features, including:

[0122] Using the optical image feature as a query and the SAR image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a first linear transformation result; using the SAR image feature as a query and the optical image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a second linear transformation result;

[0123] In a feasible implementation, the process of linear transformation of the enhanced SAR image features is expressed by the following formulas (3)-(5):

[0124] (3)

[0125] (4)

[0126] (5)

[0127] in, It represents the linear transformation of the enhanced optical image features by taking the optical image features as the query vector; It means that the SAR image feature is used as the key vector, which is the linear transformation of the enhanced SAR image feature; It represents the linear transformation of the enhanced SAR image features by taking the SAR image features as the value vector; represents the first learnable weight matrix; represents the second learnable weight matrix; represents the learnable third weight matrix.

[0128] In a feasible implementation, the process of linearly transforming the enhanced optical image features is expressed by the following formulas (6)-(8):

[0129] (6)

[0130] (7)

[0131] (8)

[0132] in, It represents the linear transformation of the SAR image feature query vector to the enhanced SAR image feature; It represents the linear transformation of the enhanced optical image feature by taking the optical image feature as the key vector; It represents the linear transformation of the optical image feature vector into the enhanced optical image feature; represents the fourth learnable weight matrix; represents the learnable fifth weight matrix; Represents the learnable sixth weight matrix.

[0133] Calculating a cross-attention result of the optical modality to the SAR modality based on the first linear transformation result; and calculating a cross-attention result of the SAR modality to the optical modality based on the second linear transformation result;

[0134] In a feasible implementation, the cross attention result of the optical modality to the SAR modality is calculated and expressed by the following formula (9):

[0135] (9)

[0136] in, represents the cross attention result of optical modality to SAR modality; Indicates the dimension of the value vector.

[0137] In a feasible implementation, the cross attention result of the SAR modality to the optical modality is calculated and expressed by the following formula (10):

[0138] (10)

[0139] in, Represents the cross attention results of SAR modality to optical modality.

[0140] The cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality are fused to obtain the fusion features.

[0141] In one feasible implementation, the process of adding the cross-attention results of the optical modality to the SAR modality and the cross-attention results of the SAR modality to the optical modality is expressed by the following formula (11):

[0142] (11)

[0143] In a feasible implementation, the process of fusion using feature cascade is expressed by the following formula (12) based on the cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality:

[0144] (12)

[0145] S5. Construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fusion information change detection prediction probability map, and the change true value label; train the network based on the total loss function to obtain a trained change detection network; obtain the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain the change detection result of the remote sensing image.

[0146] Among them, according to the type of true value labels used in training, the change detection results of remote sensing images can include: water body changes, building changes, road changes and other infrastructure changes or a combination of the above changes.

[0147] Among them, the change truth label is the change truth label shared by SAR images and optical images.

[0148] Among them, the change true value labels are obtained through manual annotation.

[0149] Among them, the back propagation method is used to update the network parameters through the Adam optimizer or the SGD optimizer, and the process of S2-S5 is repeated until the model converges to obtain a trained change detection network.

[0150] Optionally, calculating the total loss function of the network according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map in S5 includes:

[0151] S51, comparing the optical image change detection prediction probability map with the true value label of the optical image change detection, and calculating the supervision loss function of the optical modality;

[0152] Among them, the supervision loss function of the optical modality is expressed by the following formula (13):

[0153] (13)

[0154] in, Represents the supervised loss function of the optical modality; Indicates the Predicting the probability of optical image changes of the sample; Indicates the The changing ground truth labels for the sample pairs; Represents the number of sample pairs in the training batch;

[0155] S52, comparing the SAR image change detection prediction probability map with the SAR image change detection true value label, and calculating the supervision loss function of the SAR modality;

[0156] Among them, the supervision loss function of the SAR modality is expressed by the following formula (14):

[0157] (14)

[0158] in, Represents the supervised loss function of the optical modality; Indicates the Prediction probability of SAR image changes for samples;

[0159] S53, comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the SAR modality relative to the optical modality; comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the optical modality relative to the SAR modality;

[0160] Optionally, the synergy loss function of the SAR mode relative to the optical mode is expressed by the following formula (15):

[0161] (15)

[0162] in, represents the collaborative loss function of the SAR mode relative to the optical mode;

[0163] The synergy loss function of the optical mode relative to the SAR mode is expressed by the following formula (16):

[0164] (16)

[0165] in, represents the synergy loss function of the optical mode relative to the SAR mode.

[0166] S54, comparing the optical image change detection prediction probability map with the fusion information change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the optical modality relative to the fusion information; comparing the SAR image change detection prediction probability map with the fusion information change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the SAR modality relative to the fusion information;

[0167] Optionally, the collaborative loss function of the relative fusion information of the optical modalities is expressed by the following formula (17):

[0168] (17)

[0169] in, A collaborative loss function representing the relative fusion information of optical modalities; Indicates the Predict the probability of changes in sample fusion branches;

[0170] The collaborative loss function of the relative fusion information of SAR modalities is expressed by the following formula (18):

[0171] (18)

[0172] in, The collaborative loss function representing the relative fusion information of SAR modalities.

[0173] S55. Constructing a total loss function of the optical modality according to the supervision loss function of the optical modality, the collaborative loss function of the optical modality relative to the SAR modality, and the collaborative loss function of the optical modality relative to the fusion information;

[0174] S56. Construct a total loss function of the SAR modality based on the supervision loss function of the SAR modality, the collaborative loss function of the SAR modality relative to the optical modality, and the collaborative loss function of the SAR modality relative to the fusion information.

[0175] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0176] (7)

[0177] in, represents the total loss function of the optical mode; represents the first weight coefficient of the optical mode; The second weight coefficient representing the optical mode; represents the third weight coefficient of the optical mode;

[0178] The total loss function of the SAR mode is expressed by the following formula (8):

[0179] (8)

[0180] in, represents the total loss function of the SAR modality; Represents the first weight coefficient of the SAR mode; Represents the second weight coefficient of the SAR mode; Represents the third weight coefficient of the SAR mode.

[0181] Among them, such as Figure 2It is a structural diagram of a change detection method based on collaborative learning of SAR and optical images provided by an embodiment of the present invention; in a feasible implementation manner, a change detection network based on collaborative learning of SAR and optical images is constructed; a pair of SAR images of the preceding and following time phases and a pair of optical images of the preceding and following time phases are obtained; the pair of SAR images of the preceding and following time phases are input into a SAR image feature encoder for processing to obtain SAR image features; the pair of optical images of the preceding and following time phases are input into an optical image feature encoder for processing to obtain optical image features; the SAR image features are input into a feature enhancement module based on Transformer to obtain enhanced SAR image features; the optical image features are input into a feature enhancement module based on Transformer to obtain enhanced optical image features; the enhanced SAR image features and the enhanced optical image features are input into a feature enhancement module based on Transformer to obtain enhanced optical image features; The enhanced optical image features are fused to obtain fused features; the fused features are input into the fused feature change detection decoder to obtain a fused probability output layer; the enhanced SAR image features are input into the SAR image change detection decoder to obtain a SAR image change probability output layer; the enhanced optical image features are input into the optical image change detection decoder to obtain an optical image change probability output layer; a total loss function is constructed according to the fused probability output layer, the SAR image change probability output layer and the true value label of the optical image change probability output layer; wherein the total loss function includes: the total loss function of the optical mode and the total loss function of the SAR mode; the total loss function is used to train the change detection network based on the collaborative learning of SAR and optical images to obtain a trained change detection network based on the collaborative learning of SAR and optical images.

[0182] The embodiment of the present invention first obtains a pair of SAR images of the preceding and following time phases and a pair of optical images of the preceding and following time phases; constructs a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module and three decoders; the SAR image is input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical image is input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module for decoding to obtain a SAR image change detection prediction probability map; The optical image features are enhanced in the feature enhancement module of the decoder to obtain enhanced optical image features; the enhanced optical image features are input into the second decoder for decoding to obtain an optical image change detection prediction probability map; secondly, the enhanced SAR image features and the enhanced optical image features are input into the feature fusion module for fusion to obtain fused features; the fused features are input into the third decoder for decoding to obtain a change detection prediction probability map of the fused information; a total loss function of the network is constructed according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the change detection prediction probability map of the fused information and the change true value label; the network is trained according to the total loss function of the network to obtain a trained change detection network; finally, the remote sensing image to be detected is obtained; the remote sensing image to be detected is input into the trained change detection network to obtain change detection results of various types of land objects in the remote sensing image.

[0183] The embodiments of the present invention can be used on single SAR modality data and are no longer restricted by the availability of optical images under weather and time conditions. The use of the embodiments of the present invention can enhance the change detection interpretation effect and improve the accuracy of the SAR change detection network.

[0184] Figure 3 This is a block diagram of a change detection device based on collaborative learning of SAR and optical images according to an exemplary embodiment. The device is used in a change detection method based on collaborative learning of SAR and optical images. Figure 3 The device includes an acquisition and construction unit 310, a decoding unit 320, an enhancement unit 330, a fusion unit 340, and a training unit 350.

[0185] An acquisition and construction unit 310 is configured to acquire a pair of SAR images in the preceding and following time phases and a pair of optical images in the preceding and following time phases; and to construct a change detection network based on collaborative learning of SAR and optical images; the network comprising: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0186] The decoding unit 320 is configured to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features;

[0187] The enhancement unit 330 is configured to input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map;

[0188] The fusion unit 340 is configured to input the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features; and input the fused features into the third decoder for decoding to obtain a change detection prediction probability map of the fused information;

[0189] The training unit 350 is configured to construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map; train the network based on the total loss function of the network to obtain a trained change detection network; obtain a remote sensing image to be detected; and input the remote sensing image to be detected into the trained change detection network to obtain change detection results for multiple types of land objects in the remote sensing image.

[0190] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features includes:

[0191] Inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module to fuse them by feature cascade or addition to obtain fusion features; or

[0192] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused based on the cross-attention method to obtain the fused features.

[0193] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion, and fusing them based on a cross-attention method to obtain fusion features, includes:

[0194] Using the optical image feature as a query and the SAR image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a first linear transformation result; using the SAR image feature as a query and the optical image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a second linear transformation result;

[0195] Calculating a cross-attention result of the optical modality to the SAR modality based on the first linear transformation result; and calculating a cross-attention result of the SAR modality to the optical modality based on the second linear transformation result;

[0196] The cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality are fused to obtain the fusion features.

[0197] Optionally, the training unit 350 is configured to:

[0198] Compare the predicted probability map of optical image change detection with the true value label of optical image change detection to calculate the supervised loss function of the optical modality;

[0199] Among them, the supervision loss function of the optical modality is expressed by the following formula (1):

[0200] (1)

[0201] in, Represents the supervised loss function of the optical modality; Indicates the Predicting the probability of optical image changes of the sample; Indicates the The changing ground truth labels for the sample pairs; Represents the number of sample pairs in the training batch;

[0202] Compare the SAR image change detection prediction probability map with the SAR image change detection ground truth label and calculate the supervised loss function of the SAR modality;

[0203] Among them, the supervision loss function of the SAR modality is expressed by the following formula (2):

[0204] (2)

[0205] in, Represents the supervised loss function of the optical modality; Indicates the Prediction probability of SAR image changes for samples;

[0206] The predicted probability map of optical image change detection is compared with the predicted probability map of SAR image change detection, and the KL divergence is used to calculate the collaborative loss function of the SAR modality relative to the optical modality; the predicted probability map of optical image change detection is compared with the predicted probability map of SAR image change detection, and the KL divergence is used to calculate the collaborative loss function of the optical modality relative to the SAR modality;

[0207] The optical image change detection prediction probability map is compared with the change detection prediction probability map of the fusion information, and the KL divergence is used to calculate the collaborative loss function of the optical modality relative to the fusion information; the SAR image change detection prediction probability map is compared with the change detection prediction probability map of the fusion information, and the KL divergence is used to calculate the collaborative loss function of the SAR modality relative to the fusion information;

[0208] The total loss function of the optical modality is constructed based on the supervised loss function of the optical modality, the collaborative loss function of the optical modality relative to the SAR modality, and the collaborative loss function of the optical modality relative to the fusion information.

[0209] According to the supervision loss function of the SAR modality, the collaborative loss function of the SAR modality relative to the optical modality, and the collaborative loss function of the SAR modality relative to the fusion information, the total loss function of the SAR modality is constructed.

[0210] Optionally, the synergy loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0211] (3)

[0212] in, represents the collaborative loss function of the SAR mode relative to the optical mode;

[0213] The synergy loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0214] (4)

[0215] in, represents the synergy loss function of the optical mode relative to the SAR mode.

[0216] Optionally, the collaborative loss function of the optical modality relative fusion information is expressed by the following formula (5):

[0217] (5)

[0218] in, A collaborative loss function representing the relative fusion information of optical modalities; Indicates the Predict the probability of changes in sample fusion branches;

[0219] The collaborative loss function of the relative fusion information of the SAR modalities is expressed by the following formula (6):

[0220] (6)

[0221] in, The collaborative loss function representing the relative fusion information of SAR modalities.

[0222] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0223] (7)

[0224] in, represents the total loss function of the optical mode; represents the first weight coefficient of the optical mode; The second weight coefficient representing the optical mode; represents the third weight coefficient of the optical mode;

[0225] The total loss function of the SAR modality is expressed by the following formula (8):

[0226] (8)

[0227] in, represents the total loss function of the SAR modality; Represents the first weight coefficient of the SAR mode; Represents the second weight coefficient of the SAR mode; Represents the third weight coefficient of the SAR mode.

[0228] The embodiment of the present invention first obtains a pair of SAR images of the preceding and following time phases and a pair of optical images of the preceding and following time phases; constructs a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module and three decoders; the SAR image is input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical image is input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module for decoding to obtain a SAR image change detection prediction probability map; The optical image features are enhanced in the feature enhancement module of the decoder to obtain enhanced optical image features; the enhanced optical image features are input into the second decoder for decoding to obtain an optical image change detection prediction probability map; secondly, the enhanced SAR image features and the enhanced optical image features are input into the feature fusion module for fusion to obtain fused features; the fused features are input into the third decoder for decoding to obtain a change detection prediction probability map of the fused information; a total loss function of the network is constructed according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the change detection prediction probability map of the fused information and the change true value label; the network is trained according to the total loss function of the network to obtain a trained change detection network; finally, the remote sensing image to be detected is obtained; the remote sensing image to be detected is input into the trained change detection network to obtain change detection results of various types of land objects in the remote sensing image.

[0229] The embodiments of the present invention can be used on single SAR modality data and are no longer restricted by the availability of optical images under weather and time conditions. The use of the embodiments of the present invention can enhance the change detection interpretation effect and improve the accuracy of the SAR change detection network.

[0230] Figure 4 FIG is a schematic diagram of a change detection device based on collaborative learning of SAR and optical images provided by an embodiment of the present invention. Figure 4 As shown, the change detection device based on SAR and optical image collaborative learning may include the above Figure 3 Optionally, the change detection device 410 based on collaborative learning of SAR and optical images may include a first processor 2001 .

[0231] Optionally, the change detection device 410 based on collaborative learning of SAR and optical images may further include a memory 2002 and a transceiver 2003 .

[0232] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0233] The following combination Figure 4 The components of the change detection device 410 based on collaborative learning of SAR and optical images are described in detail:

[0234] The first processor 2001 is the control center of the change detection device 410 based on collaborative learning of SAR and optical images. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0235] Optionally, the first processor 2001 may execute various functions of the change detection device 410 based on collaborative learning of SAR and optical images by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0236] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0237] In a specific implementation, as an embodiment, the change detection device 410 based on collaborative learning of SAR and optical images may also include multiple processors, such as Figure 4 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0238] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0239] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0240] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0241] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0242] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the interface circuit of the change detection device 410 based on SAR and optical image collaborative learning ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0243] It should be noted that Figure 4 The structure of the change detection device 410 based on collaborative learning of SAR and optical images shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0244] In addition, the technical effects of the change detection device 410 based on collaborative learning of SAR and optical images can refer to the technical effects of the change detection method based on collaborative learning of SAR and optical images described in the above method embodiment, and will not be repeated here.

[0245] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0246] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0247] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0248] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0249] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0250] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0251] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0252] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0253] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0255] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0256] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0257] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A change detection method based on collaborative learning of SAR and optical images, characterized in that: The method comprises: S1. Acquire a pair of SAR images in the preceding and following time phases and a pair of optical images in the preceding and following time phases; construct a change detection network based on collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; S2. Inputting the SAR image into a first CNN-based feature extraction encoder to perform feature extraction to obtain SAR image features; inputting the optical image into a second CNN-based feature extraction encoder to perform feature extraction to obtain optical image features; S3. Inputting the SAR image features into a first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; inputting the enhanced SAR image features into a first decoder for decoding to obtain a SAR image change detection prediction probability map; inputting the optical image features into a second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; inputting the enhanced optical image features into a second decoder for decoding to obtain an optical image change detection prediction probability map; S4, inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features; inputting the fusion features into a third decoder for decoding to obtain a change detection prediction probability map of the fusion information; S5. Construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fusion information change detection prediction probability map, and the change true value label; train the network based on the total loss function of the network to obtain a trained change detection network; obtain a remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain a change detection result of the remote sensing image.

2. The change detection method based on collaborative learning of SAR and optical images according to claim 1, characterized in that: The step of inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features includes: Inputting the enhanced SAR image features and the enhanced optical image features into a feature fusion module, fusing them by feature concatenation or addition to obtain fused features; or The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused based on the cross-attention method to obtain the fused features.

3. The change detection method based on collaborative learning of SAR and optical images according to claim 2, characterized in that: The enhanced SAR image features and the enhanced optical image features are input into the feature fusion module for fusion, and the fusion features are obtained by fusion based on the cross attention method, including: Using the optical image feature as a query and the SAR image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a first linear transformation result; using the SAR image feature as a query and the optical image feature as a key and a value, a linear transformation is performed on the enhanced optical and SAR image features to obtain a second linear transformation result; Calculating a cross-attention result of the optical modality to the SAR modality based on the first linear transformation result; and calculating a cross-attention result of the SAR modality to the optical modality based on the second linear transformation result; The cross-attention results of the optical modality on the SAR modality and the cross-attention results of the SAR modality on the optical modality are fused to obtain the fusion features.

4. The change detection method based on collaborative learning of SAR and optical images according to claim 1, characterized in that: The step S5 calculates the total loss function of the network according to the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map, including: S51, comparing the optical image change detection prediction probability map with the true value label of the optical image change detection, and calculating the supervision loss function of the optical modality; Among them, the supervision loss function of the optical modality is expressed by the following formula (1): （1） in, Represents the supervised loss function of the optical modality; Indicates the Predicting the probability of optical image changes of the sample; Indicates the The changing ground truth labels for sample pairs; Represents the number of sample pairs in the training batch; S52, comparing the SAR image change detection prediction probability map with the SAR image change detection true value label, and calculating the supervision loss function of the SAR modality; Among them, the supervision loss function of the SAR modality is expressed by the following formula (2): （2） in, Represents the supervised loss function of the optical modality; Indicates the Prediction probability of SAR image changes for samples; S53, comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the SAR modality relative to the optical modality; comparing the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and using KL divergence to calculate the collaborative loss function of the optical modality relative to the SAR modality; S54, comparing the optical image change detection prediction probability map with the fusion information change detection prediction probability map, and calculating the collaborative loss function of the optical modality relative to the fusion information; comparing the SAR image change detection prediction probability map with the fusion information change detection prediction probability map, and calculating the collaborative loss function of the SAR modality relative to the fusion information; S55. Constructing a total loss function of the optical modality according to the supervision loss function of the optical modality, the collaborative loss function of the optical modality relative to the SAR modality, and the collaborative loss function of the optical modality relative to the fusion information; S56. Construct a total loss function of the SAR modality based on the supervision loss function of the SAR modality, the collaborative loss function of the SAR modality relative to the optical modality, and the collaborative loss function of the SAR modality relative to the fusion information.

5. The change detection method based on collaborative learning of SAR and optical images according to claim 4, characterized in that: The synergy loss function of the SAR mode relative to the optical mode is expressed by the following formula (3): （3） in, represents the collaborative loss function of the SAR mode relative to the optical mode; The synergy loss function of the optical mode relative to the SAR mode is expressed by the following formula (4): （4） in, represents the synergy loss function of the optical mode relative to the SAR mode.

6. The change detection method based on collaborative learning of SAR and optical images according to claim 4, characterized in that: The collaborative loss function of the relative fusion information of the optical modalities is expressed by the following formula (5): （5） in, A collaborative loss function representing the relative fusion information of optical modalities; Indicates the Predict the probability of changes in sample fusion branches; The collaborative loss function of the relative fusion information of the SAR modalities is expressed by the following formula (6): （6） in, The collaborative loss function representing the relative fusion information of SAR modalities.

7. The change detection method based on collaborative learning of SAR and optical images according to claim 4, characterized in that: The total loss function of the optical mode is expressed by the following formula (7): （7） in, represents the total loss function of the optical mode; represents the first weight coefficient of the optical mode; The second weight coefficient representing the optical mode; represents the third weight coefficient of the optical mode; The total loss function of the SAR modality is expressed by the following formula (8): （8） in, represents the total loss function of the SAR modality; Represents the first weight coefficient of the SAR mode; Represents the second weight coefficient of the SAR mode; Represents the third weight coefficient of the SAR mode.

8. A change detection device based on collaborative learning of SAR and optical images, wherein the change detection device based on collaborative learning of SAR and optical images is used to implement the change detection method based on collaborative learning of SAR and optical images as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition and construction unit is configured to acquire a pair of SAR images in the preceding and following time phases and a pair of optical images in the preceding and following time phases; and to construct a change detection network based on collaborative learning of SAR and optical images; the network comprises: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; A decoding unit is configured to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features; The enhancement unit is configured to input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map; A fusion unit is configured to input the enhanced SAR image features and the enhanced optical image features into a feature fusion module for fusion to obtain fusion features; and input the fusion features into a third decoder for decoding to obtain a change detection prediction probability map of the fusion information; The training unit is used to construct a total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fusion information change detection prediction probability map; train the network according to the total loss function of the network to obtain a trained change detection network; obtain a remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results for multiple types of ground objects in the remote sensing image.

9. A change detection device based on collaborative learning of SAR and optical images, characterized in that: The change detection device based on collaborative learning of SAR and optical images includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images