A change detection method and device based on SAR and optical image collaborative learning

Abstract

This invention provides a change detection method and apparatus based on SAR and optical image co-learning, relating to the field of remote sensing image processing technology. The method includes: acquiring a pair of consecutive SAR images and a pair of consecutive optical images; constructing a change detection network based on SAR and optical image co-learning; extracting features from the SAR and optical images respectively using a CNN-based feature extraction encoder to obtain SAR image features and optical image features; enhancing the SAR and optical image features using a Transformer-based feature enhancement module to obtain enhanced SAR image features and enhanced optical image features; fusing the two enhanced features to obtain fused features; calculating the total loss function of the network using change ground truth labels and the probability distribution after decoding the obtained features; and training the network based on the total loss function to obtain a trained change detection network. Using this invention can enhance the change detection interpretation effect.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a change detection method and apparatus based on SAR and optical image collaborative learning. Background Technology

[0002] Remote sensing change detection technology is widely used in disaster loss assessment, urban dynamic monitoring, illegal construction screening, and land cover analysis. Due to differences in the imaging mechanisms of various remote sensing data, different types of data exhibit different characteristics, each with its own advantages and disadvantages in image interpretation. For example, optical sensors have the advantage of high spatial resolution, but due to the influence of "different objects with the same spectrum" or "different spectra of the same object," two different objects may be identified as the same in optical images. SAR images, through the microwave backscattering mechanism, have the ability to distinguish spectral response confusion in optical images, but compared to optical images, SAR images have less spectral information and are more susceptible to speckle noise. Therefore, fusing the complementary characteristics of SAR and optical multi-source remote sensing data, overcoming the detection limitations of single data types, and combining the detail recognition capabilities of optical images with the ability of SAR to distinguish ground objects with spectral response confusion, can enhance the effectiveness of change detection interpretation.

[0003] However, in practical applications, optical imagery is limited by cloud cover and nighttime conditions, allowing for the acquisition of effective data only under clear daytime conditions. This limitation restricts the breadth of its applications. Especially in extreme weather conditions, timely observation of ground targets becomes impossible. SAR imaging, on the other hand, is unaffected by weather or nighttime, enabling all-weather data acquisition. Traditional explicit fusion methods require complete dual-modal data during both the training and inference phases, further restricting their application scope. Summary of the Invention

[0004] To address the technical problem of existing technologies being constrained by the availability of optical images under weather and time conditions, this invention provides a change detection method and apparatus based on SAR and optical image collaborative learning. The technical solution is as follows:

[0005] On the one hand, a change detection method based on SAR and optical image co-learning is provided. This method is implemented by a change detection device based on SAR and optical image co-learning, and includes:

[0006] S1. Acquire a pair of consecutive SAR images and a pair of consecutive optical images; construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0007] S2. Input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features;

[0008] S3. Input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map.

[0009] S4. Input the enhanced SAR image features and enhanced optical image features into the feature fusion module for fusion to obtain fused features; input the fused features into the third decoder for decoding to obtain the change detection and prediction probability map of the fused information;

[0010] S5. Construct the total loss function of the network based on the change detection prediction probability map of optical image, SAR image, and fused information; train the network according to the total loss function to obtain a trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results for various land cover types in the remote sensing image.

[0011] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features includes:

[0012] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module and fused through feature concatenation or addition to obtain the fused features; or

[0013] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused features are obtained by fusing them using a cross-attention method.

[0014] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion, and obtaining fused features by fusing them using a cross-attention method, includes:

[0015] Using optical image features as the query and SAR image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the first linear transformation result; using SAR image features as the query and optical image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the second linear transformation result.

[0016] Based on the first linear transformation result, calculate the cross-attention result of the optical mode on the SAR mode; based on the second linear transformation result, calculate the cross-attention result of the SAR mode on the optical mode.

[0017] The cross-attention results of optical modes on SAR modes and cross-attention results of SAR modes on optical modes are fused to obtain fused features.

[0018] Optionally, the step S5, which calculates the total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fused information change detection prediction probability map, includes:

[0019] S51. Compare the predicted probability map of optical image change detection with the ground truth label of optical image change detection, and calculate the supervised loss function of optical mode.

[0020] The supervised loss function for the optical mode is expressed by the following formula (1):

[0021] (1)

[0022] in, The supervision loss function represents the optical modes; Indicates the first Predict the probability of changes in the optical image of a sample; Indicates the first True value labels for changes in sample pairs; This represents the number of sample pairs in the training batch.

[0023] S52. Compare the SAR image change detection prediction probability map with the SAR image change detection ground truth label, and calculate the SAR mode supervision loss function;

[0024] The supervision loss function for the SAR mode is expressed by the following formula (2):

[0025] (2)

[0026] in, The supervision loss function represents the optical modes; Indicates the first Probability of SAR image change prediction for the sample;

[0027] S53. Compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the SAR mode relative to the optical mode; compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the optical mode relative to the SAR mode.

[0028] S54. Compare the optical image change detection prediction probability map with the fused information change detection prediction probability map, and use KL divergence to calculate the collaborative loss function of the optical mode relative to the fused information; compare the SAR image change detection prediction probability map with the fused information change detection prediction probability map, and use KL divergence to calculate the collaborative loss function of the SAR mode relative to the fused information.

[0029] S55. Construct the total loss function of the optical mode based on the supervision loss function of the optical mode, the cooperative loss function of the optical mode relative to the SAR mode, and the cooperative loss function of the optical mode relative to the fused information.

[0030] S56. Based on the supervision loss function of SAR mode, the cooperative loss function of SAR mode relative to optical mode, and the cooperative loss function of SAR mode relative to fused information, construct the total loss function of SAR mode.

[0031] Optionally, the cooperative loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0032] (3)

[0033] in, This represents the cooperative loss function of SAR modes relative to optical modes;

[0034] The cooperative loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0035] (4)

[0036] in, This represents the cooperative loss function of the optical mode relative to the SAR mode.

[0037] Optionally, the cooperative loss function of the optical modal relative fusion information is expressed by the following formula (5):

[0038] (5)

[0039] in, The collaborative loss function represents the relative fusion information of optical modes; Indicates the first Predict the probability of changes in the sample fusion branch;

[0040] The collaborative loss function of the SAR modal relative fusion information is expressed by the following formula (6):

[0041] (6)

[0042] in, This represents the collaborative loss function for the relative fusion of SAR modes.

[0043] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0044] (7)

[0045] in, Represents the total loss function of optical modes; The first weighting coefficient represents the optical mode; The second weighting coefficient represents the optical mode; The third weighting coefficient represents the optical mode.

[0046] The total loss function of the SAR mode is expressed by the following formula (8):

[0047] (8)

[0048] in, Represents the total loss function of SAR modes; This represents the first weighting coefficient of the SAR mode; This represents the second weighting coefficient of the SAR mode; This represents the third weighting coefficient of the SAR mode.

[0049] On the other hand, a change detection device based on SAR and optical image co-learning is provided. This device is applied to the change detection method based on SAR and optical image co-learning, and includes:

[0050] An acquisition and construction unit is used to acquire a pair of consecutive SAR images and a pair of consecutive optical images; and to construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0051] The decoding unit is used to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and to input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features.

[0052] An enhancement unit is used to input SAR image features into a first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into a first decoder for decoding to obtain a SAR image change detection prediction probability map; input optical image features into a second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into a second decoder for decoding to obtain an optical image change detection prediction probability map.

[0053] The fusion unit is used to input the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features; the fused features are then input into the third decoder for decoding to obtain a change detection and prediction probability map of the fused information.

[0054] The training unit is used to construct the total loss function of the network based on the change detection prediction probability map of optical imagery, the change detection prediction probability map of SAR imagery, the change detection prediction probability map of fused information, and the obtained change ground truth labels; train the network according to the total loss function to obtain a trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results for various land cover types in the remote sensing imagery.

[0055] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features includes:

[0056] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module and fused through feature concatenation or addition to obtain the fused features; or

[0057] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused features are obtained by fusing them using a cross-attention method.

[0058] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion, and obtaining fused features by fusing them using a cross-attention method, includes:

[0059] Using optical image features as the query and SAR image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the first linear transformation result; using SAR image features as the query and optical image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the second linear transformation result.

[0060] Based on the first linear transformation result, calculate the cross-attention result of the optical mode on the SAR mode; based on the second linear transformation result, calculate the cross-attention result of the SAR mode on the optical mode.

[0061] The cross-attention results of optical modes on SAR modes and cross-attention results of SAR modes on optical modes are fused to obtain fused features.

[0062] Optionally, the training unit is used for:

[0063] The predicted probability map of optical image change detection is compared with the ground truth label of optical image change detection, and the supervised loss function of optical mode is calculated.

[0064] The supervised loss function for the optical mode is expressed by the following formula (1):

[0065] (1)

[0066] in, The supervision loss function represents the optical modes; Indicates the first Predict the probability of changes in the optical image of a sample; Indicates the first True value labels for changes in sample pairs; This represents the number of sample pairs in the training batch.

[0067] The predicted probability map of SAR image change detection is compared with the ground truth label of SAR image change detection, and the supervision loss function of SAR mode is calculated.

[0068] The supervision loss function for the SAR mode is expressed by the following formula (2):

[0069] (2)

[0070] in, The supervision loss function represents the optical modes; Indicates the first Probability of SAR image change prediction for the sample;

[0071] The optical image change detection prediction probability map is compared with the SAR image change detection prediction probability map. KL divergence is used to calculate the cooperative loss function of the SAR mode relative to the optical mode.

[0072] The probability map for change detection prediction of optical image is compared with the probability map for change detection prediction of fused information. KL divergence is used to calculate the collaborative loss function of optical mode relative to fused information. The probability map for change detection prediction of SAR image is compared with the probability map for change detection prediction of fused information. KL divergence is used to calculate the collaborative loss function of SAR mode relative to fused information.

[0073] Based on the supervised loss function of the optical mode, the cooperative loss function of the optical mode relative to the SAR mode, and the cooperative loss function of the optical mode relative to the fused information, the total loss function of the optical mode is constructed.

[0074] Based on the supervisory loss function of SAR modes, the cooperative loss function of SAR modes relative to optical modes, and the cooperative loss function of SAR modes relative to fused information, the total loss function of SAR modes is constructed.

[0075] Optionally, the cooperative loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0076] (3)

[0077] in, This represents the cooperative loss function of SAR modes relative to optical modes;

[0078] The cooperative loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0079] (4)

[0080] in, This represents the cooperative loss function of the optical mode relative to the SAR mode.

[0081] Optionally, the cooperative loss function of the optical modal relative fusion information is expressed by the following formula (5):

[0082] (5)

[0083] in, The collaborative loss function represents the relative fusion information of optical modes; Indicates the first Predict the probability of changes in the sample fusion branch;

[0084] The collaborative loss function of the SAR modal relative fusion information is expressed by the following formula (6):

[0085] (6)

[0086] in, This represents the collaborative loss function for the relative fusion of SAR modes.

[0087] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0088] (7)

[0089] in, Represents the total loss function of optical modes; The first weighting coefficient represents the optical mode; The second weighting coefficient represents the optical mode; The third weighting coefficient represents the optical mode.

[0090] The total loss function of the SAR mode is expressed by the following formula (8):

[0091] (8)

[0092] in, Represents the total loss function of SAR modes; This represents the first weighting coefficient of the SAR mode; This represents the second weighting coefficient of the SAR mode; This represents the third weighting coefficient of the SAR mode.

[0093] On the other hand, a change detection device based on SAR and optical image co-learning is provided. The change detection device based on SAR and optical image co-learning includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-described change detection methods based on SAR and optical image co-learning is implemented.

[0094] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described change detection methods based on SAR and optical image co-learning.

[0095] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0096] This invention first acquires a pair of consecutive SAR images and a pair of consecutive optical images; then constructs a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; the SAR images are input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical images are input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module. The enhanced optical image features are obtained by enhancing the features in the ansformer feature enhancement module. These enhanced features are then input into a second decoder for decoding to obtain an optical image change detection prediction probability map. Next, the enhanced SAR image features and the enhanced optical image features are input into a feature fusion module for fusion to obtain fused features. These fused features are then input into a third decoder for decoding to obtain a fused information change detection prediction probability map. Based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fused information change detection prediction probability map, a total loss function for the network is constructed. The network is then trained using this total loss function to obtain a trained change detection network. Finally, the remote sensing image to be detected is acquired. This image is then input into the trained change detection network to obtain change detection results for various land cover types within the remote sensing image.

[0097] The embodiments of the present invention can be used on single SAR modal data, no longer constrained by the availability of optical images under weather and time conditions; the embodiments of the present invention can enhance the effect of change detection interpretation and improve the accuracy of SAR change detection network. Attached Figure Description

[0098] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0099] Figure 1 This is a flowchart of a change detection method based on SAR and optical image collaborative learning provided by an embodiment of the present invention;

[0100] Figure 2This is a schematic diagram of a change detection method based on SAR and optical image collaborative learning provided in an embodiment of the present invention;

[0101] Figure 3 This is a block diagram of a change detection device based on SAR and optical image collaborative learning provided in an embodiment of the present invention;

[0102] Figure 4 This is a schematic diagram of the structure of a change detection device based on SAR and optical image collaborative learning provided in an embodiment of the present invention. Detailed Implementation

[0103] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0104] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0105] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0106] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0107] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0108] This invention provides a change detection method based on SAR and optical image co-learning. This method can be implemented by a change detection device based on SAR and optical image co-learning, which can be a terminal or a server. Figure 1 The flowchart shown is for a change detection method based on SAR and optical image collaborative learning. The processing flow of this method may include the following steps:

[0109] S1. Acquire a pair of consecutive SAR images and a pair of consecutive optical images; construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders.

[0110] S2. Input the SAR image into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; input the optical image into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features.

[0111] S3. Input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map.

[0112] S4. Input the enhanced SAR image features and enhanced optical image features into the feature fusion module for fusion to obtain fused features; input the fused features into the third decoder for decoding to obtain the change detection and prediction probability map of the fused information.

[0113] Optionally, the enhanced SAR image features and enhanced optical image features are input into the feature fusion module for fusion to obtain fused features, including:

[0114] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module and fused through feature concatenation or addition to obtain the fused features; or

[0115] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused features are obtained by fusing them using a cross-attention method.

[0116] The implementation process of feature cascading is represented by the following formula (1):

[0117] (1)

[0118] in, Indicates fusion characteristics; Indicates the enhanced optical image features; This indicates the features of the enhanced SAR image;

[0119] The process of fusion by adding features is represented by the following formula (2):

[0120] (2)

[0121] Optionally, the enhanced SAR image features and enhanced optical image features are input into the feature fusion module for fusion. The fusion is performed using a cross-attention method to obtain fused features, including:

[0122] Using optical image features as the query and SAR image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the first linear transformation result; using SAR image features as the query and optical image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the second linear transformation result.

[0123] In one feasible implementation, the process of linearly transforming the enhanced SAR image features is represented by the following formulas (3)-(5):

[0124] (3)

[0125] (4)

[0126] (5)

[0127] in, This indicates that the optical image features are used as the query vector, which is a linear transformation of the enhanced optical image features; This indicates that SAR image features are used as key vectors, and represents a linear transformation of the enhanced SAR image features. This indicates that SAR image features are used as value vectors, and represents a linear transformation of the enhanced SAR image features. This represents the first learnable weight matrix; This represents the learnable second weight matrix; This represents the learnable third weight matrix.

[0128] In one feasible implementation, the process of linearly transforming the enhanced optical image features is represented by the following formulas (6)-(8):

[0129] (6)

[0130] (7)

[0131] (8)

[0132] in, This indicates that the SAR image feature query vector is a linear transformation of the enhanced SAR image features; This indicates that the optical image features are used as key vectors, and the result is a linear transformation of the enhanced optical image features. This represents a linear transformation of the optical image feature vector into the enhanced optical image features. This represents the learnable fourth weight matrix; This represents the learnable fifth weight matrix; This represents the learnable sixth weight matrix.

[0133] Based on the first linear transformation result, calculate the cross-attention result of the optical mode on the SAR mode; based on the second linear transformation result, calculate the cross-attention result of the SAR mode on the optical mode.

[0134] In one feasible implementation, the cross-attention result of the optical mode on the SAR mode is calculated and expressed by the following formula (9):

[0135] (9)

[0136] in, This represents the cross-attention result of the optical mode on the SAR mode; Indicates the dimension of the value vector.

[0137] In one feasible implementation, the cross-attention result of the SAR mode on the optical mode is calculated and expressed by the following formula (10):

[0138] (10)

[0139] in, This represents the cross-attention result of the SAR mode on the optical mode.

[0140] The cross-attention results of optical modes on SAR modes and cross-attention results of SAR modes on optical modes are fused to obtain fused features.

[0141] In one feasible implementation, the process of adding the cross-attention result of the optical mode on the SAR mode and the cross-attention result of the SAR mode on the optical mode is expressed by the following formula (11):

[0142] (11)

[0143] In one feasible implementation, the fusion process, which uses feature concatenation based on the cross-attention results of the optical mode on the SAR mode and the cross-attention results of the SAR mode on the optical mode, is expressed by the following formula (12):

[0144] (12)

[0145] S5. Based on the change detection prediction probability map of optical image, SAR image, fused information, and change ground truth labels, construct the total loss function of the network; train the network according to the total loss function to obtain the trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain the change detection result of the remote sensing image.

[0146] The change detection results obtained from remote sensing images, depending on the type of ground truth labels used in the training, may include: changes in water bodies, changes in buildings, changes in roads and other infrastructure, or a combination of the above changes.

[0147] Among them, the change truth label is a change truth label shared by SAR imagery and optical imagery.

[0148] The change truth labels were obtained through manual annotation.

[0149] In this process, backpropagation is used to update the network parameters through the Adam optimizer or SGD optimizer. The process S2-S5 is repeated until the model converges, and a trained change detection network is obtained.

[0150] Optionally, S5 calculates the total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fused information change detection prediction probability map, including:

[0151] S51. Compare the predicted probability map of optical image change detection with the ground truth label of optical image change detection, and calculate the supervised loss function of optical mode.

[0152] The supervised loss function for the optical mode is expressed by the following formula (13):

[0153] (13)

[0154] in, The supervision loss function represents the optical modes; Indicates the first Predict the probability of changes in the optical image of a sample; Indicates the first True value labels for changes in sample pairs; This represents the number of sample pairs in the training batch.

[0155] S52. Compare the SAR image change detection prediction probability map with the SAR image change detection ground truth label, and calculate the SAR mode supervision loss function;

[0156] The supervision loss function for the SAR mode is expressed by the following formula (14):

[0157] (14)

[0158] in, The supervision loss function represents the optical modes; Indicates the first Probability of SAR image change prediction for the sample;

[0159] S53. Compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the SAR mode relative to the optical mode; compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the optical mode relative to the SAR mode.

[0160] Optionally, the cooperative loss function of the SAR mode relative to the optical mode is expressed by the following formula (15):

[0161] (15)

[0162] in, This represents the cooperative loss function of SAR modes relative to optical modes;

[0163] The cooperative loss function of the optical mode relative to the SAR mode is expressed by the following formula (16):

[0164] (16)

[0165] in, This represents the cooperative loss function of the optical mode relative to the SAR mode.

[0166] S54. Compare the optical image change detection prediction probability map with the fused information change detection prediction probability map, and use KL divergence to calculate the collaborative loss function of the optical mode relative to the fused information; compare the SAR image change detection prediction probability map with the fused information change detection prediction probability map, and use KL divergence to calculate the collaborative loss function of the SAR mode relative to the fused information.

[0167] Optionally, the cooperative loss function of optical modalities relative to the fused information is expressed by the following formula (17):

[0168] (17)

[0169] in, The collaborative loss function represents the relative fusion information of optical modes; Indicates the first Predict the probability of changes in the sample fusion branch;

[0170] The collaborative loss function of SAR modes relative to fused information is expressed by the following formula (18):

[0171] (18)

[0172] in, This represents the collaborative loss function for the relative fusion of SAR modes.

[0173] S55. Construct the total loss function of the optical mode based on the supervision loss function of the optical mode, the cooperative loss function of the optical mode relative to the SAR mode, and the cooperative loss function of the optical mode relative to the fused information.

[0174] S56. Based on the supervision loss function of SAR mode, the cooperative loss function of SAR mode relative to optical mode, and the cooperative loss function of SAR mode relative to fused information, construct the total loss function of SAR mode.

[0175] Alternatively, the total loss function of the optical modes is expressed by the following formula (7):

[0176] (7)

[0177] in, Represents the total loss function of optical modes; The first weighting coefficient represents the optical mode; The second weighting coefficient represents the optical mode; The third weighting coefficient represents the optical mode.

[0178] The total loss function of the SAR mode is expressed by the following formula (8):

[0179] (8)

[0180] in, Represents the total loss function of SAR modes; This represents the first weighting coefficient of the SAR mode; This represents the second weighting coefficient of the SAR mode; This represents the third weighting coefficient of the SAR mode.

[0181] Among them, such as Figure 2This is a schematic diagram of a change detection method based on SAR and optical image collaborative learning provided by an embodiment of the present invention. In one feasible implementation, a change detection network based on SAR and optical image collaborative learning is constructed; a pair of consecutive SAR images and a pair of consecutive optical images are obtained; the pair of consecutive SAR images is input into a SAR image feature encoder for processing to obtain SAR image features; the pair of consecutive optical images is input into an optical image feature encoder for processing to obtain optical image features; the SAR image features are input into a Transformer-based feature enhancement module to obtain enhanced SAR image features; the optical image features are input into a Transformer-based feature enhancement module to obtain enhanced optical image features; the enhanced SAR image features and the enhanced optical image features are then processed together. The enhanced optical image features are fused to obtain fused features; the fused features are input into the fused feature change detection decoder to obtain the fused probability output layer; the enhanced SAR image features are input into the SAR image change detection decoder to obtain the SAR image change probability output layer; the enhanced optical image features are input into the optical image change detection decoder to obtain the optical image change probability output layer; based on the fused probability output layer, the SAR image change probability output layer, and the optical image change probability output layer change ground truth labels, a total loss function is constructed; the total loss function includes: the total loss function of the optical mode and the total loss function of the SAR mode; the total loss function is used to train the change detection network based on SAR and optical image co-learning to obtain the trained change detection network based on SAR and optical image co-learning.

[0182] This invention first acquires a pair of consecutive SAR images and a pair of consecutive optical images; then constructs a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; the SAR images are input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical images are input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second ... second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain a SAR image change detection prediction probability map; the enhanced SAR image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for The optical image features are enhanced in the feature enhancement module of the ORMER module to obtain enhanced optical image features. These enhanced features are then input into a second decoder for decoding to obtain an optical image change detection prediction probability map. Next, the enhanced SAR image features and the enhanced optical image features are input into a feature fusion module for fusion to obtain fused features. These fused features are then input into a third decoder for decoding to obtain a fused information change detection prediction probability map. Based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fused information change detection prediction probability map, and the change ground truth labels, a total loss function for the network is constructed. The network is then trained using this total loss function to obtain a trained change detection network. Finally, the remote sensing image to be detected is acquired. This image is then input into the trained change detection network to obtain change detection results for various land cover types within the remote sensing image.

[0183] The embodiments of the present invention can be used on single SAR modal data, no longer constrained by the availability of optical images under weather and time conditions; the embodiments of the present invention can enhance the effect of change detection interpretation and improve the accuracy of SAR change detection network.

[0184] Figure 3 This is a block diagram of a change detection device based on SAR and optical image co-learning, according to an exemplary embodiment. The device is used in a change detection method based on SAR and optical image co-learning. (Refer to...) Figure 3 The device includes an acquisition and construction unit 310, a decoding unit 320, an enhancement unit 330, a fusion unit 340, and a training unit 350. Wherein:

[0185] The acquisition and construction unit 310 is used to acquire a pair of consecutive SAR images and a pair of consecutive optical images; and to construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders;

[0186] The decoding unit 320 is used to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and to input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features.

[0187] Enhancement unit 330 is used to input SAR image features into a first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into a first decoder for decoding to obtain a SAR image change detection prediction probability map; input optical image features into a second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into a second decoder for decoding to obtain an optical image change detection prediction probability map.

[0188] The fusion unit 340 is used to input the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features; and to input the fused features into the third decoder for decoding to obtain a change detection and prediction probability map of the fused information.

[0189] The training unit 350 is used to construct the total loss function of the network based on the change detection prediction probability map of optical image, the change detection prediction probability map of SAR image, and the change detection prediction probability map of fused information; train the network according to the total loss function to obtain a trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results of various land cover types in the remote sensing image.

[0190] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features includes:

[0191] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module and fused through feature concatenation or addition to obtain the fused features; or

[0192] The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused features are obtained by fusing them using a cross-attention method.

[0193] Optionally, the step of inputting the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion, and obtaining fused features by fusing them using a cross-attention method, includes:

[0194] Using optical image features as the query and SAR image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the first linear transformation result; using SAR image features as the query and optical image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the second linear transformation result.

[0195] Based on the first linear transformation result, calculate the cross-attention result of the optical mode on the SAR mode; based on the second linear transformation result, calculate the cross-attention result of the SAR mode on the optical mode.

[0196] The cross-attention results of optical modes on SAR modes and cross-attention results of SAR modes on optical modes are fused to obtain fused features.

[0197] Optionally, the training unit 350 is used for:

[0198] The predicted probability map of optical image change detection is compared with the ground truth label of optical image change detection, and the supervised loss function of optical mode is calculated.

[0199] The supervised loss function for the optical mode is expressed by the following formula (1):

[0200] (1)

[0201] in, The supervision loss function represents the optical modes; Indicates the first Predict the probability of changes in the optical image of a sample; Indicates the first True value labels for changes in sample pairs; This represents the number of sample pairs in the training batch.

[0202] The predicted probability map of SAR image change detection is compared with the ground truth label of SAR image change detection, and the supervision loss function of SAR mode is calculated.

[0203] The supervision loss function for the SAR mode is expressed by the following formula (2):

[0204] (2)

[0205] in, The supervision loss function represents the optical modes; Indicates the first Probability of SAR image change prediction for the sample;

[0206] The optical image change detection prediction probability map is compared with the SAR image change detection prediction probability map. KL divergence is used to calculate the cooperative loss function of the SAR mode relative to the optical mode.

[0207] The probability map for change detection prediction of optical image is compared with the probability map for change detection prediction of fused information. KL divergence is used to calculate the collaborative loss function of optical mode relative to fused information. The probability map for change detection prediction of SAR image is compared with the probability map for change detection prediction of fused information. KL divergence is used to calculate the collaborative loss function of SAR mode relative to fused information.

[0208] Based on the supervised loss function of the optical mode, the cooperative loss function of the optical mode relative to the SAR mode, and the cooperative loss function of the optical mode relative to the fused information, the total loss function of the optical mode is constructed.

[0209] Based on the supervisory loss function of SAR modes, the cooperative loss function of SAR modes relative to optical modes, and the cooperative loss function of SAR modes relative to fused information, the total loss function of SAR modes is constructed.

[0210] Optionally, the cooperative loss function of the SAR mode relative to the optical mode is expressed by the following formula (3):

[0211] (3)

[0212] in, This represents the cooperative loss function of SAR modes relative to optical modes;

[0213] The cooperative loss function of the optical mode relative to the SAR mode is expressed by the following formula (4):

[0214] (4)

[0215] in, This represents the cooperative loss function of the optical mode relative to the SAR mode.

[0216] Optionally, the cooperative loss function of the optical modal relative fusion information is expressed by the following formula (5):

[0217] (5)

[0218] in, The collaborative loss function represents the relative fusion information of optical modes; Indicates the first Predict the probability of changes in the sample fusion branch;

[0219] The collaborative loss function of the SAR modal relative fusion information is expressed by the following formula (6):

[0220] (6)

[0221] in, This represents the collaborative loss function for the relative fusion of SAR modes.

[0222] Optionally, the total loss function of the optical mode is expressed by the following formula (7):

[0223] (7)

[0224] in, Represents the total loss function of optical modes; The first weighting coefficient represents the optical mode; The second weighting coefficient represents the optical mode; The third weighting coefficient represents the optical mode.

[0225] The total loss function of the SAR mode is expressed by the following formula (8):

[0226] (8)

[0227] in, Represents the total loss function of SAR modes; This represents the first weighting coefficient of the SAR mode; This represents the second weighting coefficient of the SAR mode; This represents the third weighting coefficient of the SAR mode.

[0228] This invention first acquires a pair of consecutive SAR images and a pair of consecutive optical images; then constructs a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; the SAR images are input into the first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; the optical images are input into the second CNN-based feature extraction encoder for feature extraction to obtain optical image features; the SAR image features are input into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for decoding to obtain a SAR image change detection prediction probability map; the optical image features are input into the second ... second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain a SAR image change detection prediction probability map; the optical image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain a SAR image change detection prediction probability map; the enhanced SAR image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the second Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for enhancement to obtain enhanced SAR image features; the enhanced SAR image features are input into the first decoder for The optical image features are enhanced in the feature enhancement module of the ORMER module to obtain enhanced optical image features. These enhanced features are then input into a second decoder for decoding to obtain an optical image change detection prediction probability map. Next, the enhanced SAR image features and the enhanced optical image features are input into a feature fusion module for fusion to obtain fused features. These fused features are then input into a third decoder for decoding to obtain a fused information change detection prediction probability map. Based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fused information change detection prediction probability map, and the change ground truth labels, a total loss function for the network is constructed. The network is then trained using this total loss function to obtain a trained change detection network. Finally, the remote sensing image to be detected is acquired. This image is then input into the trained change detection network to obtain change detection results for various land cover types within the remote sensing image.

[0229] The embodiments of the present invention can be used on single SAR modal data, no longer constrained by the availability of optical images under weather and time conditions; the embodiments of the present invention can enhance the effect of change detection interpretation and improve the accuracy of SAR change detection network.

[0230] Figure 4 This is a schematic diagram of a change detection device based on SAR and optical image collaborative learning provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a change detection device based on SAR and optical image collaborative learning can include the above-mentioned Figure 3 The change detection device based on SAR and optical image co-learning is shown. Optionally, the change detection device 410 based on SAR and optical image co-learning may include a first processor 2001.

[0231] Optionally, the change detection device 410 based on SAR and optical image co-learning may also include a memory 2002 and a transceiver 2003.

[0232] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0233] The following is combined with Figure 4 A detailed introduction to each component of the change detection device 410 based on SAR and optical image collaborative learning is provided below:

[0234] The first processor 2001 is the control center of the change detection device 410 based on SAR and optical image collaborative learning. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0235] Optionally, the first processor 2001 can perform various functions of the change detection device 410 based on SAR and optical image co-learning by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0236] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0237] In a specific implementation, as one example, the change detection device 410 based on SAR and optical image collaborative learning may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0238] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0239] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the SAR and optical image co-learning change detection device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0240] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0241] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0242] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the change detection device 410 based on SAR and optical image collaborative learning. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0243] It should be noted that, Figure 4 The structure of the change detection device 410 based on SAR and optical image co-learning shown does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0244] Furthermore, the technical effect of the change detection device 410 based on SAR and optical image collaborative learning can be referred to the technical effect of the change detection method based on SAR and optical image collaborative learning described in the above method embodiments, and will not be repeated here.

[0245] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0246] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0247] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0248] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0249] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0250] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0251] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0252] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0253] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0254] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0256] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0257] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A change detection method based on SAR and optical image collaborative learning, characterized in that, The method includes: S1. Acquire a pair of consecutive SAR images and a pair of consecutive optical images; construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; S2. Input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features; S3. Input the SAR image features into the first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into the first decoder for decoding to obtain a SAR image change detection prediction probability map; input the optical image features into the second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into the second decoder for decoding to obtain an optical image change detection prediction probability map. S4. Input the enhanced SAR image features and enhanced optical image features into the feature fusion module for fusion to obtain fused features; input the fused features into the third decoder for decoding to obtain the change detection and prediction probability map of the fused information; S5. Construct the total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, the fused information change detection prediction probability map, and the change ground truth label; train the network based on the total loss function to obtain a trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain the change detection result of the remote sensing image.

2. The change detection method based on SAR and optical image collaborative learning according to claim 1, characterized in that, The enhanced SAR image features and enhanced optical image features are input into the feature fusion module for fusion to obtain fused features, including: The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused by feature concatenation or addition to obtain the fused features; or The enhanced SAR image features and enhanced optical image features are input into the feature fusion module, and fused features are obtained by fusing them using a cross-attention method.

3. The change detection method based on SAR and optical image collaborative learning according to claim 2, characterized in that, The enhanced SAR image features and enhanced optical image features are input into the feature fusion module for fusion. The fusion is performed using a cross-attention method to obtain fused features, including: Using optical image features as the query and SAR image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the first linear transformation result; using SAR image features as the query and optical image features as the key and value, a linear transformation is performed on the enhanced optical and SAR image features to obtain the second linear transformation result. Based on the first linear transformation result, calculate the cross-attention result of the optical mode on the SAR mode; based on the second linear transformation result, calculate the cross-attention result of the SAR mode on the optical mode. The cross-attention results of optical modes on SAR modes and cross-attention results of SAR modes on optical modes are fused to obtain fused features.

4. The change detection method based on SAR and optical image collaborative learning according to claim 1, characterized in that, The step S5, which calculates the total loss function of the network based on the optical image change detection prediction probability map, the SAR image change detection prediction probability map, and the fused information change detection prediction probability map, includes: S51. Compare the predicted probability map of optical image change detection with the ground truth label of optical image change detection, and calculate the supervised loss function of optical mode. The supervised loss function for the optical mode is expressed by the following formula (1): in, The supervision loss function represents the optical modes; y represents the predicted probability of optical image change for the i-th sample; i The true label represents the change of the i-th sample pair; N represents the number of sample pairs in the training batch. S52. Compare the SAR image change detection prediction probability map with the SAR image change detection ground truth label, and calculate the SAR mode supervision loss function; The supervision loss function for the SAR mode is expressed by the following formula (2): in, The supervision loss function represents the optical modes; This represents the SAR image change prediction probability for the i-th sample; S53. Compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the SAR mode relative to the optical mode; compare the optical image change detection prediction probability map with the SAR image change detection prediction probability map, and use KL divergence to calculate the cooperative loss function of the optical mode relative to the SAR mode. S54. Compare the optical image change detection prediction probability map with the fused information change detection prediction probability map, and calculate the collaborative loss function of the optical mode relative to the fused information; compare the SAR image change detection prediction probability map with the fused information change detection prediction probability map, and calculate the collaborative loss function of the SAR mode relative to the fused information. S55. Construct the total loss function of the optical mode based on the supervision loss function of the optical mode, the cooperative loss function of the optical mode relative to the SAR mode, and the cooperative loss function of the optical mode relative to the fused information. S56. Based on the supervision loss function of SAR mode, the cooperative loss function of SAR mode relative to optical mode, and the cooperative loss function of SAR mode relative to fused information, construct the total loss function of SAR mode.

5. The change detection method based on SAR and optical image collaborative learning according to claim 4, characterized in that, The cooperative loss function of the SAR mode relative to the optical mode is expressed by the following formula (3): in, This represents the cooperative loss function of SAR modes relative to optical modes; The cooperative loss function of the optical mode relative to the SAR mode is expressed by the following formula (4): in, This represents the cooperative loss function of the optical mode relative to the SAR mode.

6. The change detection method based on SAR and optical image collaborative learning according to claim 5, characterized in that, The cooperative loss function of the optical modal relative fusion information is expressed by the following formula (5): in, The collaborative loss function represents the relative fusion information of optical modes; This represents the predicted probability of change in the fusion branch of the i-th sample pair; The collaborative loss function of the SAR modal relative fusion information is expressed by the following formula (6): in, This represents the collaborative loss function for the relative fusion of SAR modes.

7. The change detection method based on SAR and optical image collaborative learning according to claim 6, characterized in that, The total loss function of the optical mode is expressed by the following formula (7): in, The total loss function for optical modes; α opt The first weighting coefficient representing the optical mode; β opt The second weighting coefficient represents the optical mode; γ opt The third weighting coefficient represents the optical mode. The total loss function of the SAR mode is expressed by the following formula (8): in, The total loss function for SAR modes; α sar The first weighting coefficient for the SAR mode; β sar The second weighting coefficient for the SAR mode; γ sar This represents the third weighting coefficient of the SAR mode.

8. A change detection device based on SAR and optical image co-learning, wherein the change detection device based on SAR and optical image co-learning is used to implement the change detection method based on SAR and optical image co-learning as described in any one of claims 1-7, characterized in that, The device includes: An acquisition and construction unit is used to acquire a pair of consecutive SAR images and a pair of consecutive optical images; and to construct a change detection network based on the collaborative learning of SAR and optical images; the network includes: two CNN-based feature extraction encoders, two Transformer-based feature enhancement modules, a feature fusion module, and three decoders; The decoding unit is used to input the SAR image into a first CNN-based feature extraction encoder for feature extraction to obtain SAR image features; and to input the optical image into a second CNN-based feature extraction encoder for feature extraction to obtain optical image features. An enhancement unit is used to input SAR image features into a first Transformer-based feature enhancement module for enhancement to obtain enhanced SAR image features; input the enhanced SAR image features into a first decoder for decoding to obtain a SAR image change detection prediction probability map; input optical image features into a second Transformer-based feature enhancement module for enhancement to obtain enhanced optical image features; input the enhanced optical image features into a second decoder for decoding to obtain an optical image change detection prediction probability map. The fusion unit is used to input the enhanced SAR image features and the enhanced optical image features into the feature fusion module for fusion to obtain fused features; the fused features are then input into the third decoder for decoding to obtain a change detection and prediction probability map of the fused information. The training unit is used to construct the total loss function of the network based on the change detection prediction probability map of optical imagery, the change detection prediction probability map of SAR imagery, and the change detection prediction probability map of fused information; train the network according to the total loss function to obtain a trained change detection network; acquire the remote sensing image to be detected; input the remote sensing image to be detected into the trained change detection network to obtain change detection results of various land cover types in the remote sensing imagery.

9. A change detection device based on SAR and optical image collaborative learning, characterized in that, The change detection device based on SAR and optical image collaborative learning includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images