A multi-resolution multi-source remote sensing image change detection method

Abstract

The application provides a multi-resolution multi-source remote sensing image change detection method, which comprises the following steps: for a first image conversion network and a second image conversion network, the similarity between a pseudo-optical image and a real optical image and the similarity between a pseudo-SAR image and a real SAR image are constrained by a partial generation loss function; the real optical image is down-sampled, and the real SAR image is up-sampled; the processed images are respectively input into the first and second image conversion networks, and the similarity between the outputs of the two image conversion networks and the original images is constrained by another partial generation loss function; a discrimination and detection network is used to realize the discrimination between the generated pseudo-image and the source image and the change detection task; the discriminator network parameters are optimized by a discrimination loss and a change detection loss; finally, the trained network is used to detect the changes of the SAR image and the optical image, and a high-precision detection result is obtained. The application can realize the change detection between multi-source remote sensing images with different resolutions and different modalities.

Description

Technical Field

[0001] This application relates to the field of remote sensing image change detection technology, and in particular to a method for detecting changes in multi-resolution, multi-source remote sensing images. Background Technology

[0002] Remote sensing images are crucial carriers of information about the Earth's surface features. Processing and analyzing these images can drive the development of intelligent remote sensing data processing technology. Large-scale surface change detection refers to identifying areas of significant change by comparing remote sensing data of the same region over different time periods. This technology has important applications in areas such as surface environment monitoring, disaster emergency response, and urban and rural planning.

[0003] With the advancement of remote sensing technology, remote sensing data from different sensors provides data resources for change detection. As different types of remote sensing images are continuously acquired, research on change detection among multi-source remote sensing data is becoming increasingly urgent. For example, optical remote sensing images and synthetic aperture radar (SAR) are the two most commonly used multi-source remote sensing images for change detection.

[0004] However, multi-source heterogeneous remote sensing images have different imaging mechanisms and significant differences in image modality, resolution, and other characteristics. For example, SAR images and high-resolution optical images typically have a certain resolution difference, and SAR images have less detail and are more susceptible to noise compared to optical remote sensing images. Current methods for change detection in multi-source, multi-resolution remote sensing images mainly use resampling strategies, such as bilinear bicubic interpolation, to equalize resolution differences. However, for pixel-level prediction tasks like change detection, the difference in image resolution before and after the change inevitably affects the detection accuracy. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first objective of this application is to propose a method for detecting changes in multi-resolution, multi-source remote sensing images.

[0007] The second objective of this application is to propose a multi-resolution, multi-source remote sensing image change detection device.

[0008] The third objective of this application is to propose an electronic device.

[0009] The fourth objective of this application is to provide a computer-readable storage medium.

[0010] The fifth objective of this application is to provide a computer program product.

[0011] To achieve the above objectives, the first aspect of this application proposes a method for detecting changes in multi-resolution, multi-source remote sensing images, comprising:

[0012] An image transformation network is constructed, and the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images is constrained by a partial generation loss function. The first image transformation network is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network is used to convert optical images into pseudo-SAR images.

[0013] The real optical image is downsampled and the real SAR image is upsampled. The processed images are then input into the first and second image transformation networks, respectively. Another part of the generation loss function constrains the similarity between the outputs of the two image transformation networks and the original image.

[0014] A discrimination and detection network is constructed, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical images and real optical images, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR images and real SAR images.

[0015] The first discriminative loss is calculated, and the parameters of the image conversion network are updated through backpropagation based on a joint optimization strategy of multiple generation losses and the first discriminative loss.

[0016] The second discrimination loss and change detection loss are calculated. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation.

[0017] Using the trained image conversion network and discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo optical image, and the pseudo optical image and the original optical image are input together into the first discrimination and detection network to obtain the final change detection result.

[0018] Optionally, the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images, can be constrained using a partial generation loss function, including:

[0019] Given dual-temporal low-resolution SAR images of different resolutions I A With optical image I B Image I A The image is input into the first image transformation network G1 to generate the first pseudo-optical image I. B′ , where I B′ resolution and I B Same; Image I BThe image is input into the second image transformation network G2 to generate the first pseudo-SAR image I. A′ Image I A′ With Image I A They have the same resolution;

[0020] The first pseudo-optical image I generated B′ The input is fed into the second image transformation network G2 to generate the second pseudo-SAR image I. A″ and the first pseudo SAR image I generated A′ The image is input into the first image transformation network G1 to generate the second pseudo-optical image I. B″ ;

[0021] Calculate I respectively A with I A″ The generation loss between them, and I B with I B″ The generation losses between them, and their loss function formulas are as follows:

[0022] Loss1 = L1(I A″ ,I A )

[0023] Loss2=L1(I B″ ,I B )

[0024] Where Loss1 is I A with I A″ The loss function between them, Loss2 is I B with I B″ The loss function between them.

[0025] Optionally, the real optical image is downsampled, and the real SAR image is upsampled. The processed images are then input into the first and second image transformation networks, respectively. A separate generation loss function constrains the similarity between the outputs of the two image transformation networks and the original image, including:

[0026] To further constrain the generation performance of the first image transformation network G1, the image I is transformed using... B Downsampled image I B Using _down as input, calculate the output image and image I. B The loss, whose loss function Loss3 is:

[0027] Loss3 = L1(G1(I B _down),I B )

[0028] In the formula, G1() represents the transformation process of the first image transformation network G1;

[0029] To further constrain the generation performance of the second image transformation network G2, the image I is used... A Interpolated image I A Using _up as input, calculate the output image and image I. A The loss, and its loss function Loss4 is:

[0030] Loss4=L1(G2(I A _up),I A )

[0031] In the formula, G2() represents the transformation process of the second image transformation network G2.

[0032] Optionally, the calculation process for the first discrimination and detection network and the first discrimination and detection network includes:

[0033] The first pseudo-optical image I generated by the first image transformation network G1 B′ With the original optical image I B The input is fed into the first discrimination and detection network D1 to calculate the first pseudo-optical image I. B′ With the original optical image I B The discrimination loss between them, the discrimination loss function Loss5 of the first discrimination and detection network D1 is:

[0034] Loss5=L2(D1(I B ),lbl B )

[0035] Among them, lbl B It is an optical image label composed of 1s;

[0036] The first pseudoSAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A The input is fed into the second discrimination and detection network D2 to calculate the first pseudo-SAR image I. A′ Compared with the original SAR image I A The discrimination loss between the two, and the discrimination loss function Loss6 of the second discrimination and detection network D2 is:

[0037] Loss6=L2(D2(I A ),lbl A )

[0038] Among them, lbl A A SAR image label is composed of 1s.

[0039] Optionally, based on a joint optimization strategy of multiple generation losses and a first discriminant loss, the parameters of the image transformation network are updated through backpropagation, including:

[0040] The loss functions Loss1 to Loss6 are combined according to a preset weight ratio to calculate the comprehensive loss function Loss. G Backpropagation updates the network parameters of the first image transformation network G1 and the second image transformation network G2, and the combined loss function is used. G The calculation formula is:

[0041]

[0042] Where, ω i These are the weighting coefficients for each loss function, used to adjust the contribution of each part of the loss to the total loss.

[0043] Optionally, the calculation process for the second discrimination loss and change detection loss of the first discrimination and detection network and the second discrimination and detection network includes:

[0044] The first pseudo-optical image I generated by the first image transformation network G1 B′ With the original optical image I B In the first discrimination and detection network D1, the second discrimination loss is calculated on one hand, and the first pseudo-optical image I is calculated on the other hand. B′ With the original optical image I B The change detection loss is calculated between the two loss functions, where the discriminant loss function Loss7 and the change detection loss function Loss8 are respectively:

[0045] Loss7=L2(D1(I B′ ),lbl′ B )

[0046] Loss8 = Focal_Loss1

[0047] Among them, lbl′ B It is an optical image label composed of 0s;

[0048] The first pseudoSAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A In the second discrimination and detection network D2, the second discrimination loss is calculated on one hand, and the first pseudo-SAR image I is calculated on the other hand. A′ Compared with the original SAR image I A The change detection loss between them, where the discriminant loss function Loss9 and the change detection loss function Loss9 are used. 10 They are respectively:

[0049] Loss9=L2(D2(I A′ ),lbl′ A )

[0050] Loss 10 =Focal_Loss2

[0051] Among them, lbl′ A SAR image labels are composed of 0s.

[0052] Optionally, based on a joint optimization strategy of the first discrimination loss, the second discrimination loss, and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation, including:

[0053] The above loss function Loss5 to Loss 10 The combined loss function (Loss) is calculated based on the preset weight ratios. D Backpropagation updates the network parameters of the first discriminant and detection network D1 and the second discriminant and detection network D2, and the combined loss function is used. D The calculation formula is:

[0054]

[0055] Where, ω i These are the weighting coefficients of the corresponding loss function, used to adjust the contribution of each sub-loss to the total loss.

[0056] To achieve the above objectives, a second aspect of this application provides a multi-resolution, multi-source remote sensing image change detection device, comprising:

[0057] The transformation network construction and generation constraint module is used to construct an image transformation network and constrain the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images through a partial generation loss function. The first image transformation network is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network is used to convert optical images into pseudo-SAR images.

[0058] The network constraint enhancement module is used to downsample real optical images and upsample real SAR images. The processed images are input into the first and second image transformation networks respectively. Another part of the generation loss function constrains the similarity between the outputs of the two image transformation networks and the original images.

[0059] The discrimination and detection network construction module is used to construct the discrimination and detection network. The first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical images and real optical images, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR images and real SAR images.

[0060] The first joint optimization module is used to calculate the first discriminative loss. Based on the joint optimization strategy of multiple generation losses and the first discriminative loss, the parameters of the image conversion network are updated through backpropagation.

[0061] The second joint optimization module is used to calculate the second discrimination loss and the change detection loss. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation.

[0062] The final change detection module is used to input the original SAR image into the first image conversion network to generate a pseudo-optical image using the trained image conversion network and discrimination and detection network, and then input the pseudo-optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result.

[0063] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0064] The memory stores computer-executed instructions;

[0065] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0066] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0067] To achieve the above objectives, a fifth aspect of this application provides a computer program product that, when executed by a processor, implements the method described in any one of the first aspects.

[0068] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0069] It can realize the mutual transformation between multi-source remote sensing with different resolutions and modalities, thereby aligning the resolutions of multi-source images, and then processing the change detection between multi-source remote sensing images after resolution alignment.

[0070] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0071] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0072] Figure 1 This is a flowchart illustrating a method for detecting changes in multi-resolution, multi-source remote sensing images provided in an embodiment of this application.

[0073] Figure 2 This is a schematic diagram of a multi-source remote sensing change detection dataset with different resolutions provided in an embodiment of this application;

[0074] Figure 3(a) is a schematic diagram of the change detection truth results provided in the embodiments of this application;

[0075] Figure 3(b) is a schematic diagram of the detection results of the HPT method provided in the embodiments of this application;

[0076] Figure 3(c) is a schematic diagram of the detection results of the method of this application provided in the embodiment of this application.

[0077] Figure 4 This is a schematic diagram of the structure of a multi-resolution, multi-source remote sensing image change detection device provided in an embodiment of this application. Detailed Implementation

[0078] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0079] To address this issue, this application provides a method for detecting changes in multi-resolution, multi-source remote sensing images. This method enables the transformation between multi-source remote sensing images of different resolutions and modalities, thereby aligning the resolutions of the multi-source images and then processing the changes between the aligned multi-source remote sensing images.

[0080] It should be noted that this application relates to a method for detecting changes in multi-resolution, multi-source remote sensing images. "Multi-source remote sensing images" refers to image data acquired by different types of remote sensing sensors, including but not limited to optical images, SAR (Synthetic Aperture Radar) images, multispectral images, and infrared images. Although optical images and SAR images are used as examples in the embodiments of this application for detailed description, this method is equally applicable to other types of multi-source remote sensing images.

[0081] Figure 1 This is a schematic flowchart illustrating a multi-resolution, multi-source remote sensing image change detection method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0082] Step 101: Construct an image conversion network and constrain the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images, through a partial generation loss function.

[0083] In this embodiment, the image transformation network includes a first image transformation network G1 and a second image transformation network G2. The first image transformation network G1 is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network G2 is used to convert optical images into pseudo-SAR images. The constructed deep neural networks G1 and G2 can employ commonly used deep neural networks for low-level visual task feature extraction; this application does not impose any additional limitations on this.

[0084] In the specific implementation of this application, optical images and SAR images are chosen as examples because they are typical combinations for multi-source remote sensing image fusion and change detection, and are commonly used in practical applications. However, this application is not limited to these image types. This method is applicable to any type of multi-source remote sensing image, including but not limited to multispectral images, thermal infrared images, and combinations of LiDAR and optical images. Users can select different types of multi-source images for change detection tasks according to their actual application needs; this application does not impose specific limitations on this.

[0085] In one possible embodiment, the dataset used in this application is a flood change detection dataset for the Gloucester region of the UK. This dataset contains two types of heterogeneous remote sensing images: pre-change images (synthetic aperture radar (SAR) images) used to capture surface features of the region before flooding; and post-change images (normalized difference vegetation index (NDVI) maps) used to describe vegetation changes after flooding. Both types of images were collected from different periods in the same region, forming heterogeneous, multi-temporal, and multi-resolution change detection remote sensing image pairs.

[0086] Figure 2 (a)-2(c) are schematic diagrams of multi-source remote sensing change detection datasets with different resolutions used in the embodiments of this application, wherein:

[0087] Figure 2 (a) and Figure 2 (c): Shows the distribution of the original SAR images and NDVI images in the change detection dataset. The original SAR images and NDVI images are remote sensing images of the same area at different times, forming heterogeneous image pairs for change detection.

[0088] Figure 2(b): The low-resolution SAR image obtained after bicubic downsampling forms a heterogeneous image pair with the high-resolution optical NDVI image, exhibiting a significant difference in resolution. The resolution of the low-resolution SAR image is reduced to 1 / 8 of the original resolution, while the NDVI image retains its original resolution.

[0089] To achieve the task of change detection in multi-resolution, multi-source remote sensing images, this application first performs resolution processing on the dataset: the SAR image before the change is downsampled by 8 times using a bicubic downsampling method to form a low-resolution SAR image (e.g., Figure 2 (b) shows that the optical image (NDVI image) retains its original resolution and forms a heterogeneous remote sensing image pair with the low-resolution SAR image.

[0090] To accommodate subsequent deep learning model training, this application also performs cropping processing on the images after the resolution processing described above: low-resolution SAR images are cropped into 16×16 blocks; optical images (NDVI) are cropped into 128×128 blocks. If the original image size is not divisible by 16 or 128, cropping is completed by padding with zeros on the right and bottom of the image. The cropped image pairs will be used as standardized input data for training subsequent image transformation and change detection networks.

[0091] Through the aforementioned resolution preprocessing, this application achieves standardized processing of low-resolution SAR images and high-resolution NDVI optical images, providing high-quality input data for the subsequent training of image conversion networks, discrimination and detection networks, and ensuring the accuracy and robustness of change detection tasks.

[0092] After completing multi-resolution processing and cropping, a standardized image slice dataset was obtained. This application further partitions these cropped image slices to meet the model training requirements. The specific steps are as follows: the cropped image slice data is divided into a training set and a test set; 20% of the image slices are randomly selected from all cropped slices as the training set to train the first image transformation network G1 and the second image transformation network G2, as well as the discrimination and detection networks D1 and D2.

[0093] Specifically, this application proposes a change detection method based on an image transformation network and a generation loss function for dual-temporal low-resolution SAR images and optical images of different resolutions. The method includes the following steps:

[0094] First, for a given dual-temporal low-resolution SAR image I at different resolutions A With optical image I B Image I AThe image is input into the first image transformation network G1 to generate the first pseudo-optical image I. B′ , where I B′ resolution and I B Same; Image I B The image is input into the second image transformation network G2 to generate the first pseudo-SAR image I. A′ Image I A′ With Image I A They have the same resolution.

[0095] Furthermore, this application will generate the first pseudo-optical image I B′ The input is fed into the second image transformation network G2 to generate the second pseudo-SAR image I. A″ Simultaneously, the first pseudo-SAR image I generated... A′ The image is input into the first image transformation network G1 to generate the second pseudo-optical image I. B″ Through the two transformation operations described above, multiple generation relationships are established between the pseudo-optical image and the real optical image, and between the pseudo-SAR image and the real SAR image.

[0096] It should be noted that the generation loss function not only optimizes the image quality of a single generation, but also ensures the similarity of the generated pseudo-image to the real image in terms of structure and resolution through multiple transformation processes. For example, the second pseudo-SAR image needs to be as close as possible to the real low-resolution SAR image in terms of texture and resolution, while the second pseudo-optical image needs to be close to the real optical image in terms of optical features. Specifically, to constrain the generation effect, the embodiments of this application calculate I respectively. A with I A″ The generation loss between them, and I B with I B″ The generation loss between the pseudo and real images is constrained by the following generation loss functions:

[0097] Loss1 = L1(I A″ ,I A )

[0098] Loss2=L1(I B″ ,I B )

[0099] Where Loss1 is I A with I A″ The loss function between them, Loss2 is I B with I B″ The loss function between them.

[0100] These generation loss functions work together to ensure that the pseudo-images gradually approach the features of the real image during multiple image transformations. Through multi-stage generation and transformation, this application not only improves the realism of the pseudo-images but also provides high-quality input data for subsequent discrimination and change detection networks, significantly improving the overall performance of the change detection task.

[0101] Step 102: Downsample the real optical image and upsample the real SAR image. Input the processed images into the first and second image transformation networks respectively. Use another part of the generation loss function to constrain the similarity between the output of the two image transformation networks and the original image.

[0102] Based on the construction of the image transformation network, in order to further optimize the similarity between the pseudo image and the real image, this application performs downsampling and upsampling processing on the real optical image and the real SAR image respectively, and inputs the processed images into the first and second image transformation networks respectively. The similarity between the output of the two image transformation networks and the original image is constrained by another part of the generation loss function.

[0103] First, the real optical image is downsampled, reducing its resolution from high to a low level consistent with the SAR image. Simultaneously, the real SAR image is upsampled, increasing its resolution to the same high level as the optical image. Then, the processed images are input into the first and second image transformation networks, respectively. The first and second image transformation networks, by adding downsampling and upsampling layers, respectively change the resolution of the downsampled optical image and the upsampled SAR image after passing through the first image transformation network G1 and the second image transformation network G2. Specifically:

[0104] An upsampling layer is added to the G1 network to ensure improved image resolution, i.e., to achieve a transition from low-resolution to high-resolution image size. Simultaneously, a downsampling layer is added to the G2 network to achieve a mapping from high-resolution to low-resolution image size. The upsampling layer can be implemented using pixel shuffle layers or deconvolution layers, while the downsampling layer can be implemented using convolutions with stride.

[0105] Subsequently, regarding the generation effect of the first image conversion network G1, this application uses a downsampled optical image I. B The image is taken as input and compared with the real optical image I. B By comparing the results, the loss function Loss3 is calculated, as shown in the following formula:

[0106] Loss3 = L1(G1(I B _down),I B )

[0107] In the formula, G1() represents the transformation process of the first image transformation network G1. Loss3 is the generation loss of the first image transformation network when processing downsampled optical images, used to improve the generation effect of G1, so that it can better generate images similar to the real optical images I. B Similar results.

[0108] Similarly, to further constrain the generation effect of the second image transformation network G2, this application uses the transformed image I A Interpolated image I A Using _up as input, calculate the output image and image I. A The loss, and its loss function Loss4 is:

[0109] Loss4=L1(G2(I A _up),I A )

[0110] In the formula, G2() represents the transformation process of the second image transformation network G2. Loss4 is the generation loss of the second image transformation network when processing upsampled low-resolution SAR images, used to improve the generation effect of G2, so that it can better generate images similar to the real SAR image I. A Similar results.

[0111] The generation loss Loss3 and Loss4 described above further optimize the generation effect of the image transformation network, ensuring the quality of the generated image in terms of resolution, texture and features, and improving the accuracy of multi-stage mapping between low-resolution SAR images and optical images.

[0112] Step 103: Construct the discrimination and detection network.

[0113] In this embodiment, a discrimination and detection network is constructed to introduce a authenticity discrimination and change detection mechanism between the generated pseudo-images and real images. Specifically, a first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo-optical images and real optical images, while a second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo-SAR images and real SAR images.

[0114] By leveraging the realism and change detection functions of both the first and second discrimination and detection networks, the quality of the generated images can be significantly improved. Furthermore, the performance of the image transformation and discrimination and detection networks can be optimized through discrimination loss and change detection loss. Ultimately, this joint constraint mechanism ensures collaborative optimization between the generation and detection networks, achieving high-quality generated images and accurate change detection.

[0115] The definition and optimization process of the specific discrimination loss and change detection loss for the discrimination and detection network will be described in detail in subsequent steps. Through joint optimization of multiple generation losses, discrimination losses, and change detection losses, the performance of the entire network in image transformation and change detection tasks will be further improved.

[0116] Step 104: Calculate the first discriminant loss. Based on the joint optimization strategy of multiple generation losses and the first discriminant loss, update the parameters of the image conversion network through backpropagation.

[0117] In this embodiment of the application, by calculating the first discriminant loss, the aim is to optimize the performance of the image conversion network, so that the pseudo-optical images it generates are closer to real optical images, while enhancing the image conversion network's ability to learn image features.

[0118] In the calculation of the first loss determination:

[0119] For the first discrimination and detection network D1, the first pseudo-optical image I generated by the first image transformation network G1 B′ With the original optical image I B The input is fed into the first discrimination and detection network D1 to calculate the first pseudo-optical image I. B′ With the original optical image I B The discrimination loss between them, the discrimination loss function Loss5 of the first discrimination and detection network D1 is:

[0120] Loss5=L2(D1(I B ),lbl B )

[0121] Among them, lbl B The optical image label consists of 1s. Loss5 is the discrimination loss of the first discrimination and detection network D1, which measures the authenticity between the fake optical image and the real optical image.

[0122] For the second discrimination and detection network D2, the first pseudo-SAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A The input is fed into the second discrimination and detection network D2 to calculate the first pseudo-SAR image I. A′ Compared with the original SAR image I A The discrimination loss between the two, and the discrimination loss function Loss6 of the second discrimination and detection network D2 is:

[0123] Loss6=L2(D2(I A ),lbl A )

[0124] Among them, lbl AThe SAR image label consists of 1s. Loss6 is the discrimination loss of the second discrimination and detection network D2, which measures the authenticity between fake SAR images and real SAR images.

[0125] To further optimize the performance of the image conversion network, the aforementioned loss functions Loss1 to Loss6 are combined according to a preset weight ratio to calculate the comprehensive loss function Loss. G The network parameters of the first image transformation network G1 and the second image transformation network G2 are updated through backpropagation. The overall loss function is Loss. G The calculation formula is:

[0126]

[0127] Where, ω i These are the weighting coefficients for each loss function, used to adjust the contribution of each part of the loss to the total loss.

[0128] Step 105: Calculate the second discrimination loss and change detection loss. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, update the parameters of the discrimination and detection networks through backpropagation.

[0129] Next, the discriminator is trained. The second discriminant loss and change detection loss of the first and second discriminant and detection networks are calculated through the following steps:

[0130] First, the first pseudo-optical image I generated by the first image transformation network G1 is... B′ With the original optical image I B In the first discrimination and detection network D1, the second discrimination loss is calculated on one hand, and the first pseudo-optical image I is calculated on the other hand. B′ With the original optical image I B The loss is calculated based on the change detection loss. Specifically, the discriminant loss function Loss7 and the change detection loss function Loss8 are as follows:

[0131] Loss7=L2(D1(I B′ ),lbl′ B )

[0132] Loss8 = Focal_Loss1

[0133] Among them, lbl′ B It is an optical image label composed of 0s.

[0134] Loss7 is the discrimination loss of the first discrimination and detection network D1, used to measure the pseudo-optical image I generated by the first image transformation network G1. B′ and real optical image I BThe accuracy of classification between them.

[0135] Loss8 is the change detection loss of the first discriminant and detection network D1, used to measure the pseudo-optical image I. B′ With real optical image I B Detection performance in changing regions.

[0136] Similarly, the first pseudo-SAR image I generated by the second image transformation network G2 is... A′ Compared with the original SAR image I A′ In the second discrimination and detection network D2, the second discrimination loss is calculated on one hand, and the first pseudo-SAR image I is calculated on the other hand. A′ Compared with the original SAR image I A′ The change detection loss is calculated between the discriminant loss function Loss9 and the change detection loss function Loss9. 10 They are respectively:

[0137] Loss9=L2(D2(I A′ ),lbl′ A )

[0138] Loss 10 =Focal_Loss2

[0139] Among them, lbl′ A SAR image labels are composed of 0s.

[0140] Loss9 is the discrimination loss of the second discrimination and detection network D2. It measures the pseudo-SAR image I generated by the second image transformation network G2. A′ Compared with real SAR image I A The accuracy of classification between them.

[0141] Loss 10 It is the change detection loss of the second discrimination and detection network D2, which measures the pseudo-SAR image I generated by the second image transformation network G2. A′ Compared with real SAR image I A Detection performance in changing regions.

[0142] Using the above loss function Loss7 to Loss 10 This optimizes the authenticity judgment and change region detection of generated images. These losses work together to train the first and second discrimination and detection networks, thereby improving the model's change detection performance on multi-resolution, multi-source remote sensing images.

[0143] To further optimize the performance of the discrimination and detection network, this application embodiment modifies all loss functions of the discrimination and detection network from Loss5 to Loss.10 The components are combined according to a preset weight ratio to form a comprehensive loss function, Loss. D Through backpropagation, the comprehensive loss is used to update the network parameters of the first discriminant and detection network D1 and the second discriminant and detection network D2. The comprehensive loss function is Loss. D The definition is as follows:

[0144]

[0145] Where, ω i These are the weighting coefficients of the corresponding loss function, used to adjust the contribution of each sub-loss to the total loss.

[0146] This optimization strategy combines the ability to distinguish between fake and real images and to detect changes, thereby ensuring a balance between generation quality and change detection performance between the discrimination and detection network and the image transformation network, ultimately improving the overall effect of change detection in multi-resolution, multi-source remote sensing images.

[0147] Step 106: Using the trained image conversion network and discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo-optical image, and the pseudo-optical image and the original optical image are input together into the first discrimination and detection network to obtain the final change detection result.

[0148] Using the trained image transformation network and discrimination and detection network, this application achieves change detection between low-resolution SAR images and optical images. The specific process is as follows:

[0149] First, the original low-resolution SAR image I A The image is input into the first image conversion network G1 to generate a pseudo-optical image I. B′ Pseudo-optical image I B′ The resolution and optical features are comparable to those of a real optical image. B To maintain consistency, a pseudo-optical mapping of the original low-resolution SAR image is used.

[0150] Subsequently, the generated pseudo-optical image I B′ With the original optical image I B The images are input into the first discrimination and detection network D1. The first discrimination and detection network uses the authenticity discrimination module and the change detection module to distinguish between the fake optical image and the real optical image, and at the same time detects the change area between the two.

[0151] Finally, based on the change detection results output by the first discrimination and detection network D1, the original low-resolution SAR image I is obtained. A and optical image I BAchieve high-precision change detection of multi-resolution, multi-source remote sensing images in regions of spatiotemporal variation.

[0152] In one possible embodiment, all slices are used as test data. The low-resolution SAR image slices are input into the first image transformation network G1. Then, the output pseudo-optical image and the original optical image are input together into the first discrimination and change detection network D1 to obtain the change detection results. Finally, the detection result slices are stitched together to obtain the final change detection result of the data.

[0153] As shown in Figure 3, the change detection method of this application outperforms traditional methods in the experiment. Figure 3(a) shows the ground truth results of change detection; Figure 3(b) shows the detection results of the HPT method, where the SAR image input for the HPT method is... Figure 2 Figure 3(b) shows the image after bicubic interpolation; Figure 3(c) shows the detection results of the method of this application. By comparison, it can be found that the method of this invention has fewer missed detections and false alarms in the detection of changing regions, and significantly improves the detection accuracy compared to the HPT method.

[0154] To further verify the detection effect of the present invention, Table 1 shows the numerical comparison results of the method of the present invention with other methods. Specific evaluation indicators include overall precision (OA), F1 score (F1), expected precision (PRE), and Kappa coefficient (KC), and their calculation formulas are as follows:

[0155]

[0156] Table 1 shows the numerical results of change detection for this application and the HPT method.

[0157] Table 1

[0158] method OA F1 KC HPT 95.97 82.62 80.34 This application method 98.13 91.57 90.53

[0159] As shown in Table 1, compared with the HPT method, the method proposed in this application achieves significant improvements in all indicators. In particular, the improvement in the F1 score and Kappa coefficient indicates that the method of this application performs excellently in reducing false alarms and missed detections. Experimental results show that the multi-resolution remote sensing image change detection method proposed in this application can more effectively improve the accuracy of change detection in multi-source remote sensing images.

[0160] To achieve the above embodiments, this application also proposes a multi-resolution, multi-source remote sensing image change detection device. Figure 4 This is a schematic diagram of the structure of a multi-resolution, multi-source remote sensing image change detection device provided in an embodiment of this application. Figure 4 As shown, the device includes:

[0161] The transformation network construction and generation constraint module 100 is used to construct an image transformation network and constrain the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images through a partial generation loss function. The first image transformation network is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network is used to convert optical images into pseudo-SAR images.

[0162] The network constraint enhancement module 200 is used to downsample the real optical image and upsample the real SAR image. The processed images are input into the first and second image transformation networks respectively. Another part of the generation loss function constrains the similarity between the output of the two image transformation networks and the original image.

[0163] The discrimination and detection network construction module 300 is used to construct a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical images and real optical images, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR images and real SAR images.

[0164] The first joint optimization module 400 is used to calculate the first discriminative loss and update the parameters of the image conversion network through backpropagation based on the joint optimization strategy of multiple generation losses and the first discriminative loss.

[0165] The second joint optimization module 500 is used to calculate the second discrimination loss and the change detection loss. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation.

[0166] The final change detection module 600 is used to input the original SAR image into the first image conversion network to generate a pseudo optical image using the trained image conversion network and discrimination and detection network, and input the pseudo optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result.

[0167] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0168] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0169] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0170] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0171] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0172] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0173] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0174] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0175] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0176] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0177] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0178] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0180] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0181] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting changes in multi-resolution, multi-source remote sensing images, characterized in that, Includes the following steps: An image transformation network is constructed, and the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images is constrained by a partial generation loss function. The first image transformation network is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network is used to convert optical images into pseudo-SAR images. The real optical image is downsampled and the real SAR image is upsampled. The processed images are then input into the first and second image transformation networks, respectively. Another part of the generation loss function constrains the similarity between the outputs of the two image transformation networks and the original image. A discrimination and detection network is constructed, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical images and real optical images, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR images and real SAR images. The first discriminant loss is calculated, and the parameters of the image conversion network are updated through backpropagation based on a joint optimization strategy of multiple generation losses and the first discriminant loss. The second discrimination loss and change detection loss are calculated. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation. Using the trained image conversion network and discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo optical image, and the pseudo optical image and the original optical image are input together into the first discrimination and detection network to obtain the final change detection result; The similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images, is constrained through a partial generation loss function, including: Given dual-temporal low-resolution SAR images of different resolutions With optical images , to image Input to the first image transformation network Generate the first pseudo-optical image ,in resolution and Same; Image Input to the second image transformation network Generate the first pseudoSAR image , where the image With images They have the same resolution; The first pseudo-optical image generated Input to the second image transformation network Generate a second pseudoSAR image and the first pseudo SAR image generated Input to the first image transformation network Generate a second pseudo-optical image ; Calculate separately and The generation loss between them, and and The generation loss between the two is used to constrain the generation effects of fake and real images respectively.

2. The method according to claim 1, characterized in that, and The generation loss between them, and and The loss function formulas for the generation loss between them are as follows: in, for and The loss function between for and The loss function between them.

3. The method according to claim 2, characterized in that, The real optical image is downsampled, and the real SAR image is upsampled. The processed images are then input into the first and second image transformation networks, respectively. A separate generation loss function constrains the similarity between the outputs of the two image transformation networks and the original images, including: To further constrain the first image transformation network The generation effect uses an image processed by downsampled image As input, calculate the output image and image. The loss, its loss function for: In the formula, Represents the first image transformation network The conversion process; To further constrain the second image transformation network The generation effect uses an image processed by Interpolated image As input, calculate the output image and image. The loss, its loss function for: In the formula, Represents the second image transformation network The conversion process.

4. The method according to claim 3, characterized in that, The calculation process of the first discrimination and detection network and the first discrimination and detection network's first discrimination loss includes: The first image transformation network The first pseudo-optical image generated With the original optical image Input to the first discrimination and detection network In the process, the first pseudo-optical image is calculated. With the original optical image Discriminant loss between them, first discriminant and detection network Discriminant loss function for: in, It is an optical image label composed of 1s; The second image transformation network The first pseudo-SAR image generated Compared with the original SAR image Input to the second discrimination and detection network In the process, the first pseudo-SAR image is calculated. Compared with the original SAR image Discriminant loss between them, second discriminant and detection network Discriminant loss function for: in, A SAR image label is composed of 1s.

5. The method according to claim 4, characterized in that, Based on a joint optimization strategy combining multiple generation losses and a first discriminant loss, the parameters of the image transformation network are updated via backpropagation, including: The above loss function to The combined loss function is calculated based on the preset weight ratio. Backpropagation updates the first image transformation network With the second image transformation network Network parameters, comprehensive loss function The calculation formula is: in, These are the weighting coefficients for each loss function, used to adjust the contribution of each part of the loss to the total loss.

6. The method according to claim 5, characterized in that, The calculation process of the second discrimination loss and change detection loss of the first discrimination and detection network and the second discrimination and detection network includes: The first image transformation network The first pseudo-optical image generated With the original optical image Input value first discrimination and detection network In the process, the second discrimination loss is calculated on one hand, and the first pseudo-optical image is calculated on the other. With the original optical image The loss for detecting changes between them, where the discriminant loss function is... and change detection loss function They are respectively: in, It is an optical image label composed of 0s; The second image transformation network The first pseudo-SAR image generated Compared with the original SAR image Input value second discrimination and detection network In the process, the second discrimination loss is calculated on one hand, and the first pseudoSAR image is calculated on the other. Compared with the original SAR image The loss for detecting changes between them, where the discriminant loss function is... and change detection loss function They are respectively: in, SAR image labels are composed of 0s.

7. The method according to claim 6, characterized in that, Based on a joint optimization strategy of the first discriminant loss, the second discriminant loss, and the change detection loss, the parameters of the discriminant and detection networks are updated through backpropagation, including: The above loss function to The combined loss function is calculated based on the preset weight ratio. Backpropagation updates the first discrimination and detection network Second discrimination and detection network Network parameters, comprehensive loss function The calculation formula is: in, These are the weighting coefficients of the corresponding loss function, used to adjust the contribution of each sub-loss to the total loss.

8. A multi-resolution, multi-source remote sensing image change detection device, characterized in that, include: The transformation network construction and generation constraint module is used to construct an image transformation network and constrain the similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images through a partial generation loss function. The first image transformation network is used to convert low-resolution SAR images into pseudo-optical images, and the second image transformation network is used to convert optical images into pseudo-SAR images. The network constraint enhancement module is used to downsample real optical images and upsample real SAR images. The processed images are input into the first and second image transformation networks respectively. Another part of the generation loss function constrains the similarity between the outputs of the two image transformation networks and the original images. The discrimination and detection network construction module is used to construct the discrimination and detection network. The first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical images and real optical images, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR images and real SAR images. The first joint optimization module is used to calculate the first discriminative loss. Based on the joint optimization strategy of multiple generation losses and the first discriminative loss, the parameters of the image conversion network are updated through backpropagation. The second joint optimization module is used to calculate the second discrimination loss and the change detection loss. Based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through backpropagation. The final change detection module is used to input the original SAR image into the first image conversion network to generate a pseudo optical image using the trained image conversion network and discrimination and detection network, and input the pseudo optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result. The similarity between pseudo-optical images and real optical images, and between pseudo-SAR images and real SAR images, is constrained through a partial generation loss function, including: Given dual-temporal low-resolution SAR images of different resolutions With optical images , to image Input to the first image transformation network Generate the first pseudo-optical image ,in resolution and Same; Image Input to the second image transformation network Generate the first pseudoSAR image , where the image With images They have the same resolution; The first pseudo-optical image generated Input to the second image transformation network Generate a second pseudoSAR image and the first pseudo SAR image generated Input to the first image transformation network Generate a second pseudo-optical image ; Calculate separately and The generation loss between them, and and The generation loss between the two is used to constrain the generation effects of fake and real images respectively.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images