Multi-resolution multi-source remote sensing image change detection method

Abstract

The invention provides a multi-resolution multi-source remote sensing image change detection method, which comprises the following steps of: for a first image conversion network and a second image conversion network, constraining similarity between a pseudo optical image and a real optical image and similarity between a pseudo SAR image and a real SAR image by partially generating a loss function; performing down-sampling on a real optical image, performing up-sampling on a real SAR image, respectively inputting the processed image into the first image conversion network and the second image conversion network, and generating a loss function through the other part to constrain the similarity between the output of the two image conversion networks and the original image; the judgment and detection network is used for realizing the tasks of judgment and change detection of the generated pseudo image and the source image; and optimizing discriminator network parameters through discriminating loss and change detection loss, and finally performing change detection on the SAR image and the optical image by using the trained network to obtain a high-precision detection result. According to the invention, change detection among multi-source remote sensing images with different resolutions and different modalities can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image change detection, and in particular to a multi-resolution multi-source remote sensing image change detection method. Background Art

[0002] Remote sensing images are the key carriers for recording surface feature information. By processing and analyzing them, we can promote the development of remote sensing data processing technology towards intelligence. Large-scale surface change detection refers to identifying areas with significant changes by comparing remote sensing data of the same area at different time periods. This technology has important application value in the fields of surface environmental monitoring, disaster emergency response, urban and rural planning, etc.

[0003] With the advancement of remote sensing technology, remote sensing data from different sensors provide data resources for change detection. As different types of remote sensing images are continuously acquired, the research on change detection between multi-source remote sensing data is becoming more and more urgent. For example, optical remote sensing images and synthetic aperture radar (SAR) are the two most commonly used multi-source remote sensing images for change detection.

[0004] However, multi-source heterogeneous remote sensing images have different imaging mechanisms, and their image modalities, resolutions and other features vary greatly. For example, SAR images and high-resolution optical images usually have certain resolution differences, and SAR images are less detailed than optical remote sensing images and are more affected by noise. Current multi-source and multi-resolution remote sensing image change detection methods mainly use resampling strategies, such as bilinear bicubic interpolation, to equalize resolution differences. However, when faced with pixel-level prediction tasks such as change detection, the difference in image resolution before and after the change inevitably affects detection accuracy. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first objective of this application is to propose a multi-resolution and multi-source remote sensing image change detection method.

[0007] The second objective of the present application is to provide a multi-resolution multi-source remote sensing image change detection device.

[0008] The third objective of the present application is to provide an electronic device.

[0009] A fourth objective of the present application is to provide a computer-readable storage medium.

[0010] A fifth object of the present application is to provide a computer program product.

[0011] To achieve the above objectives, the first embodiment of the present application proposes a multi-resolution multi-source remote sensing image change detection method, comprising:

[0012] An image transformation network is constructed to constrain the similarity between the pseudo optical image and the real optical image, and between the pseudo SAR image and the real SAR image by partially generating a loss function, wherein the first image transformation network is used to transform the low-resolution SAR image into the pseudo optical image, and the second image transformation network is used to transform the optical image into the pseudo SAR image;

[0013] Down-sampling the real optical image and up-sampling the real SAR image are performed, and the processed images are input into the first and second image transformation networks respectively, and a loss function is generated by another part to constrain the similarity between the outputs of the two image transformation networks and the original image;

[0014] Constructing a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image;

[0015] Calculate the first discriminant loss, and update the parameters of the image conversion network through back propagation based on the joint optimization strategy of multiple generation losses and the first discriminant loss;

[0016] Calculate the second discriminant loss and change detection loss, and update the parameters of the discriminant and detection networks through back propagation based on the joint optimization strategy of the first discriminant loss, the second discriminant loss and the change detection loss;

[0017] Using the trained image conversion network and the discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo optical image, and the pseudo optical image and the original optical image are input into the first discrimination and detection network together to obtain the final change detection result.

[0018] Optionally, similarity between the pseudo optical image and the real optical image, and between the pseudo SAR image and the real SAR image is constrained by partially generating a loss function, including:

[0019] Given dual-phase low-resolution SAR images with different resolutions I A With optical image I B , image I A Input to the first image transformation network G1 to generate the first pseudo optical image I B′ , where I B′ The resolution and I B Same; image I BInput to the second image transformation network G2 to generate the first pseudo SAR image I A′ , where image I A′ With image I A The resolution is the same;

[0020] The first pseudo optical image I generated B′ Input to the second image transformation network G2 to generate the second pseudo SAR image I A″ , and the first pseudo SAR image I generated A′ Input to the first image transformation network G1 to generate the second pseudo optical image I B″ ;

[0021] Calculate I separately A with I A″ The generation loss between B with I B″ The generation loss between them is:

[0022] Loss1=L1(I A″ ,I A )

[0023] Loss2=L1(I B″ ,I B )

[0024] Among them, Loss1 is I A with I A″ The loss function between Loss2 and I B with I B″ The loss function between .

[0025] Optionally, down-sampling is performed on the real optical image, and up-sampling is performed on the real SAR image, and the processed images are respectively input into the first and second image transformation networks, and a loss function is generated by another part to constrain the similarity between the outputs of the two image transformation networks and the original image, including:

[0026] In order to further constrain the generation effect of the first image transformation network G1, the image I B Downsampled image I B _down as input, calculate the output image and image I B The loss function Loss3 is:

[0027] Loss3=L1(G1(I B _down),I B )

[0028] Where G1() represents the conversion process of the first image transformation network G1;

[0029] In order to further constrain the generation effect of the second image transformation network G2, the image I A Interpolated image I A _up as input, calculate the output image and image I A The loss function Loss4 is:

[0030] Loss4=L1(G2(I A _up),I A )

[0031] Where G2() represents the conversion process of the second image transformation network G2.

[0032] Optionally, the calculation process of the first discrimination loss of the first discrimination and detection network and the second discrimination and detection network includes:

[0033] The first pseudo optical image I generated by the first image transformation network G1 B′ With the original optical image I B Input into the first discrimination and detection network D1 to calculate the first pseudo optical image I B′ With the original optical image I B The discriminant loss between the first discriminant and detection network D1 is Loss5:

[0034] Loss5=L2(D1(I B ),lbl B )

[0035] Among them, lbl B is an optical image label consisting of 1;

[0036] The first pseudo SAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A Input to the second discrimination and detection network D2 to calculate the first pseudo SAR image I A′ Compared with the original SAR image I A The discriminant loss between the two networks, the discriminant loss function Loss6 of the second discriminant and detection network D2 is:

[0037] Loss6=L2(D2(I A ),lbl A )

[0038] Among them, lbl A is a SAR image label consisting of 1s.

[0039] Optionally, based on a joint optimization strategy of multiple generation losses and first discriminant losses, the parameters of the image conversion network are updated through back propagation, including:

[0040] Combine the above loss functions Loss1 to Loss6 according to the preset weight ratio to calculate the comprehensive loss function Loss G , back propagation updates the network parameters of the first image transformation network G1 and the second image transformation network G2, and the comprehensive loss function Loss G The calculation formula is:

[0041]

[0042] Among them, ω i is the weight coefficient of each loss function, which is used to adjust the contribution of each part of the loss to the total loss.

[0043] Optionally, the calculation process of the second discrimination loss and the change detection loss of the first discrimination and detection network and the second discrimination and detection network includes:

[0044] The first pseudo optical image I generated by the first image transformation network G1 B′ With the original optical image I B In the input value first discrimination and detection network D1, on the one hand, the second discrimination loss is calculated, and on the other hand, the first pseudo optical image I is calculated. B′ With the original optical image I B The change detection loss between, where the discriminant loss function Loss7 and the change detection loss function Loss8 are:

[0045] Loss7=L2(D1(I B′ ),lbl′ B )

[0046] Loss8=Focal_Loss1

[0047] Among them, lbl′ B is an optical image label consisting of 0s;

[0048] The first pseudo SAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A In the second discrimination and detection network D2, the second discrimination loss is calculated on the one hand, and the first pseudo SAR image I is calculated on the other hand. A′ Compared with the original SAR image I A The change detection loss between, where the discriminant loss function Loss9 and the change detection loss function Loss 10 They are:

[0049] Loss9=L2(D2(I A′ ),lbl′ A )

[0050] Loss 10 =Focal_Loss2

[0051] Among them, lbl′ A is the SAR image label consisting of 0s.

[0052] Optionally, based on the joint optimization strategy of the first discriminant loss, the second discriminant loss and the change detection loss, the parameters of the discriminant and detection networks are updated through back propagation, including:

[0053] The above loss function Loss5 to Loss 10 According to the preset weight ratio, calculate the comprehensive loss function Loss D , back propagation updates the network parameters of the first discriminant and detection network D1 and the second discriminant and detection network D2, and the comprehensive loss function Loss D The calculation formula is:

[0054]

[0055] Among them, ω i It is the weight coefficient of the corresponding loss function, which is used to adjust the contribution of each sub-loss to the total loss.

[0056] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a multi-resolution multi-source remote sensing image change detection device, comprising:

[0057] A conversion network construction and generation constraint module is used to construct an image conversion network, and constrain the similarity between the pseudo optical image and the real optical image, and the pseudo SAR image and the real SAR image by partially generating a loss function, wherein the first image conversion network is used to convert the low-resolution SAR image into the pseudo optical image, and the second image conversion network is used to convert the optical image into the pseudo SAR image;

[0058] Generate a network constraint enhancement module, which is used to downsample the real optical image and upsample the real SAR image, input the processed images into the first and second image transformation networks respectively, and constrain the similarity between the outputs of the two image transformation networks and the original image through another part of the generated loss function;

[0059] A discrimination and detection network construction module, used to construct a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image;

[0060] A first joint optimization module, used for calculating a first discriminant loss, updating the parameters of the image conversion network through back propagation based on a joint optimization strategy of multiple generation losses and the first discriminant loss;

[0061] The second joint optimization module is used to calculate the second discrimination loss and the change detection loss, and based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through back propagation;

[0062] The final change detection module is used to use the trained image conversion network and the discrimination and detection network to input the original SAR image into the first image conversion network to generate a pseudo optical image, and input the pseudo optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result.

[0063] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0064] The memory stores computer-executable instructions;

[0065] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any one of the first aspects.

[0066] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0067] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, which implements any method in the first aspect when executed by a processor.

[0068] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0069] It can realize the mutual conversion between multi-source remote sensing images with different resolutions and different modalities, thereby aligning the resolution of multi-source images, and then processing the change detection between multi-source remote sensing images after resolution alignment.

[0070] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0072] Figure 1 A schematic diagram of a flow chart of a multi-resolution multi-source remote sensing image change detection method provided in an embodiment of the present application;

[0073] Figure 2 A schematic diagram of a multi-source remote sensing change detection data set with different resolutions provided in an embodiment of the present application;

[0074] FIG3( a ) is a schematic diagram of a true value result of change detection provided in an embodiment of the present application;

[0075] FIG3( b) is a schematic diagram of the detection results of the HPT method provided in an embodiment of the present application;

[0076] FIG3( c ) is a schematic diagram of the detection results of the method of the present application provided in an embodiment of the present application.

[0077] Figure 4 A schematic diagram of the structure of a multi-resolution multi-source remote sensing image change detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0079] To address this problem, an embodiment of the present application provides a multi-resolution multi-source remote sensing image change detection method to achieve mutual conversion between multi-source remote sensing images of different resolutions and different modalities, thereby aligning the resolution of multi-source images, and then processing change detection between multi-source remote sensing images after resolution alignment.

[0080] It should be noted that the present application relates to a method for detecting changes in multi-resolution multi-source remote sensing images, wherein "multi-source remote sensing images" refer to image data acquired by different types of remote sensing sensors, including but not limited to optical images, SAR (synthetic aperture radar) images, multispectral images, infrared images, etc. Although optical images and SAR images are used as examples for detailed description in the embodiments of the present application, the present method is also applicable to other types of multi-source remote sensing images.

[0081] Figure 1 The following is a flow chart of a multi-resolution multi-source remote sensing image change detection method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0082] Step 101, construct an image conversion network, and constrain the similarity between the pseudo optical image and the real optical image, and the pseudo SAR image and the real SAR image by partially generating a loss function.

[0083] In the embodiment of the present application, the image conversion network includes a first image conversion network G1 and a second image conversion network G2. Among them, the first image conversion network G1 is used to convert the low-resolution SAR image into a pseudo-optical image, and the second image conversion network G2 is used to convert the optical image into a pseudo-SAR image. The constructed deep neural networks G1 and G2 can adopt the deep neural networks commonly used for underlying visual task feature extraction, and the present application does not make additional restrictions on this.

[0084] In the specific implementation process of this application, optical images and SAR images are selected as examples because they are typical combinations of multi-source remote sensing image fusion and change detection, and are more common in practical applications. However, this application is not limited to such image types. This method is applicable to any type of multi-source remote sensing image, including but not limited to multi-spectral images, thermal infrared images, LiDAR images and optical images. Users can select different types of multi-source images for change detection tasks according to actual application requirements, and this application does not make specific limitations on this.

[0085] In a possible embodiment, the dataset used in this application is a flood change detection dataset in the Gloucester area of ​​the United Kingdom. The dataset contains two types of heterogeneous remote sensing images: pre-change images: synthetic aperture radar (SAR) images, which are used to capture the surface features of the area before the flood; post-change images: normalized difference vegetation index (NDVI) images, which are used to describe the changes in vegetation after the flood. Both images were collected at different times in the same area, forming heterogeneous, multi-time, and multi-resolution change detection remote sensing image pairs.

[0086] Figure 2 (a)-2(c) are schematic diagrams of multi-source remote sensing change detection data sets with different resolutions used in the embodiments of the present application, where:

[0087] Figure 2 (a) and Figure 2 (c) shows the distribution of original SAR images and NDVI images in the change detection dataset. The original SAR images and NDVI images are remote sensing images of the same area at different times, forming heterogeneous image pairs for change detection.

[0088] Figure 2(b) The low-resolution SAR image obtained after bicubic downsampling forms a heterogeneous image pair with a high-resolution optical NDVI image with a significant difference in resolution. The resolution of the low-resolution SAR image is reduced to 1 / 8 of the original resolution, while the NDVI image maintains the original resolution.

[0089] In order to realize the change detection task of multi-resolution and multi-source remote sensing images, this application first processes the resolution of the dataset: the SAR image before the change is downsampled by 8 times using the bicubic downsampling method to form a low-resolution SAR image (such as Figure 2 (b)). The optical image (NDVI image) keeps its original resolution unchanged and forms a heterogeneous remote sensing image pair with the low-resolution SAR image.

[0090] To adapt to the subsequent deep learning model training, this application also crops the images after the above resolution processing: the low-resolution SAR image is cropped into a small block of 16×16; the optical image (NDVI) is cropped into a small block of 128×128. If the original image size is not divisible by 16 or 128, the cropping is completed by padding the right and bottom of the image with zeros. The cropped image pairs will be used as standardized input data for subsequent image conversion network and change detection network training.

[0091] Through the above-mentioned resolution preprocessing, this application realizes the standardized processing of low-resolution SAR images and high-resolution NDVI optical images, providing high-quality input data for the subsequent construction of image conversion networks, discrimination and detection network training, ensuring the accuracy and robustness of change detection tasks.

[0092] After completing multi-resolution processing and cropping, a standardized image slice data set is obtained. This application further divides the data of these cropped image slices to meet the model training requirements. The specific steps are as follows: divide the cropped image slice data into a training set and a test set; randomly select 20% of the image slices from all cropped slices as the training set, which is used to train the image conversion network first image transformation network G1 and the second image transformation network G2, as well as the discrimination and detection networks D1 and D2.

[0093] Specifically, this application proposes a change detection method based on an image conversion network and a generated loss function for dual-phase low-resolution SAR images and optical images of different resolutions. Specifically, it includes the following steps:

[0094] First, for a given dual-phase low-resolution SAR image I with different resolutions A With optical image I B , image I AInput to the first image transformation network G1 to generate the first pseudo optical image I B′ , where I B′ The resolution and I B Same; image I B Input to the second image transformation network G2 to generate the first pseudo SAR image I A′ , where image I A′ With image I A The resolution is the same.

[0095] Furthermore, the present application generates a first pseudo optical image I B′ Input to the second image transformation network G2 to generate the second pseudo SAR image I A″ At the same time, the first pseudo SAR image I A′ Input to the first image transformation network G1 to generate the second pseudo optical image I B″ Through the above two conversion operations, multiple generative relationships are formed between the pseudo optical image and the real optical image, and between the pseudo SAR image and the real SAR image.

[0096] It should be noted that the role of the generated loss function is not only to optimize the quality of a single generated image, but also to ensure the similarity of the generated pseudo image with the real image in structure and resolution through multiple conversion processes. For example, the second pseudo SAR image needs to be as close as possible to the real low-resolution SAR image in terms of texture and resolution, while the second pseudo optical image needs to be close to the real optical image in terms of optical features. Specifically, in order to constrain the generation effect, the embodiment of the present application calculates I A with I A″ The generation loss between B with I B″ The generation loss between , and the generation effect of pseudo image and real image is constrained by the following generation loss function:

[0097] Loss1=L1(I A″ ,I A )

[0098] Loss2=L1(I B″ ,I B )

[0099] Among them, Loss1 is I A with I A″ The loss function between Loss2 and I B with I B″ The loss function between .

[0100] These generation loss functions work together to ensure that the pseudo image gradually approaches the real image features during multiple image conversions. Through multi-stage generation and conversion, this application not only improves the authenticity of the pseudo image, but also provides high-quality input data for subsequent discrimination and change detection networks, significantly improving the overall performance of the change detection task.

[0101] Step 102, down-sampling the real optical image and up-sampling the real SAR image, inputting the processed images into the first and second image transformation networks respectively, and generating a loss function through another part to constrain the similarity between the outputs of the two image transformation networks and the original image.

[0102] Based on the construction of the image conversion network, in order to further optimize the similarity between the pseudo image and the real image, the present application downsamples and upsamples the real optical image and the real SAR image respectively, and inputs the processed images into the first and second image transformation networks respectively, and generates a loss function through another part to constrain the similarity between the outputs of the two image transformation networks and the original image.

[0103] First, the real optical image is downsampled to reduce its resolution from high resolution to low resolution consistent with the SAR image. At the same time, the real SAR image is upsampled to increase its resolution from low resolution to high resolution, which is the same as the optical image. Subsequently, the processed images are input into the first and second image transformation networks respectively. The first and second image transformation networks respectively realize the change of the resolution of the downsampled optical image and the upsampled SAR image after passing through the first image transformation network G1 and the second image transformation network G2 by adding downsampling layers and upsampling layers. Specifically:

[0104] An upsampling layer is added to the G1 network to ensure the improvement of image resolution, that is, to achieve the image size from low resolution to high resolution; at the same time, a downsampling layer is added to the G2 network to achieve the mapping from high resolution to low resolution. The upsampling layer can be implemented using the Pixel Shuffle layer and the deconvolution layer, and the downsampling layer can be implemented using the convolution with a step size.

[0105] Subsequently, for the generation effect of the first image conversion network G1, the present application uses the downsampled optical image I B _down as input and compare the generated image with the real optical image I B For comparison, the loss function Loss3 is calculated and generated. The formula is as follows:

[0106] Loss3=L1(G1(I B _down),I B )

[0107] Where G1() represents the conversion process of the first image transformation network G1. Loss3 is the generation loss of the first image transformation network when processing the downsampled optical image, which is used to improve the generation effect of G1 so that it can better generate the same image as the real optical image I B Similar results.

[0108] Similarly, in order to further constrain the generation effect of the second image transformation network G2, the present application uses the image I A Interpolated image I A _up as input, calculate the output image and image I A The loss function Loss4 is:

[0109] Loss4=L1(G2(I A _up),I A )

[0110] Where G2() represents the conversion process of the second image transformation network G2. Loss4 is the generation loss of the second image transformation network when processing the upsampled low-resolution SAR image, which is used to improve the generation effect of G2 so that it can better generate the same SAR image as the real SAR image I. A Similar results.

[0111] Through the above-mentioned generation losses Loss3 and Loss4, the generation effect of the image conversion network is further optimized, ensuring the quality of the generated images in terms of resolution, texture and features, and improving the accuracy of the multi-stage mapping between low-resolution SAR images and optical images.

[0112] Step 103, constructing a discrimination and detection network.

[0113] In the embodiment of the present application, a discrimination and detection network is constructed, and an authenticity discrimination and change detection mechanism is introduced between the generated pseudo image and the real image. Among them, the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image.

[0114] Through the authenticity discrimination and change detection functions of the first discriminant and detection network and the second discriminant and detection network, not only can the quality of the generated image be significantly improved, but also the performance of the image conversion network and the discriminant and detection network can be optimized through the discriminant loss and change detection loss. Ultimately, this joint constraint mechanism can ensure the coordinated optimization of the generation network and the detection network to achieve high-quality generated images and accurate change detection.

[0115] The definition and optimization process of the specific discriminant loss and change detection loss for the discriminant and detection networks will be described in detail in the subsequent steps. By jointly optimizing multiple generation losses, discriminant losses, and change detection losses, the performance of the entire network in image conversion and change detection tasks is further improved.

[0116] Step 104, calculate the first discriminant loss, and update the parameters of the image conversion network through back propagation based on the joint optimization strategy of multiple generation losses and the first discriminant loss.

[0117] In the embodiment of the present application, by calculating the first discriminant loss, the performance of the image conversion network is optimized so that the pseudo optical image generated by it is closer to the real optical image, while enhancing the learning ability of the image conversion network for image features.

[0118] During the calculation of the first discriminant loss:

[0119] For the first discrimination and detection network D1, the first pseudo optical image I generated by the first image transformation network G1 is B′ With the original optical image I B Input into the first discrimination and detection network D1 to calculate the first pseudo optical image I B′ With the original optical image I B The discriminant loss between the first discriminant and detection network D1 is Loss5:

[0120] Loss5=L2(D1(I B ),lbl B )

[0121] Among them, lbl B is an optical image label consisting of 1. Loss5 is the discrimination loss of the first discrimination and detection network D1, which measures the authenticity between the pseudo optical image and the real optical image.

[0122] For the second discrimination and detection network D2, the first pseudo SAR image I generated by the second image transformation network G2 is A′ Compared with the original SAR image I A Input to the second discrimination and detection network D2 to calculate the first pseudo SAR image I A′ Compared with the original SAR image I A The discriminant loss between the two networks, the discriminant loss function Loss6 of the second discriminant and detection network D2 is:

[0123] Loss6=L2(D2(I A ),lbl A )

[0124] Among them, lbl AIt is a SAR image label consisting of 1. Loss6 is the discrimination loss of the second discrimination and detection network D2, which measures the authenticity between the pseudo-SAR image and the real SAR image.

[0125] In order to further optimize the performance of the image conversion network, the above loss functions Loss1 to Loss6 are combined according to the preset weight ratio to calculate the comprehensive loss function Loss G , update the network parameters of the first image transformation network G1 and the second image transformation network G2 through back propagation. Comprehensive loss function Loss G The calculation formula is:

[0126]

[0127] Among them, ω i is the weight coefficient of each loss function, which is used to adjust the contribution of each part of the loss to the total loss.

[0128] Step 105, calculate the second discrimination loss and the change detection loss, and update the parameters of the discrimination and detection networks through back propagation based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss.

[0129] Then train the discriminator, and the second discriminant loss and change detection loss of the first discriminant and detection network and the second discriminant and detection network are calculated by the following steps:

[0130] First, the first pseudo optical image I generated by the first image transformation network G1 is B′ With the original optical image I B In the input value first discrimination and detection network D1, on the one hand, the second discrimination loss is calculated, and on the other hand, the first pseudo optical image I is calculated. B′ With the original optical image I B Specifically, the discriminant loss function Loss7 and the change detection loss function Loss8 are:

[0131] Loss7=L2(D1(I B′ ),lbl′ B )

[0132] Loss8=Focal_Loss1

[0133] Among them, lbl′ B is an optical image label consisting of 0s.

[0134] Loss7 is the discriminant loss of the first discriminant and detection network D1, which is used to measure the pseudo optical image I generated by the first image conversion network G1. B′ and the real optical image I BThe classification accuracy between .

[0135] Loss8 is the change detection loss of the first discriminant and detection network D1, which is used to measure the pseudo optical image I B′ With the real optical image I B Detection performance on changing regions.

[0136] Similarly, the first pseudo SAR image I generated by the second image transformation network G2 is A′ Compared with the original SAR image I A′ In the second discrimination and detection network D2, the second discrimination loss is calculated on the one hand, and the first pseudo SAR image I is calculated on the other hand. A′ Compared with the original SAR image I A′ Specifically, the discriminant loss function Loss9 and the change detection loss function Loss 10 They are:

[0137] Loss9=L2(D2(I A′ ),lbl′ A )

[0138] Loss 10 =Focal_Loss2

[0139] Among them, lbl′ A is the SAR image label consisting of 0s.

[0140] Loss9 is the discriminant loss of the second discriminant and detection network D2. It measures the pseudo SAR image I generated by the second image conversion network G2. A′ Compared with the real SAR image I A The classification accuracy between .

[0141] Loss 10 is the change detection loss of the second discriminant and detection network D2, which measures the pseudo SAR image I generated by the second image conversion network G2 A′ Compared with the real SAR image I A Detection performance on changing regions.

[0142] Through the above loss function Loss7 to Loss 10 , to optimize the authenticity discrimination and change region detection of the generated image. These losses work together to train the first discrimination and detection network and the second discrimination and detection network, thereby improving the change detection performance of the model on multi-resolution and multi-source remote sensing images.

[0143] In order to further optimize the performance of the discrimination and detection network, the embodiment of the present application sets all loss functions Loss5 to Loss6 of the discrimination and detection network to10 Combine according to the preset weight ratio to form a comprehensive loss function Loss D Through back propagation, the comprehensive loss is used to update the network parameters of the first discrimination and detection network D1 and the second discrimination and detection network D2. Comprehensive loss function Loss D is defined as follows:

[0144]

[0145] Among them, ω i It is the weight coefficient of the corresponding loss function, which is used to adjust the contribution of each sub-loss to the total loss.

[0146] This optimization strategy jointly constrains the authenticity discrimination and change detection capabilities of pseudo images and real images, thereby ensuring a balance between the generation quality and change detection performance of the discrimination and detection network and the image conversion network, ultimately improving the overall effect of change detection in multi-resolution and multi-source remote sensing images.

[0147] Step 106, using the trained image conversion network and the discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo optical image, and the pseudo optical image and the original optical image are input into the first discrimination and detection network to obtain the final change detection result.

[0148] Using the trained image conversion network and the discrimination and detection network, this application realizes the change detection between low-resolution SAR images and optical images. The specific process is as follows:

[0149] First, the original low-resolution SAR image I A Input to the first image conversion network G1 to generate a pseudo optical image I B′ . Pseudo-optical image I B′ The resolution and optical characteristics of the real optical image I B Remain consistent as a pseudo-optical mapping of the original low-resolution SAR image.

[0150] Then, the generated pseudo optical image I B′ With the original optical image I B The first discrimination and detection network D1 performs authenticity discrimination on the pseudo optical image and the real optical image through the authenticity discrimination module and the change detection module, and detects the change area between the two.

[0151] Finally, based on the change detection results output by the first discrimination and detection network D1, the original low-resolution SAR image I is obtained. A and optical image I BHigh-precision change detection of multi-resolution and multi-source remote sensing images is achieved in changing areas in time and space.

[0152] In a possible embodiment, all slices are used as test data, low-resolution SAR image slices are input into the first image transformation network G1, and then the output pseudo-optical image and the original optical image are input into the first discrimination and change detection network D1 to obtain the change detection result, and finally the detection result slices are spliced ​​to obtain the final change detection result of the data.

[0153] As shown in Figure 3, the change detection method of the present application performs better than the traditional method in the experiment. Figure 3(a) is the true value result of change detection; Figure 3(b) is the detection result of the HPT method, where the SAR image input of the HPT method is Figure 2 (b) is the image processed by bicubic interpolation; Figure 3(c) is the detection result of the method of the present application. By comparison, it can be found that the method of the present invention has fewer missed detections and false alarms in the detection of change areas, and significantly improves the detection accuracy compared with the HPT method.

[0154] To further verify the detection effect of the present invention, Table 1 shows the numerical comparison results of the method of the present invention and other methods. The specific evaluation indicators include overall accuracy (OA), F1 value (F1), expected accuracy (PRE) and Kappa coefficient (KC), and the calculation formula is as follows:

[0155]

[0156] Table 1 shows the numerical results of change detection of this application and the HPT method.

[0157] Table 1

[0158] method OA F1 KC HPT 95.97 82.62 80.34 This application method 98.13 91.57 90.53

[0159] As can be seen from Table 1, compared with the HPT method, the method of this application has achieved significant improvements in all indicators. In particular, the improvement of the F1 value and the Kappa coefficient shows that the method of this application performs well in reducing missed detections and false alarms. The experimental results show that the multi-resolution remote sensing image change detection method proposed in this application can more effectively improve the accuracy of multi-source remote sensing image change detection.

[0160] In order to implement the above embodiment, the present application also proposes a multi-resolution multi-source remote sensing image change detection device. Figure 4 The structure diagram of a multi-resolution multi-source remote sensing image change detection device provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the device comprises:

[0161] The conversion network construction and generation constraint module 100 is used to construct an image conversion network, and constrain the similarity between the pseudo optical image and the real optical image, and the pseudo SAR image and the real SAR image by partially generating a loss function, wherein the first image conversion network is used to convert the low-resolution SAR image into the pseudo optical image, and the second image conversion network is used to convert the optical image into the pseudo SAR image;

[0162] Generate a network constraint enhancement module 200, which is used to downsample the real optical image and upsample the real SAR image, input the processed images into the first and second image transformation networks respectively, and constrain the similarity between the outputs of the two image transformation networks and the original image through another part of the generated loss function;

[0163] The discrimination and detection network construction module 300 is used to construct a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image;

[0164] A first joint optimization module 400 is used to calculate a first discriminant loss, and update the parameters of the image conversion network through back propagation based on a joint optimization strategy of multiple generation losses and the first discriminant loss;

[0165] A second joint optimization module 500 is used to calculate the second discrimination loss and the change detection loss, and update the parameters of the discrimination and detection networks through back propagation based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss;

[0166] The final change detection module 600 is used to use the trained image conversion network and the discrimination and detection network to input the original SAR image into the first image conversion network to generate a pseudo optical image, and input the pseudo optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result.

[0167] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0168] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0169] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0170] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0171] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0172] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0173] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0174] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0175] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0176] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0177] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0178] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0180] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0182] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multi-resolution multi-source remote sensing image change detection method, characterized in that: The following steps are involved: An image transformation network is constructed to constrain the similarity between the pseudo optical image and the real optical image, and between the pseudo SAR image and the real SAR image by partially generating a loss function, wherein the first image transformation network is used to transform the low-resolution SAR image into the pseudo optical image, and the second image transformation network is used to transform the optical image into the pseudo SAR image; Down-sampling the real optical image and up-sampling the real SAR image are performed, and the processed images are input into the first and second image transformation networks respectively, and a loss function is generated by another part to constrain the similarity between the outputs of the two image transformation networks and the original image; Constructing a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image; Calculate the first discriminant loss, and update the parameters of the image conversion network through back propagation based on the joint optimization strategy of multiple generation losses and the first discriminant loss; Calculate the second discriminant loss and change detection loss, and update the parameters of the discriminant and detection networks through back propagation based on the joint optimization strategy of the first discriminant loss, the second discriminant loss and the change detection loss; Using the trained image conversion network and the discrimination and detection network, the original SAR image is input into the first image conversion network to generate a pseudo optical image, and the pseudo optical image and the original optical image are input into the first discrimination and detection network together to obtain the final change detection result.

2. The method according to claim 1, characterized in that The similarity between the pseudo optical image and the real optical image, the pseudo SAR image and the real SAR image is constrained by partially generating loss functions, including: Given dual-phase low-resolution SAR images with different resolutions I A With optical image I B , image I A Input to the first image transformation network G1 to generate the first pseudo optical image I B′ , where I B′ The resolution and I B Same; image I B Input to the second image transformation network G2 to generate the first pseudo SAR image I A′ , where image I A′ With image I A The resolution is the same; The first pseudo optical image I generated B′ Input to the second image transformation network G2 to generate the second pseudo SAR image I A″ , and the first pseudo SAR image I generated A′ Input to the first image transformation network G1 to generate the second pseudo optical image I B″ ; Calculate I separately A with I A″ The generation loss between B with I B″ The generation loss between them is: Loss1=L1(I A″ ,I A ) Loss2=L1(I B″ ,I B ) Among them, Loss1 is I A with I A″ The loss function between Loss2 and I B with I B″ The loss function between .

3. The method according to claim 2, characterized in that The real optical image is downsampled, and the real SAR image is upsampled. The processed images are input into the first and second image transformation networks respectively, and the loss function is generated by another part to constrain the similarity between the output of the two image transformation networks and the original image, including: In order to further constrain the generation effect of the first image transformation network G1, the image I B Downsampled image I B _down as input, calculate the output image and image I B The loss function Loss3 is: Loss3=L1(G1(I B _down),I B ) Where G1() represents the conversion process of the first image transformation network G1; In order to further constrain the generation effect of the second image transformation network G2, the image I A Interpolated image I A _up as input, calculate the output image and image I A The loss function Loss4 is: Loss4=L1(G2(I A _up),I A ) Where G2() represents the conversion process of the second image transformation network G2.

4. The method according to claim 3, characterized in that The calculation process of the first discriminant loss of the first discriminant and detection network and the second discriminant and detection network includes: The first pseudo optical image I generated by the first image transformation network G1 B′ With the original optical image I B Input into the first discrimination and detection network D1 to calculate the first pseudo optical image I B′ With the original optical image I B The discriminant loss between the first discriminant and detection network D1 is Loss5: <h2 style=";text-align:left;direction:ltr">Loss5 = L2(D1(I<h2 style=";text-align:left;direction:ltr"> B <h2 style=";text-align:left;direction:ltr"> ),lbl<h2 style=";text-align:left;direction:ltr"> B <h2 style=";text-align:left;direction:ltr"> ) Among them, lbl B is an optical image label consisting of 1; The first pseudo SAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A Input to the second discrimination and detection network D2 to calculate the first pseudo SAR image I A′ Compared with the original SAR image I A The discriminant loss between the two networks, the discriminant loss function Loss6 of the second discriminant and detection network D2 is: <h2 style=";text-align:left;direction:ltr">Loss6 = L2(D2(I<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> ),lbl<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> ) Among them, lbl A is a SAR image label consisting of 1s.

5. The method according to claim 4, characterized in that Based on the joint optimization strategy of multiple generation losses and first discriminant losses, the parameters of the image conversion network are updated through back propagation, including: Combine the above loss functions Loss1 to Loss6 according to the preset weight ratio to calculate the comprehensive loss function Loss G , back propagation updates the network parameters of the first image transformation network G1 and the second image transformation network G2, and the comprehensive loss function Loss G The calculation formula is: Among them, ω i is the weight coefficient of each loss function, which is used to adjust the contribution of each part of the loss to the total loss.

6. The method according to claim 5, characterized in that The calculation process of the first discriminant and detection network and the second discriminant loss and the change detection loss of the second discriminant and detection network includes: The first pseudo optical image I generated by the first image transformation network G1 B′ With the original optical image I B In the input value first discrimination and detection network D1, on the one hand, the second discrimination loss is calculated, and on the other hand, the first pseudo optical image I is calculated. B′ With the original optical image I B The change detection loss between, where the discriminant loss function Loss7 and the change detection loss function Loss8 are: <h2 style=";text-align:left;direction:ltr">Loss7 = L2(D1(I<h2 style=";text-align:left;direction:ltr"> B′ <h2 style=";text-align:left;direction:ltr"> ),lbl′<h2 style=";text-align:left;direction:ltr"> B <h2 style=";text-align:left;direction:ltr"> ) Loss8=Focal_Loss1 Among them, lbl′ B is an optical image label consisting of 0s; The first pseudo SAR image I generated by the second image transformation network G2 A′ Compared with the original SAR image I A In the second discrimination and detection network D2, the second discrimination loss is calculated on the one hand, and the first pseudo SAR image I is calculated on the other hand. A′ Compared with the original SAR image I A The change detection loss between, where the discriminant loss function Loss9 and the change detection loss function Loss 10 They are: <h2 style=";text-align:left;direction:ltr">Loss9 = L2(D2(I<h2 style=";text-align:left;direction:ltr"> A′ <h2 style=";text-align:left;direction:ltr"> ),lbl′<h2 style=";text-align:left;direction:ltr"> A <h2 style=";text-align:left;direction:ltr"> ) Loss 10 =Focal_Loss2 Among them, lbl′ A is the SAR image label consisting of 0s.

7. The method according to claim 6, characterized in that Based on the joint optimization strategy of the first discriminant loss, the second discriminant loss and the change detection loss, the parameters of the discriminant and detection networks are updated through back propagation, including: The above loss function Loss5 to Loss 10 According to the preset weight ratio, calculate the comprehensive loss function Loss D , back propagation updates the network parameters of the first discriminant and detection network D1 and the second discriminant and detection network D2, and the comprehensive loss function Loss D The calculation formula is: Among them, ω i It is the weight coefficient of the corresponding loss function, which is used to adjust the contribution of each sub-loss to the total loss.

8. A multi-resolution multi-source remote sensing image change detection device, characterized in that: include: A conversion network construction and generation constraint module is used to construct an image conversion network, and constrain the similarity between the pseudo optical image and the real optical image, and the pseudo SAR image and the real SAR image by partially generating a loss function, wherein the first image conversion network is used to convert the low-resolution SAR image into the pseudo optical image, and the second image conversion network is used to convert the optical image into the pseudo SAR image; Generate a network constraint enhancement module, which is used to downsample the real optical image and upsample the real SAR image, input the processed images into the first and second image transformation networks respectively, and constrain the similarity between the outputs of the two image transformation networks and the original image through another part of the generated loss function; A discrimination and detection network construction module, used to construct a discrimination and detection network, wherein the first discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo optical image and the real optical image, and the second discrimination and detection network is used to perform authenticity discrimination and change detection on the generated pseudo SAR image and the real SAR image; A first joint optimization module, used for calculating a first discriminant loss, updating the parameters of the image conversion network through back propagation based on a joint optimization strategy of multiple generation losses and the first discriminant loss; The second joint optimization module is used to calculate the second discrimination loss and the change detection loss, and based on the joint optimization strategy of the first discrimination loss, the second discrimination loss and the change detection loss, the parameters of the discrimination and detection networks are updated through back propagation; The final change detection module is used to use the trained image conversion network and the discrimination and detection network to input the original SAR image into the first image conversion network to generate a pseudo optical image, and input the pseudo optical image and the original optical image into the first discrimination and detection network to obtain the final change detection result.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images