A method and device for reconstructing missing information of a remote sensing image under radiation decoupling

Abstract

This invention discloses a method and apparatus for reconstructing missing information in remote sensing images under radiometric decoupling, which can effectively reconstruct information loss in remote sensing images caused by cloud and cloud shadow occlusion and sensor failure. First, the radiometric information recorded in the remote sensing image is decoupled into intrinsic content radiometric components C, which are related to surface knowledge, and imaging condition radiometric components A and B, which are related to the imaging environment, through an autoencoder network. Then, a radiometric decoupling model is constructed based on a radiometric correction model, and a remote sensing image missing information decoupling and reconstruction network, DecRecNet, is designed. This network uses intrinsic content radiometric consistency penalty to protect the image's content information, imaging radiometric penalty and imaging condition smoothness loss to coordinate the image imaging environment, and imaging radiometric guidance and semantic guidance in the radiometric guidance module to perform targeted radiometric adjustments on the foreground and background images respectively. Finally, based on the constructed one-to-one paired missing information reconstruction dataset, the missing information is reconstructed.

Description

Technical Field

[0001] This invention relates to the field of remote sensing technology, and in particular to a method and apparatus for reconstructing missing information in remote sensing images under radiation decoupling. Background Technology

[0002] Optical remote sensing imagery boasts advantages such as wide swath width and high resolution, making it a crucial tool for Earth observation. However, optical satellites are passive remote sensing systems, with imaging bands ranging from 0.38 to 0.76 μm. Satellite sensors are often affected by atmospheric conditions (fog, clouds, etc.) and actual operating conditions, resulting in information loss in imagery due to cloud cover, cloud shadows, and sensor malfunctions. Regarding cloud cover, statistics show that 35% of the global land area is covered by clouds annually. Sensor malfunctions causing image information loss are most notably seen in Landsat-7 imagery, where, since 2003, each Landsat-7 image has exhibited a 22% scan gap, severely impacting image usability. Therefore, reconstructing information from missing locations in optical remote sensing imagery is of significant importance and practical value for improving the data utilization rate of remote sensing images.

[0003] In recent years, thanks to the powerful nonlinear expression and generalization capabilities of deep learning, deep learning methods have been gradually applied in fields such as super-resolution reconstruction of optical remote sensing images and reconstruction of missing information in remote sensing images. However, existing methods suffer from the technical problem of poor reconstruction results. Summary of the Invention

[0004] This invention provides a method for reconstructing missing information in remote sensing images under radiation decoupling, which solves or at least partially solves the technical problem of poor reconstruction effect in the prior art.

[0005] To address the aforementioned technical problems, the first aspect of this invention provides a method for reconstructing missing information in remote sensing images under radiation decoupling, comprising:

[0006] S1: Obtain the image to be reconstructed and the reference image, and extract the missing mask from the image to be reconstructed;

[0007] S2: Based on the image to be reconstructed, the reference image, and the extracted missing mask, a synthetic image is obtained, and a dataset is constructed based on the synthetic image and the missing mask, and a training dataset is divided.

[0008] S3: Construct DecRecNet, a remote sensing image missing information decoupling and reconstruction network. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and the semantic feature guidance map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain the imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain the semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric guidance feature map, and the semantic guidance feature map. The result output module is used to calculate the intrinsic content components and imaging conditional components of the reconstructed output image based on the radiometric decoupling model to obtain the reconstructed image. The input data includes the synthetic image and the missing mask.

[0009] S4: Use the training dataset to train the constructed remote sensing image missing information decoupling and reconstruction network to obtain the trained remote sensing image missing information decoupling and reconstruction network.

[0010] S5: Use the decoupling and reconstruction network with the missing information of the trained remote sensing image to reconstruct the image to be reconstructed.

[0011] In one implementation, step S1, extracting the missing mask from the image to be reconstructed, includes:

[0012] To address the information loss problem caused by clouds and cloud shadows, the Perlin noise method is used to simulate clouds and cloud shadows to obtain cloud loss masks.

[0013] To address the information loss problem caused by sensor malfunction, a stripe loss mask is obtained by extracting the actual missing stripes.

[0014] The acquired cloud-deficient mask and stripe-deficient mask are used as the deficient mask M, where the area with a pixel value of 1 represents the deficient area and the area with a pixel value of 0 represents the non-deficient area.

[0015] In one embodiment, step S2, which involves obtaining a synthesized image based on the image to be reconstructed, a reference image, and an extracted missing mask, includes:

[0016] Background image I is obtained from the image to be reconstructed and the missing mask. src ·(1-M), where I src M represents the image to be reconstructed, and M is the missing mask.

[0017] Foreground image I is obtained from reference image and missing mask. T ·M, where I T For reference image;

[0018] Based on the acquired background and foreground images, a composite image I is obtained. src_T I src_T =I src ·(1-M)+I T ·M.

[0019] In one implementation, in the radiation guidance module of the network in step S3, the imaging guide extracts the foreground and background regions in the feature map through masking operations, calculates the mean and variance of the corresponding regions, and then uses a transformation technique to transfer the radiation conditions from the background to the foreground, eliminating the radiation difference between the foreground and background, and obtaining the imaging radiation guidance feature map; the semantic guide continues to encode the imaging radiation guidance feature map obtained by the imaging guide, and constrains the same type of ground features in the foreground and background through semantic constraint radiation consistency penalty, and obtains the semantic guidance feature map.

[0020] In one implementation, the intrinsic content radiometric correction module of the network in step S3 takes the input data and semantic feature guidance map as input, performs semantic constraints on similar ground features in the foreground and background, and obtains the intrinsic content components of the reconstructed output image.

[0021] In one implementation, the imaging radiometric correction module of the network in step S3 takes the input data, the imaging radiometric guidance feature map, and the semantic guidance feature map as inputs to correct the radiometric differences in the foreground and background, and obtains the imaging conditional components of the reconstructed output image, including gain and offset.

[0022] In one embodiment, the method further includes using the following overall loss function during the training process in step S4:

[0023]

[0024] Where l is the overall loss function, l content The intrinsic content radiation consistency penalty function, Let l be the imaging radiation penalty function. AB Let be the imaging conditional radiation smoothing loss function. Let l be the radiation uniformity penalty function. rec The loss function for reconstructing missing image information is defined by λ1, λ2, λ3, λ4, and λ5, which are corresponding weighting factors used to balance the contributions of different losses. The calculation formulas for each loss function are as follows:

[0025]

[0026] C src For the intrinsic content components of the missing image, C out The intrinsic content components of the reconstructed output image. The gradient of the intrinsic content components of the reconstructed output image. The gradient of the missing image; Represents the encoder or decoder in the neural network computation process;

[0027]

[0028] A out B out For the conditional components of the reconstructed output image, I src Let E be the image to be reconstructed, i.e. the missing image, and E be the identity matrix.

[0029]

[0030] The gradient of the conditional component of the reconstructed output image.

[0031]

[0032] S is a similarity function. srcdown Ω(f) is the feature map obtained after downsampling, f is the feature map input to the imaging-guided radiation guide, and Ω(f) is the feature map obtained after processing by the imaging condition transfer module in the imaging-guided radiation guide.

[0033]

[0034] I out The image output after reconstruction. These represent the gradients of the reconstructed image and the image to be reconstructed, respectively.

[0035] Based on the same inventive concept, a second aspect of the present invention provides a device for reconstructing missing information in remote sensing images under radiation decoupling, comprising:

[0036] The image acquisition module is used to acquire the image to be reconstructed and the reference image, and to extract the missing mask from the image to be reconstructed.

[0037] The dataset construction module is used to obtain a synthetic image based on the image to be reconstructed, the reference image, and the extracted missing mask, and to construct a dataset based on the synthetic image and the missing mask, and to divide the training dataset.

[0038] The network construction module is used to construct the DecRecNet network for decoupling and reconstructing missing information in remote sensing images. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and semantic feature map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain an imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain a semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric feature map, and the semantic guidance feature map. The result output module is used to calculate the intrinsic content components and imaging conditional components of the reconstructed output image based on the radiometric decoupling model to obtain the reconstructed image. The input data includes the synthetic image and the missing mask.

[0039] The network training module is used to train the constructed remote sensing image missing information decoupling and reconstruction network using the training dataset, and obtain the trained remote sensing image missing information decoupling and reconstruction network.

[0040] The image reconstruction module is used to reconstruct the image to be reconstructed by decoupling the reconstruction network using the missing information of the trained remote sensing image.

[0041] Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in the first aspect.

[0042] Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0043] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0044] (1) This invention proposes a novel method for reconstructing missing information in remote sensing images. A decoupled reconstruction network for missing information in remote sensing images is constructed. From the perspectives of the remote sensing imaging object and the imaging process, the radiometric information of remote sensing images is modeled as intrinsic content radiometrics and imaging conditional radiometrics. By making targeted radiometric adjustments to each, multi-temporal image restoration is achieved, improving the image reconstruction effect.

[0045] (2) To address the differences in radiometric properties of the same type of land cover in the foreground and background of the synthetic image, as well as the differences caused by imaging conditions, a radiometric guidance module was designed, comprising an imaging radiometric guide and a semantic guide. For radiometric differences caused by imaging conditions, the imaging guide automatically learns the external radiometric conditions and then transfers the external radiometric conditions of the background to the foreground. The semantic guide learns the semantic radiometric consistency relationship between the foreground and background of the same type of land cover from the synthetic image, adaptively guiding the reconstruction of local image information. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an overall structure diagram of the DecRecNet model constructed in this embodiment of the invention;

[0048] Figure 2 This is a flowchart of the reconstruction steps based on missing information from DecRecNet in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the imaging conditional radiation guide in an embodiment of the present invention;

[0050] Figure 4 This refers to the reconstruction results on Landsat-8, Gaofen-1, and Landsat-7 data in this embodiment of the invention;

[0051] Figure 5 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0053] This invention leverages the advantages of deep learning, treating the image to be reconstructed as the background and multi-temporal images (reference images) as the foreground. The reasons for the radiometric differences between the foreground and background images are summarized into two parts: (1) Intrinsic content radiometric differences. Intrinsic content radiometrics represents the reflectance information of the land cover itself, containing information such as land cover type and spatial distribution. Due to differences in phenological conditions, the same land cover may exhibit radiometric differences in remote sensing images at different temporal phases, which will cause intrinsic content differences between the foreground and background images; (2) Imaging condition radiometric differences. Imaging condition radiometric differences include image radiometric deviations caused by factors such as sensor, solar altitude angle, and atmospheric conditions.

[0054] This invention, starting from the imaging mechanism of remote sensing images, provides a novel remote sensing image missing information decoupling and reconstruction network, DecRecNet (Decoupling Reconstruction Network). For intrinsic content differences, it utilizes intrinsic content radiometric consistency penalties to perform local radiometric correction on foreground and background images, preserving content information and eliminating radiometric differences between features at edges. For imaging condition radiometric differences, based on the radiometric continuity assumption, it adjusts imaging conditions using imaging radiometric penalties and imaging condition smoothness losses. For radiometric differences between imaging conditions and similar features in the foreground and background, a radiometric guidance module is used to perform targeted overall radiometric adjustments on the foreground and background images.

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] This invention provides a method for reconstructing missing information in remote sensing images under radiometric decoupling, comprising:

[0058] S1: Obtain the image to be reconstructed and the reference image, and extract the missing mask from the image to be reconstructed;

[0059] S2: Based on the image to be reconstructed, the reference image, and the extracted missing mask, a synthetic image is obtained, and a dataset is constructed based on the synthetic image and the missing mask, and a training dataset is divided.

[0060] S3: Construct DecRecNet, a remote sensing image missing information decoupling and reconstruction network. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and the semantic feature guidance map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain the imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain the semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric feature map, and the semantic feature guidance map. The result output module is used to calculate the intrinsic content components and imaging conditional components of the reconstructed output image based on the radiometric decoupling model to obtain the reconstructed image. The input data includes the synthetic image and the missing mask.

[0061] S4: Use the training dataset to train the constructed remote sensing image missing information decoupling and reconstruction network to obtain the trained remote sensing image missing information decoupling and reconstruction network.

[0062] S5: Use the decoupling and reconstruction network with the missing information of the trained remote sensing image to reconstruct the image to be reconstructed.

[0063] In summary, this invention discloses a method for reconstructing missing information in remote sensing images under radiometric decoupling, which can effectively reconstruct information loss caused by clouds and cloud shadows and sensor malfunctions in remote sensing images. This invention is achieved through the following technical solution: First, the radiometric information recorded in the remote sensing image is decoupled into intrinsic content radiometric components C, which are related to surface knowledge, and imaging condition radiometric components A and B, which are related to the imaging environment, through an autoencoder network. Then, a radiometric decoupling model is constructed based on a radiometric correction model, and a remote sensing image missing information decoupling and reconstruction network, DecRecNet (Decoupling Reconstruction Network), is designed. This network uses intrinsic content radiometric consistency penalty to protect the image's content information, imaging radiometric penalty and imaging condition smoothness loss to coordinate the image imaging environment, and imaging radiometric guidance and semantic guidance in the radiometric guidance module to perform targeted radiometric adjustments on the foreground and background images respectively. Finally, based on the constructed one-to-one paired missing information reconstruction dataset, the missing information is reconstructed.

[0064] The main concept of this invention is as follows:

[0065] First, a radiometric decoupling model is constructed. During the imaging process, remote sensing images are affected by imaging conditions such as sensor, solar elevation angle, illumination, and atmospheric conditions, resulting in differences between the measured values ​​of the image and the true reflectance values ​​of the target objects. Based on the radiometric correction model Y = A·X + B (where A and B are the gain and offset related to imaging conditions, and X and Y represent the images before and after radiometric correction, respectively), the radiometric decoupling module forces the encoder to convert the remote sensing image I during training. src Decoupling is performed on the intrinsic content-related component C, and the imaging condition-related components A and B, for the synthetic image I. src_T Then we can get I src_T =A src_T ·C src_T +B src_T .

[0066] In the dataset construction phase, the Perlin noise algorithm is used to simulate the information loss caused by clouds and cloud shadows to obtain cloud loss masks. Since sensor failures have certain regularities, strip loss is obtained by extracting real missing information. In this invention, cloud loss masks and strip loss masks are uniformly represented as M. In M, the area with a pixel value of 1 is the area to be reconstructed (foreground), and the area with a pixel value of 0 is the non-reconstructed area (background). For paired images to be reconstructed and multi-temporal reference images, Landsat-8 30m, GaoFen-1 2m, and Landsat-7 30m paired data are obtained according to the principle of time as close as possible.

[0067] During the training phase, our goal is to acquire a reconstructed image with harmonious foreground and background images. For the intrinsic content correction module, the goal of the intrinsic content radiometric correction module is to achieve the reconstructed image I... src and reference image I T In the latent space, the intrinsic features of land features are made as similar as possible, thus the assumption of the intrinsic content radiometric correction module can be obtained: C src ≈C src_T Our goal is to make the intrinsic content of the output image as close as possible to the intrinsic content of the real image. Therefore, the assumptions of the intrinsic content module can be further summarized as C. src ≈C src_T ≈C out In the network, the gradient approximation is used to represent the intrinsic content radiation consistency penalty function, thereby achieving quantitative constraints and protection of image texture features. For the imaging conditional radiometric correction module, we hope that I is the gradient; out and I srcThe imaging radiation is consistent. If we assume that the image's intrinsic content is empty, it can be represented by the identity matrix E. In this case, the imaging conditional radiation can directly represent the image. We force the imaging condition A by adding an imaging radiation loss. out B out with I src Consistency; for the differences in imaging conditions and radiation of the same type of land cover in the foreground and background images, the imaging radiation guide and semantic guide in the radiation guidance module are used to perform radiation guidance adjustment on the foreground and background of the synthetic image respectively.

[0068] During the testing phase, based on the trained DecRecNet model, the input image I to be reconstructed is... src Multi-temporal reference image I T And mask M, output the reconstructed image I out .

[0069] Please see Figure 1 and Figure 2 ,in Figure 1 This is an overall structure diagram of the DecRecNet model constructed in this embodiment of the invention; Figure 2 This is a flowchart of the reconstruction steps based on missing information from DecRecNet in an embodiment of the present invention.

[0070] In one implementation, step S1, extracting the missing mask from the image to be reconstructed, includes:

[0071] To address the information loss problem caused by clouds and cloud shadows, the Berlin noise method is used to simulate clouds and cloud shadows to obtain cloud loss masks.

[0072] To address the information loss problem caused by sensor malfunction, a stripe loss mask is obtained by extracting the actual missing stripes.

[0073] The acquired cloud-deficient mask and stripe-deficient mask are used as the deficient mask M, where the area with a pixel value of 1 represents the deficient area and the area with a pixel value of 0 represents the non-deficient area.

[0074] Specifically, the Berlin noise method is the Perlin noise algorithm.

[0075] In one embodiment, step S2, which involves obtaining a synthesized image based on the image to be reconstructed, a reference image, and an extracted missing mask, includes:

[0076] Background image I is obtained from the image to be reconstructed and the missing mask. src ·(1-M), where I src M represents the image to be reconstructed, and M is the missing mask.

[0077] Foreground image I is obtained from reference image and missing mask. T·M, where I T For reference image;

[0078] Based on the acquired background and foreground images, a composite image I is obtained. src_T I src_T =I src ·(1-M)+I T ·M.

[0079] Specifically, using the WHU Cloud Dataset as a reference, and adding image data from Liaoning, Shandong, Henan and other regions (based on the principle of being as close as possible in time), we constructed paired datasets of Landsat-8 30m, GaoFen-1 2ms and Landsat-7 30m images to be reconstructed and multi-temporal reference images.

[0080] To facilitate subsequent training, the image to be reconstructed I src Multi-temporal image reference image I T The missing mask M is cropped to a size of 256×256 to obtain I. src I T A training dataset with a one-to-one correspondence between M and I. Then, based on this, a synthetic image is obtained, and M and I are... src_T As input to the model.

[0081] Figure 4 These are the reconstruction results on Landsat-8, Gaofen-1, and Landsat-7 data in this embodiment of the invention. To more clearly demonstrate the effectiveness of the method provided by this invention, Figure 1 , Figure 3 and Figure 4 The original image can be found in the audit reference materials.

[0082] In one implementation, in the radiation guidance module of the network in step S3, the imaging guide extracts the foreground and background regions in the feature map through masking operations, calculates the mean and variance of the corresponding regions, and then uses a transformation technique to transfer the radiation conditions from the background to the foreground, eliminating the radiation difference between the foreground and background, and obtaining the imaging radiation guidance feature map; the semantic guide continues to encode the imaging radiation guidance feature map obtained by the imaging guide, and constrains the same type of ground features in the foreground and background through semantic constraint radiation consistency penalty, and obtains the semantic guidance feature map.

[0083] Please see Figure 3 This is a schematic diagram of the imaging conditional radiation guide in an embodiment of the present invention.

[0084] Specifically, for the difference in imaging condition radiation between the foreground and background, the imaging radiation guide in the radiation guidance module extracts the foreground and background regions from the feature map of the sampled Mask, and calculates the mean and variance σ of the foreground and background feature maps. core μ core and σ back μ back Core represents the foreground and back represents the background. Then, the radiation conditions are transferred from the background to the foreground using the Transferer technique (imaging condition transfer) (see appendix). Figure 3 In (b), the obtained neural network feature map (imaging radiation-guided feature map) is input into the imaging conditional radiation correction module to eliminate the radiation difference between the foreground and background.

[0085] In one implementation, the intrinsic content radiometric correction module of the network in step S3 takes the input data and semantic feature guidance map as input, performs semantic constraints on similar ground features in the foreground and background, and obtains the intrinsic content components of the reconstructed output image.

[0086] For the intrinsic content correction module, the encoder consists of 3 convolutional layers (Convolution, Instance Norm and LeakyReLU, with kernel sizes of 7×7, 4×4, and 4×4) and 4 residual layers (with kernel size of 3×3). The intermediate layer consists of 2 convolutional layers (Convolution, Instance Norm and LeakyReLU, with kernel size of 3×3). The decoder consists of 2 up-coupling layers and 3 convolutional layers (Convolution, Instance Norm and LeakyReLU, with kernel sizes of 5×5, 5×5, and 7×7). See the appendix for details. Figure 3 Part (a). This module uses M and I. src_T As input, based on the assumption: C src ≈C src_T ≈C out Construct an intrinsic content radiation consistency penalty function (l) content This achieves quantitative constraints and protection of image texture features, resulting in C. out .

[0087] In one implementation, the imaging radiometric correction module of the network in step S3 takes the input data, the imaging radiometric guidance feature map, and the semantic guidance feature map as inputs to correct the radiometric differences in the foreground and background, and obtains the imaging conditional components of the reconstructed output image, including gain and offset.

[0088] For the imaging conditional radiometric correction module, the encoder consists of 3 convolutional layers (Convolution, Instance Norm, and Leaky ReLU, with kernel sizes of 7×7, 4×4, and 4×4) and 4 residual layers (with kernel size of 3×3). The intermediate layer consists of 2 convolutional layers (Convolution, Instance Norm, and Leaky ReLU, with kernel size of 3×3). The decoder consists of 2 up-layer layers and 3 convolutional layers (Convolution, Instance Norm, and Leaky ReLU, with kernel sizes of 5×5, 5×5, and 7×7). See the appendix for details. Figure 3 (c) This module forces imaging condition A by adding an imaging radiation penalty. out B out with I src Consistent with the imaging radiation penalty function, additionally, by adding imaging guide module features to the imaging radiation correction module.

[0089] Remote sensing images often exhibit radiometric continuity during the imaging process, thus their conditional radiometric properties are smooth. Further utilizing the image gradient to constrain the smoothness of A and B leads to the following: By constructing an imaging conditional radiation smoothing loss, I is forced out Imaging radiation smoothing,

[0090] In the process of reconstructing missing image information, a reconstruction loss is added to the objective function to ensure the consistency of foreground and background image textures, spectral consistency, and overall consistency.

[0091] In one embodiment, the method further includes using the following overall loss function during the training process in step S4:

[0092]

[0093] Where l is the overall loss function, l content The intrinsic content radiation consistency penalty function, Let l be the imaging radiation penalty function. AB Let be the imaging conditional radiation smoothing loss function. Let l be the radiation uniformity penalty function. rec The loss function for reconstructing missing image information is defined by λ1, λ2, λ3, λ4, and λ5, which are corresponding weighting factors used to balance the contributions of different losses. The calculation formulas for each loss function are as follows:

[0094]

[0095] Csrc For the intrinsic content components of the missing image, C out The intrinsic content components of the reconstructed output image. The gradient of the intrinsic content components of the reconstructed output image. The gradient of the missing image; Represents the encoder or decoder in the neural network computation process;

[0096]

[0097] A out B out For the conditional components of the reconstructed output image, I src Let E be the image to be reconstructed, i.e. the missing image, and E be the identity matrix.

[0098]

[0099] The gradient of the conditional component of the reconstructed output image.

[0100]

[0101] S is a similarity function. srcdown Ω(f) is the feature map obtained after downsampling, f is the feature map input to the imaging-guided radiation guide, and Ω(f) is the feature map obtained after processing by the imaging condition transfer module in the imaging-guided radiation guide.

[0102]

[0103] I out The image output after reconstruction. These represent the gradients of the reconstructed image and the image to be reconstructed, respectively.

[0104] Specifically, in the radiometric decoupling phase, based on the radiometric correction model, the radiometric decoupling module forces the encoder to convert the remote sensing image I during training. src Decoupling into intrinsic content-related component C and imaging condition-related components A and B, for the synthetic image I src_T Then we can get I src_T =A src_T ·C src_T +B src_T ;

[0105] In the dataset construction phase, cloud and cloud shadow missing masks and strip missing masks M are obtained through the Perlin noise algorithm and extraction from real strip images, where 1 represents the region to be reconstructed and 0 represents the non-reconstructed region. For paired images I to be reconstructed... src and multi-temporal reference image I TBased on the principle of obtaining Landsat-8 30m, GaoFen-1 2m, and Landsat-7 30m data respectively;

[0106] During the network training phase, the model uses synthetic images I src_T Using the missing mask M as input, the intrinsic content radiometric correction results are obtained through the intrinsic content radiometric correction module and the imaging conditional radiometric correction module, respectively. out and imaging conditional radiometric correction results A out and B out During this process, the imaging guide in the radiation guidance module extracts foreground and background regions from the feature map using a mask, calculates the mean and variance of the corresponding regions, and then uses the Transferer technique to transfer radiation conditions from the background to the foreground, eliminating radiation differences between the foreground and background. The semantic guide in the radiation guidance module further encodes the feature map of the imaging guide, constraining the same type of land cover in the foreground and background through semantic constraints on radiation consistency penalties. Finally, based on the radiation decoupling model I... out =A out ·C out +B out Obtain the reconstruction results;

[0107] During the testing phase, based on the trained DecRecNet model, the input image I to be reconstructed is... src Reference Image I T And mask M, output the reconstructed image I out .

[0108] Regarding the various loss functions, l content In the intrinsic content radiometric correction module, an intrinsic content radiometric consistency penalty function is constructed to achieve quantitative constraints and protection of image texture features; In the imaging condition radiometric correction module, imaging condition A is forced by adding an imaging radiometric penalty. out B out , and I src Consistent, the constructed imaging radiation penalty function; l AB By constructing imaging conditional radiation smoothing loss to force I out The imaging radiation smoothing function, For radiation differences among similar land features, a radiation consistency penalty function is constructed based on the structural similarity function SSIM. rec It is a reconstruction loss function used to ensure consistency in texture, spectral consistency, and overall consistency between foreground and background images during the reconstruction of missing image information.

[0109] During training, the objective function (overall loss function) is continuously backpropagated to make the reconstructed image as close as possible to the missing image in the latent space, thus obtaining the final trained reconstruction model Φ.

[0110] In specific applications or testing, the missing optical image data to be reconstructed I src Multi-temporal reference image data I T The DenRecNet network model, after training with the corresponding missing mask M as input, ultimately obtains the reconstruction result I of the corresponding region. out .

[0111] Example 2

[0112] Based on the same inventive concept, this embodiment provides a device for reconstructing missing information in remote sensing images under radiation decoupling, including:

[0113] The image acquisition module is used to acquire the image to be reconstructed and the reference image, and to extract the missing mask from the image to be reconstructed.

[0114] The dataset construction module is used to obtain a synthetic image based on the image to be reconstructed, the reference image, and the extracted missing mask, and to construct a dataset based on the synthetic image and the missing mask, and to divide the training dataset.

[0115] The network construction module is used to construct the DecRecNet network for decoupling and reconstructing missing information in remote sensing images. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and the semantic feature guidance map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain the imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain the semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric guidance map, and the semantic feature guidance map. The result output module is used to calculate the intrinsic content components and imaging conditional components of the reconstructed output image based on the radiometric decoupling model to obtain the reconstructed image. The input data includes the synthetic image and the missing mask.

[0116] The network training module is used to train the constructed remote sensing image missing information decoupling and reconstruction network using the training dataset, and obtain the trained remote sensing image missing information decoupling and reconstruction network.

[0117] The image reconstruction module is used to reconstruct the image to be reconstructed by decoupling the reconstruction network using the missing information of the trained remote sensing image.

[0118] Since the apparatus described in Embodiment 2 of this invention is the same apparatus used to implement the remote sensing image missing information reconstruction method under radiation decoupling in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0119] Example 3

[0120] Based on the same inventive concept, please refer to Figure 5 The present invention also provides a computer-readable storage medium 300 having a computer program 311 stored thereon, which, when executed, implements the method described in Embodiment 1.

[0121] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the remote sensing image missing information reconstruction method under radiation decoupling in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0122] Example 4

[0123] Based on the same inventive concept, this application also provides a computer device, such as... Figure 6 As shown, it includes a memory 401, a processor 402, and a computer program 403 stored in the memory and executable on the processor. When the processor executes the program, it implements the method in Embodiment 1.

[0124] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the remote sensing image missing information reconstruction method under radiation decoupling in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0128] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for reconstructing missing information in remote sensing images under radiometric decoupling, characterized in that, include: S1: Obtain the image to be reconstructed and the reference image, and extract the missing mask from the image to be reconstructed; S2: Based on the image to be reconstructed, the reference image, and the extracted missing mask, a synthetic image is obtained, and a dataset is constructed based on the synthetic image and the missing mask, and a training dataset is divided. S3: Construct DecRecNet, a remote sensing image missing information decoupling and reconstruction network. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and the semantic feature guidance map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain an imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain a semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric guidance feature map, and the semantic feature guidance map. The result output module is used for radiometric decoupling and reconstruction based on the radiometric decoupling model. The reconstructed image is obtained by calculating the intrinsic content components and imaging condition components of the reconstructed output image. The input data includes the synthetic image and the missing mask. The imaging radiometric guide is used to obtain the imaging radiometric guide feature map based on the input data. Specifically, it includes: extracting the foreground and background regions in the feature map through masking operations, calculating the mean and variance of the corresponding regions, and then using transformation techniques to transfer the radiometric conditions from the background to the foreground to eliminate the radiometric difference between the foreground and the background, thereby obtaining the imaging radiometric guide feature map. The semantic guide obtains the semantic feature guide map based on the input data. The semantic guide specifically includes: further encoding the imaging radiometric guide feature map obtained by the imaging guide, and constraining the same type of ground features in the foreground and background through semantic constraint radiometric consistency penalty, thereby obtaining the semantic guide feature map. S4: The constructed remote sensing image missing information decoupling and reconstruction network is trained using the training dataset to obtain the trained network. Specifically, this includes: using synthetic images and missing masks as input, obtaining intrinsic content radiometric correction results and imaging condition radiometric correction results through the intrinsic content radiometric correction module and imaging condition radiometric correction module, respectively. During this process, the imaging guide in the radiometric guidance module uses a mask to extract foreground and background regions from the feature map, calculates the mean and variance of the corresponding regions, and then uses the Transferer technique to transfer the radiometric conditions from the background to the foreground, eliminating the radiometric differences between the foreground and background; the semantic guide in the radiometric guidance module further encodes the imaging guide feature map, and constrains the same type of ground features in the foreground and background through semantic constraint radiometric consistency penalty; finally, the reconstruction result is obtained based on the radiometric decoupling model. The overall loss function used during training is: （1） in, For the overall loss function, The intrinsic content radiation consistency penalty function, Let be the imaging radiation penalty function. Let be the imaging conditional radiation smoothing loss function. This is the semantic constraint radiation consistency penalty function. The loss function for reconstructing missing image information is... , , , , These are the corresponding weighting factors used to balance the contributions of different losses. The calculation formulas for each loss function are as follows: （2） The intrinsic content components of the missing image. The intrinsic content components of the reconstructed output image. The gradient of the intrinsic content components of the reconstructed output image. The gradient of the missing image; Represents the encoder or decoder in the neural network computation process; （3） , The conditional components of the reconstructed output image. The image to be reconstructed, i.e., the missing image. It is the identity matrix; （4） , The gradient of the conditional component of the reconstructed output image. （5） For similarity functions, This is the feature map obtained after downsampling. The feature map is input to the imaging-guided radiation guide. This is a feature map obtained after processing by the imaging conditional transfer module in the imaging-guided radiation guide; （6） The image output after reconstruction. , These represent the gradients of the reconstructed image and the image to be reconstructed, respectively. S5: Use the decoupling and reconstruction network with the missing information of the trained remote sensing image to reconstruct the image to be reconstructed.

2. The method for reconstructing missing information in remote sensing images under radiometric decoupling as described in claim 1, characterized in that, Step S1 involves extracting the missing mask from the image to be reconstructed, including: To address the information loss problem caused by clouds and cloud shadows, the Berlin noise method is used to simulate clouds and cloud shadows to obtain cloud loss masks. To address the information loss problem caused by sensor malfunction, a stripe loss mask is obtained by extracting the actual missing stripes. The acquired cloud-based missing mask and stripe-based missing mask are used as missing masks. The regions with a pixel value of 1 represent missing regions, and the regions with a pixel value of 0 represent non-missing regions.

3. The method for reconstructing missing information in remote sensing images under radiometric decoupling as described in claim 1, characterized in that, Step S2 involves obtaining a synthesized image based on the image to be reconstructed, the reference image, and the extracted missing mask, including: Background image obtained from the image to be reconstructed and the missing mask. ,in, For images to be reconstructed, For missing masks; Obtain the foreground image based on the reference image and the missing mask. ,in, For reference image; A composite image is obtained from the acquired background and foreground images. , = .

4. The method for reconstructing missing information in remote sensing images under radiometric decoupling as described in claim 1, characterized in that, In step S3, the intrinsic content radiometric correction module of the network takes the input data and semantically guided feature map as input, performs semantic constraints on the same ground features in the foreground and background, and obtains the intrinsic content components of the reconstructed output image.

5. The method for reconstructing missing information in remote sensing images under radiometric decoupling as described in claim 1, characterized in that, The imaging radiometric correction module of the network in step S3 takes the input data, the imaging radiometric guidance feature map, and the semantic feature guidance map as inputs to correct the radiometric differences in the foreground and background, and obtains the imaging conditional components of the reconstructed output image, including gain and offset.

6. A device for reconstructing missing information in remote sensing images under radiation decoupling, characterized in that, Based on the method described in claim 1, the apparatus comprises: The image acquisition module is used to acquire the image to be reconstructed and the reference image, and to extract the missing mask from the image to be reconstructed. The dataset construction module is used to obtain a synthetic image based on the image to be reconstructed, the reference image, and the extracted missing mask, and to construct a dataset based on the synthetic image and the missing mask, and to divide the training dataset. The network construction module is used to construct the DecRecNet network for decoupling and reconstructing missing information in remote sensing images. DecRecNet includes an intrinsic content radiometric correction module, a radiometric guidance module, an imaging conditional radiometric correction module, and a result output module. The intrinsic content radiometric correction module is used to obtain the intrinsic content components of the reconstructed output image based on the input data and the semantic feature guidance map. The radiometric guidance module includes an imaging radiometric guide and a semantic guide. The imaging radiometric guide is used to obtain an imaging radiometric guidance feature map based on the input data, and the semantic guide is used to obtain a semantic feature guidance map based on the input data. The imaging conditional radiometric correction module is used to obtain the imaging conditional components of the reconstructed output image based on the input data, the imaging radiometric guidance feature map, and the semantic feature guidance map. The result output module is used to calculate the intrinsic content components and imaging conditional components of the reconstructed output image based on the radiometric decoupling model to obtain the reconstructed image. The input data includes the synthetic image and the missing mask. The network training module is used to train the constructed remote sensing image missing information decoupling and reconstruction network using the training dataset, and obtain the trained remote sensing image missing information decoupling and reconstruction network. The image reconstruction module is used to reconstruct the image to be reconstructed by decoupling the reconstruction network using the missing information of the trained remote sensing image.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1 to 5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

Images