A Method and System for Reconstructing Missing Information from Multi-Temporal Remote Sensing Images with Dual Decoupling and Mutual Correction

Abstract

This invention discloses a method and system for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction. The method includes acquiring multiple cloud-occluded images of the same region at different time phases and resolutions as a multi-temporal mutual correction dataset, and creating a cloud mask dataset; cropping a training dataset with a one-to-one correspondence between the cloud-occluded images and the masks; merging multiple cloud-occluded images from different time phases into a single cloud-free composite image according to complementary rules, and creating mask data; constructing a dual decoupling and mutual correction multi-temporal missing information reconstruction network model, including an imaging object mutual correction module, an imaging environment mutual correction module, and an imaging environment enhancement module; training the reconstruction network using the training set data; and inputting the composite image and mask obtained from the time-series images to be reconstructed and their corresponding masks according to complementary rules into the network model to obtain the reconstructed image. This invention can effectively reconstruct information missing caused by cloud occlusion in high-resolution remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image cloud occlusion missing information reconstruction technology, specifically involving a method and system for reconstructing missing information in dual-decoupled mutual correction multi-temporal remote sensing images. Background Technology

[0002] Optical remote sensing data is a crucial source of information for Earth research, especially high-resolution optical imagery, which has been widely applied in atmospheric environmental monitoring, natural resource surveys, global land cover analysis, and emergency rescue. However, due to limitations in the physical imaging conditions of optical remote sensing satellite sensors and interference from the actual working environment, optical remote sensing satellite imagery is inevitably contaminated by thick clouds and cloud shadows, resulting in varying degrees of information loss in almost all remote sensing images, significantly reducing the usability of optical data. Existing cloud-loss information reconstruction models typically assume that the temporal image is a cloud-free, intact image, an assumption rarely held true in real-world scenarios. Furthermore, compared to medium- and low-resolution imagery, high-resolution imagery provides richer information on ground texture and spatial distribution, further increasing the difficulty of reconstruction.

[0003] To address the two issues mentioned above, this invention leverages the powerful nonlinear representation capabilities of deep learning to reconstruct missing information from high-resolution time-series cloud-occluded images even without complete, cloudless temporal data. It treats non-cloudy areas in the time-series images to be reconstructed as background areas and other cloudy areas as foreground areas. The reasons for the radiation differences between the foreground and background are summarized into two parts: 1) differences in the imaging object itself, representing the attributes of the imaging object, such as phenological characteristics, spatial characteristics, and ground features; 2) differences in the imaging environment, including differences in brightness and hue caused by external environmental factors such as atmospheric conditions and solar altitude angle. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for reconstructing missing information in multitemporal remote sensing images through dual decoupling and mutual correction. By decoupling the remote sensing image into two parts, the imaging object and the imaging environment, a novel dual decoupling and mutual correction multitemporal missing information reconstruction network DDM-RecNet (Dual Decoupling Inter-correction Multitemporal Reconstruction Network) is constructed to achieve reconstruction of missing information in multitemporal images obscured by cloud cover.

[0005] To achieve the above objectives, this invention provides a method for reconstructing missing information in dual-decoupled, mutually corrected multi-temporal remote sensing images, comprising the following processes.

[0006] Multiple cloud-occluded images of the same region at different times and resolutions are acquired as a dataset for mutual correction of cloud occlusion over multiple times, and a cloud mask dataset is created.

[0007] Multiple cloud-occluded images and masks from different time periods are cropped to the same size to obtain a training dataset in which cloud-occluded images and masks correspond one-to-one.

[0008] Multiple cloud-occluded images from different time periods are merged into a single cloudless composite image according to a complementary rule, and mask data is generated.

[0009] A dual-decoupled, mutually corrected, multi-temporal missing information reconstruction network model is constructed. This network includes an imaging object mutual correction module, an imaging environment mutual correction module, and an imaging environment enhancement module. The imaging object mutual correction module and the imaging environment mutual correction module decouple the synthesized image into two parts: imaging object mutual correction features and imaging environment mutual correction features, and output the imaging object features and imaging environment features respectively. The imaging environment enhancement module further extracts the imaging environment features of the background region in the reference image, and the extraction results are input into the intermediate layer of the imaging environment mutual correction module to ensure that the imaging environment of the reconstructed image matches the imaging environment features of the reference image. Finally, the reconstructed image is obtained by fusing the imaging object features and imaging environment features.

[0010] The training set data is used to train the dual-decoupled mutual correction multi-temporal missing information reconstruction network; the time series images to be reconstructed and the corresponding masks are used to obtain synthetic images and masks according to the complementarity rules, and then input into the trained network model to obtain the reconstructed images.

[0011] Furthermore, in the cloud mask dataset, the mask value for cloudless pixels is assigned to 0, and the mask value for cloudy pixels is assigned to 1.

[0012] Furthermore, let N represent the image sequence number, and use the input order as the priority of each image. Based on the complementarity rule, multiple cloud-occluded images from different time phases are merged into a single cloudless image I. com and create mask data. com The complementary rules are implemented as follows:

[0013]

[0014] In the formula, I com (x,y) represents the pixel value at (x,y) in the composite image.

[0015] when T1 (c,y)+ T2 (x,y)+…+ TN When (x,y)=0, it indicates that multiple images at (x,y) from different time phases are cloudless, and at this time I com(x, y) represents the pixel value at (x, y) in the first cloud-occluded image. T1 (x,y), com (x,y) is the mask at (x,y) of the first cloud-occluded image. T1 (x,y);

[0016] when T1 (x,y)+ T2 (x,y)+…+ TN When (x,y) < 0, it indicates that there are one or more images with clouds at (x,y). In this case, the cloudless pixel at (x,y) is selected as I based on the image priority. com Assign values ​​to (x, y) and update com (x,y);

[0017] when T1 (x,y)+ T2 (x,y)+…+ TN When (x,y) = , it indicates that multiple images at (x,y) from different time phases all contain clouds. Therefore, reconstruction at (x,y) is not possible, resulting in an I value. com (x,y) is assigned the value Nodata. com Assign 1 to (x,y).

[0018] Furthermore, the imaging object mutual correction module continuously approximates the synthetic image to the real image from the perspective of image gradient, thereby maintaining the spectral variation information of ground objects in the foreground and background.

[0019] Furthermore, the imaging environment mutual correction module employs CNN and transformer decoder to ensure that the nonlinear imaging environment characteristics of the synthesized image remain consistent overall.

[0020] Moreover, the loss function of the dual-decoupled mutual correction multi-temporal missing information reconstruction network model includes three types of losses: reconstruction, imaging object, and imaging environment.

[0021] On the other hand, the present invention provides a dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction system, used to realize the dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction method as described above.

[0022] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction method as described above.

[0023] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements the method described above for reconstructing missing information in dual-decoupled, mutually corrected multi-temporal remote sensing images.

[0024] This invention discloses a scheme for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction. Even without complete cloud-free temporal data, it can effectively reconstruct information missing caused by cloud obstruction in high-resolution remote sensing images. Compared with existing technologies, this invention has the following advantages:

[0025] 1) This invention provides a novel dual-decoupled mutual correction multi-temporal missing information reconstruction network. Starting from the imaging object and imaging environment of remote sensing images, the remote sensing images are decoupled into two parts, the imaging object and the imaging environment, in order to improve the reconstruction performance of high-resolution images.

[0026] 2) Based on the radiometric difference between the background region and the sequential foreground image in multi-temporal images, we designed imaging object mutual correction consistency loss and imaging environment mutual correction consistency loss to perform pixel-level correction of missing regions, and through the imaging environment enhancement module, we maintain the imaging environment consistency between the foreground region and the background region in the synthetic image. Attached Figure Description

[0027] Figure 1 This is an overall structural diagram of the DDM-RecNet model in an embodiment of the present invention.

[0028] Figure 2 This is a flowchart of the reconstruction steps based on missing information from DDM-RecNet in an embodiment of the present invention. Detailed Implementation

[0029] This invention provides a method for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction. The technical solution of this invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] like Figure 2 As shown, the process of a method for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction provided by an embodiment of the present invention includes the following steps:

[0031] Step 1: Acquire multiple Sentinel-2 (10m) and Gaofen-1 (2m) cloud-occluded images of the same region at different time phases as a multi-temporal cross-correction dataset for cloud occlusion. TN (N = 1, 2, ..., n), and create a cloud mask dataset M. TN (N = 1, 2, ... n).

[0032] Where n represents the number of images and N represents the image sequence number.

[0033] In the cloud mask dataset M TN In (N=1,2,…n), the mask value of the cloudless pixels is assigned to 0, and the mask value of the cloudy pixels is assigned to 1.

[0034] In practice, cloud obstruction can be simulated using Burmester noise in the simulation experiment to obtain the mask M corresponding to each time phase data in the simulation experiment. N (N = 1, 2, ..., n), where 1 represents the cloud-covered area and 0 represents the non-clouded area; in the actual experiment, ArcGIS software was used to draw the cloud-covered areas in each time phase, and the mask M corresponding to the data of each time phase in the actual experiment was obtained. N (N = 1, 2, ..., n), where 1 represents the cloud-covered area and 0 represents the non-clouded area;

[0035] Step 2, acquire multiple cloud-occlusion images from different time periods I TN and mask M TN The image was cropped to 256×256 to obtain the cloud-occluded image I. TN and mask M TN A one-to-one corresponding training dataset.

[0036] In practice, it is recommended to use a strategy based on block-based acceleration of network training for pruning.

[0037] Step 3: Combine multiple cloud-occluded images from different time periods into a single cloudless composite image according to the complementarity rule, and create mask data.

[0038] Using the input order as the priority for each image, multiple cloud-occluded images from different time phases are merged into a single cloudless image I based on a complementarity rule. com And create mask data M com .

[0039] The specific calculation method for the complementarity rule is as follows:

[0040]

[0041] In the formula, I com (x,y) represents the pixel value at (x,y) in the composite image.

[0042] When M T1 (x,y)+M T2 (x,y)+…+M TN When (x,y)=0, it indicates that multiple images at (x,y) from different time phases are cloudless, and at this time I com (x,y) is the pixel value I at (x,y) in the first cloud-occluded image. T1 (x,y), v com (x,y) is the mask M at (x,y) of the first cloud-occluded image. T1 (x,y);

[0043] When M T1 (c,y)+M T2 (x,y)+…+MTN When (x, y) < N, it indicates that there is one or more images with clouds at (x, y). At this time, cloud-free pixels at (x, y) are selected according to the priority of the images as I com Assign a value to (x, y) and update M com (x, y);

[0044] When M T1 (x, y)+M T2 (x, y)+…+M TN When (x, y) = N, it indicates that multiple images with different temporal phases at (x, y) are all clouded. At this time, reconstruction cannot be performed at (x, y), and it is I com Assign Noadta to (x, y), M<A com Assign 1 to (x, y).

[0045] Step 4, construct a dual decoupling inter-correction multitemporal missing information reconstruction network model.

[0046] See Figure 1 , the dual decoupling inter-correction multitemporal reconstruction network (Dual Decoupling Inter-correction Multitemporal Reconstruction Network, DDM-RecNet) includes three modules: an imaging object inter-correction module, an imaging environment inter-correction module, and an imaging environment enhancement module.

[0047] Combined with the remote sensing image imaging principle, the present invention generalizes the fundamental reasons for the radiation differences between remote sensing images into two parts: imaging object differences and imaging environment differences.

[0048] The imaging object inter-correction module and the imaging environment inter-correction module respectively decouple the synthetic image into imaging object inter-correction features and imaging environment inter-correction features through the ground object feature extraction network, the imaging environment feature extraction network and the corresponding loss functions, and respectively output the imaging object feature A out and the imaging environment feature B out . The imaging object inter-correction module makes the synthetic image continuously approach the real image from the perspective of image gradient to maintain the spectral change information of the ground objects in the foreground and background. The imaging environment inter-correction module uses CNN and transformer decoder to make the non-linear imaging environment features of the synthetic image maintain overall consistency, such as brightness, hue and other information. Since the foreground region images in the synthetic image are from multiple temporal phase data, to ensure that the imaging environment of the reconstructed image conforms to the imaging environment features of the first image (i.e., the reference image), the background region in the first image is further extracted through the imaging environment enhancement module The imaging environment features are extracted and the results are input into the intermediate layer of the imaging environment mutual correction module. Finally, the reconstructed image I is obtained by fusing the imaging object features with the imaging environment features. out ,Right now

[0049] Since the revisit period for remote sensing images is generally 1-16 days, this invention assumes that the boundaries, semantics, and spectral information of the imaging objects remain unchanged within a short period. Furthermore, since the boundaries, semantics, and spectral information of the imaging objects can be characterized by image gradients, the imaging object mutual correction module combines the Sobel operator to construct an imaging object mutual correction consistency loss function to decouple the imaging objects from the remote sensing images. The imaging object mutual correction module is implemented using an encoder-decoder structure. The encoder uses 7×7, 4×4, and 4×4 three-layer convolutions to extract multi-channel features, and the intermediate layer uses a 2-head 9-layer transformer encoder to obtain multi-dimensional global context information f. G The decoder uses 5×5, 5×5, and 7×7 convolutional layers for upsampling, ultimately obtaining the imaging object features A. out The corresponding loss function is expressed as follows:

[0050]

[0051] In the formula, l content This represents the mutual calibration consistency loss function for the imaging objects. Let A be the Sobel operator and A be the feature of the imaged object. Indicates the characteristics of the imaged object in each image. This represents the imaging environment module, and ||||1 represents L1 loss.

[0052] For the imaging environment mutual correction module, under the assumption that the imaging object remains unchanged, the radiation difference between the foreground and background regions of the synthesized image originates from the imaging environment. In this module, the multi-channel imaging environment features f are first obtained through 7×7, 4×4, and 4×4 methods. IEIM Secondly, since the imaging environment also has holistic characteristics, this module will incorporate the imaging environment features f IEIM The imaging object features f obtained by the imaging object mutual correction module G Combined Then in the middle layer of this module, The image is divided into 4×4 blocks as source (key-value) tokens, and the imaging environment enhancement features are used. As a query, it is input into a 2-head 9-layer transformer decoder to obtain multi-dimensional global features. Finally, it is upsampled through 5×5, 7×7, and 7×7 convolutional layers, and combined with the imaging environment consistency loss function to decouple the remote sensing image imaging environment, thus obtaining the imaging environment feature B. out The imaging environment consistency loss function is calculated as follows:

[0053]

[0054] In the formula, S[] is the SSIM similarity function, and B is the imaging environment feature. This indicates the imaging environment characteristics of each image.

[0055] For the imaging environment enhancement module, in cloud-occluded time-series images I TN In the sequence (N = 1, 2, ..., n), the larger N is, the more temporal data there is, and the greater the radiometric difference between the foreground and background regions in the composite image. The reconstruction performance then becomes more dependent on the ability to extract global features. This invention utilizes I... T1 ·(1-M) Obtain the background image of this module Then and M com The data is concatenated along the channel dimensions to serve as input data. Background features are extracted using 5×5, 3×3, and 3×3 convolutional kernels, and then processed by a 9-layer transformer decoder from I... T1 Extracting global imaging environment enhancement features from reference images

[0056] Among them, M com This is the background area of ​​the first image.

[0057] For imaging feature A out and imaging environment features B out The images are fused to obtain the reconstructed image I. out .

[0058] Step 5: Train the dual-decoupled mutual correction multi-temporal missing information reconstruction network using training set data.

[0059] To achieve an infinite approximation between the reconstructed image and the real image in the latent space, this invention also constructs a reconstruction loss function, the specific calculation method of which is as follows:

[0060]

[0061] The loss function used in this invention includes three types of losses: those for reconstruction, those for the imaging object, and those for the imaging environment. The calculation formula for loss function l is as follows:

[0062] l=λ1l rec +λ2l Imaging +λ3l content (5)

[0063] In the formula, λ1, λ2, and λ3 are the weights of each loss, l rec Indicates reconstruction losses, l Imaging Indicates imaging environment loss, l ground Indicates the loss of the imaged object. Let A be the Sobel operator, S be the SSIM similarity function, and B be the imaging environment features.

[0064] The objective function is repeatedly backpropagated to make the reconstructed image as close as possible to the real image in the latent space, thus obtaining the reconstruction result I. out and the trained reconstruction model

[0065] Step 6: The time series image to be reconstructed and the corresponding mask are used to obtain the synthetic image and mask according to the complementarity rule. The synthetic image and mask are then input into the trained DDM-RecNet network model to obtain the reconstructed image.

[0066] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0067] In some possible embodiments, a dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction system is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction method as described above.

[0068] In some possible embodiments, a dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction system is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction method as described above.

[0069] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for reconstructing missing information from multi-temporal remote sensing images with dual decoupling and mutual correction, characterized in that: Includes the following processing, Multiple cloud-occluded images of the same region at different times and resolutions are acquired as a dataset for mutual correction of cloud occlusion over multiple times, and a cloud mask dataset is created. Multiple cloud-occluded images and masks from different time periods are cropped to the same size to obtain a training dataset in which cloud-occluded images and masks correspond one-to-one. Multiple cloud-occluded images from different time periods are merged into a single cloudless composite image according to a complementary rule, and mask data is generated. A dual-decoupled, mutually corrected, multi-temporal missing information reconstruction network model is constructed. This network includes an imaging object mutual correction module, an imaging environment mutual correction module, and an imaging environment enhancement module. The imaging object mutual correction module and the imaging environment mutual correction module decouple the synthesized image into two parts: imaging object mutual correction features and imaging environment mutual correction features, and output the imaging object features and imaging environment features respectively. The imaging environment enhancement module further extracts the imaging environment features of the background region in the reference image, and the extraction results are input into the intermediate layer of the imaging environment mutual correction module to ensure that the imaging environment of the reconstructed image matches the imaging environment features of the reference image. Finally, the reconstructed image is obtained by fusing the imaging object features and imaging environment features. The training set data is used to train the dual decoupled mutual correction multi-temporal missing information reconstruction network; the time series images to be reconstructed and the corresponding masks are combined according to the complementarity rules to obtain synthetic images and masks, which are then input into the trained network model to obtain the reconstructed images; The imaging environment mutual correction module uses CNN and transformer decoder to ensure that the nonlinear imaging environment characteristics of the synthesized image maintain overall consistency. The loss function of the dual-decoupled mutual correction multi-temporal missing information reconstruction network model includes three types of losses: reconstruction, imaging object, and imaging environment.

2. The method for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction according to claim 1, characterized in that: In the cloud mask dataset, the mask value for pixels without clouds is assigned to 0, and the mask value for pixels with clouds is assigned to 1.

3. The method for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction according to claim 1, characterized in that: set up The image number represents the sequence number of each image, with the input order serving as the priority. Multiple cloud-occluded images from different time periods are merged into a single cloudless image based on a complementarity rule. and create mask data. The complementary rules are implemented as follows: In the formula, Indicates composite image Pixel value at that location, when When, it indicates Multiple images taken at different times show no clouds. Take the first image obscured by clouds. pixel value at , Take the first image obscured by clouds. mask at the location ; when When, it indicates If one or more images show clouds, the selection should be based on image priority. The number of cloudless pixels is Assign and update ; when When, it indicates Multiple images from different times all show clouds at this point. The site cannot be rebuilt, for Assignment , Assign 1.

4. The method for reconstructing missing information in multi-temporal remote sensing images with dual decoupling and mutual correction according to claim 1, characterized in that: The imaging object mutual correction module continuously approximates the synthetic image to the real image from the perspective of image gradient, thereby maintaining the spectral variation information of ground objects in the foreground and background.

5. A system for reconstructing missing information from dual-decoupled, cross-corrected multi-temporal remote sensing images, characterized in that: This method is used to implement the missing information reconstruction method for dual-decoupled mutual correction multi-temporal remote sensing images as described in any one of claims 1-4.

6. The dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction system according to claim 5, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the method for reconstructing missing information in dual-decoupled mutual-correction multi-temporal remote sensing images as described in any one of claims 1-4.

7. The dual-decoupled mutual correction multi-temporal remote sensing image missing information reconstruction system according to claim 5, characterized in that: The method includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the method for reconstructing missing information in dual-decoupled mutual correction multi-temporal remote sensing images as described in any one of claims 1-4.

Images