A method and system for cooperative dual coordination remote sensing image cloud blocking missing reconstruction

Abstract

The application discloses a kind of synergic dual coordination remote sensing image cloud shelter missing reconstruction method and system, realize high-resolution image large-area cloud shelter missing information reconstruction target.The application is realized by the following technical solutions: first, synthetic image I com And mask M are obtained by shallow feature extraction module (I f , M f ).Second, in synergic dual coordination module, pixel-to-pixel fine-grained local coordination module and image-to-image global coordination module are constructed, with shallow feature (I f , M f ) As input, respectively obtain the feature I pix Of pixel level and the feature I igh Of global level, finally obtain reconstruction image after feature fusion by I out =I pix ·I IGH Second, the application designs similarity constraint function and gradient constraint function, maintains the overall consistency of foreground area and background area in synthetic image, finally realizes high-resolution image large-area cloud shelter missing information reconstruction target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud occlusion missing information reconstruction of remote sensing images based on deep learning, and in particular to a high-resolution remote sensing image large-area cloud occlusion missing information reconstruction method and system. BACKGROUND

[0002] High-resolution remote sensing images have the characteristics of rich spectral information, complex geographical texture and dense targets, and are the main data source for fine observation of the earth. At present, high-resolution remote sensing images have been applied in the fields of national defense construction, natural resource detection and the like. However, due to the existence of large-area cloud occlusion, the high-resolution remote sensing images cannot be directly applied to subsequent research, and reducing or removing the influence of cloud occlusion on high-resolution remote sensing images is a problem that must be faced in the process of remote sensing image processing.

[0003] In the existing cloud occlusion missing information reconstruction model, multi-temporal images are often used to complete the missing reconstruction task. In the multi-temporal reconstruction task, the images of the cloud occlusion area are derived from the multi-temporal images, also known as the reconstruction area, and the images outside the cloud occlusion area are derived from the cloud occlusion images, also known as the non-reconstruction area. The reconstruction area images and the non-reconstruction area images are spliced to obtain a composite image, and a deep learning model is used to continuously learn the non-linear relationship between the composite image and the cloud occlusion image, and then the reconstructed image is obtained. However, in this process, the feature extraction method of the reconstruction area and the non-reconstruction area without distinction exacerbates the problem of overall inconsistency and boundary effect in the reconstructed image. In addition, large-area cloud occlusion exacerbates the difficulty of using remote sensing images, but in the actual image acquisition process, it is impossible to avoid the problem of large-area cloud occlusion. Achieving the goal of large-area cloud occlusion missing information reconstruction is crucial to improving the efficiency of using remote sensing images.

[0004] In view of the above two problems, the present application constructs a collaborative dual-harmonization high-resolution large-area cloud occlusion missing information reconstruction network (CDH-RecNet) with the powerful non-linear expression ability of deep learning, coordinates the composite image from the angles of pixel-to-pixel fine-grained local coordination and image-to-image global coordination, and achieves the goal of high-resolution image large-area cloud occlusion missing information reconstruction. SUMMARY

[0005] The present application provides a collaborative dual-harmonization high-resolution image large-area cloud occlusion information reconstruction method to overcome the shortcomings of the prior art in high-resolution cloud occlusion large-area missing information reconstruction, and realizes missing information reconstruction under high-resolution complete cloud-free temporal images, which comprises the following steps: first, a to-be-reconstructed image is given Multi-temporal images Binary mask (representing missing areas, 0 representing non-missing areas), reconstruct the image Train the model Φ according to I com = I ori ·(1-M)+I T ·M to obtain the input data synthesis image I com of the model and M. Secondly, in the double harmonization module, the pixel-to-pixel fine-grained local harmonization module (PFLH) and the image-to-image global harmonization module (IGH) downsample the input images I com and M to (I f , M f ), the PFLH combines the self-calibrated block with pixel attention (SC-PA) module and the fusion feature I SSIM of the ground object constraint to obtain I pix , so as to achieve the goal of local pixel-level fine-grained harmonization. The IGH realizes the targeted harmonization of the information difference between the reconstructed area, the non-reconstructed area and the target area by constructing the fore-background aware module (FBAM) and fusing the ground object constraint feature I SSIM and obtains I igh , so that the reconstructed area and the non-reconstructed area achieve the goal of overall harmonization. Then, the ground constraint module (GCM) performs similarity constraint on a large number of recurring ground objects in different scales and across different scales in the image to obtain I SSIM . Finally, the PFLH pixel-level reconstruction loss, the IGH overall consistency loss (i.e. the structural similarity function), the reconstruction loss, etc. are realized through back propagation to obtain

[0006] The specific technical scheme of the present application comprises the following steps:

[0007] Step 1, based on the time proximity principle, obtain the remote sensing pair of cloud-shading images I ori and cloud-free images I T , and obtain the mask M corresponding to the cloud and cloud shadow, and then randomly divide them into a training set and a test set;

[0008] Step 2, based on the cloud-shading images I ori and the cloud-free images IT image I com ;

[0009] Step 3, to synthesize image I com and M as input, a reconstruction model is constructed, including a shallow feature extraction module and a collaborative double coordination module;

[0010] The shallow feature extraction module is composed of multiple convolutional layers and multiple residual layers, and takes I com and M as input to obtain shallow features (I f , M f );

[0011] The collaborative double coordination module is composed of two parts, namely a pixel-to-pixel fine-grained local coordination module and an image-to-image global coordination module, which take shallow features (I f , M f ) as input to obtain pixel-level features I pix and global-level features I igh respectively, and finally obtain the reconstructed image through feature fusion I out = I pix · I IGH ;

[0012] Step 4, combine the training set data, and use the loss function to train the constructed reconstruction model;

[0013] Step 5, input the synthetic image and the corresponding mask in the test set into the trained reconstruction model to obtain the reconstructed image.

[0014] Further, the specific calculation formula of the synthetic image is as follows:

[0015] I com = I ori ·(1-M)+I T ·M.

[0016] Further, the shallow feature extraction module includes multiple convolutional layers and multiple residual layers.

[0017] Further, the reconstruction model further includes a ground feature constraint module, and the ground feature constraint module includes five convolutional layers, and the input of the ground feature constraint module is the synthetic image I com and M, and the output of the ground feature constraint module is a ground feature constraint feature I ssim .

[0018] Further, the pixel-to-pixel fine-grained local coordination module includes a pixel self-correction module SC-PA, a convolution module and an up-sampling module, and the specific processing process is as follows: shallow features (I f , M f) after processing by the SC-PA module and added to the original input feature as the input of the convolution module, and then input into an SC-PA module again and added to the input feature of the SC-PA module, and then added to the ground feature constraint feature I ssim In the channel dimension, splicing is performed, and finally, pixel-level feature I is output through the upsampling module pix ;

[0019] The SC-PA module includes two convolution branches, the first convolution branch includes the following processing process: first, the input feature is processed through a convolution layer, then the output of the two small branches each including a convolution layer is multiplied and input into the convolution layer, the second convolution branch includes two convolution layers, the outputs of the two convolution branches are combined and then convolution processed, finally added to the original input to obtain the final output;

[0020] The convolution module includes two convolution layers, and the upsampling module includes two upsampling layers and three convolution layers.

[0021] Further, the image-to-image global coordination module includes a front background attention module FBAM, a convolution module and an upsampling module, and the specific processing process is as follows: the shallow feature (I f , M f ) after processing by the FBAM module and added to the original input feature as the input of the convolution module, and then input into an FBAM module again and added to the input feature of the FBAM module, and then added to the ground feature constraint feature I ssim In the channel dimension, splicing is performed, and finally, pixel-level feature I is output through the upsampling module igh ;

[0022] The FBAM module includes two branches, I fg branch and I bg branch, in the I fg branch, there are I mix branch and I L branch; in the I fg branch, I fg is first calculated by an average pooling layer, a maximum pooling and two linear layers, and second, the information change feature between the reconstructed region and the target image is input into I f , M f Further, the information change feature between the reconstructed region and the target image is calculated by an average pooling layer, a maximum pooling and two linear layers; the invariant feature between the reconstructed region and the target image is calculated by the branch I mix Further, the information change feature between the reconstructed region and the target image is calculated by an average pooling layer, a maximum pooling and two linear layers; the invariant feature between the reconstructed region and the target image is calculated by the branch I LBy improving the traditional residual by incorporating self-learning scaling factors α and β, the extraction of difference features between the input and target images in the reconstruction region is enhanced, thereby improving reconstruction accuracy. The improved IL structure multiplies the two branches by α and β respectively, where α and β are obtained through self-learning in the model. Then, I... mix and I L Feature fusion is performed by addition, and then multiplied by M to obtain I. fg Branch characteristics, I bg The branch is calculated using average pooling, max pooling, and two linear layers, and then multiplied by 1-M to obtain I. bg Characteristics of branches; finally, I fg Branches and I bg The features obtained from the branching are obtained by addition (I′) figh M′ figh The calculation process of the internal structure of FBAM can be summarized as follows:

[0023] I FBAM =M×[I L (I fg ((I f M f ))+I mix ((I f M f ))]+(1-M)×I bg ((I f M f In the formula, (I) f M f ) is the input feature map, I FBAM The features are FBAM features, and M is a binary mask;

[0024] The convolution module consists of two convolutional layers, and the upsampling module consists of two upsampling layers and three convolutional layers.

[0025] Furthermore, the loss function includes a gradient loss constraint function and a reconstruction loss constraint function;

[0026] The formula for calculating the gradient loss constraint function is as follows:

[0027]

[0028] In the formula, Let be the gradient loss constraint function. To compute the Sobel operator for the gradient, I out For the output reconstructed image, I ori The image represents a cloudless, real-world image; 1 represents L1 loss.

[0029] The formula for calculating the reconstruction loss constraint function is as follows:

[0030]

[0031] In the formula, is the reconstruction loss, I out is the output reconstructed image, I ori is the cloud-free real image, I is the L1loss.

[0032] Further, the loss function further comprises a similarity constraint function;

[0033] The calculation formula of the similarity constraint function is as follows:

[0034]

[0035] In the formula, S is a structural similarity function, I' ori is the real cloud-free image down-sampling feature, I ssim is the feature of the surface constraint module, is the similarity constraint function;

[0036] The final total loss is:

[0037]

[0038] In the formula, total loss, is a structural similarity function, is a gradient loss constraint function, is a reconstruction loss constraint function, λ1, λ2, λ3 are weights of each loss function.

[0039] The application also provides a synergistic dual coordination remote sensing image cloud occlusion missing reconstruction system, comprising the following units:

[0040] A data acquisition unit is configured to acquire remote sensing paired cloud-occluded images I ori and cloud-free images I T and acquire masks M corresponding to the clouds and cloud shadows, and then randomly divide the masks into a training set and a test set.

[0041] A synthetic image acquisition unit is configured to acquire synthetic images I ori based on the cloud-occluded images I T and the cloud-free images I com .

[0042] A model construction unit is configured to construct a reconstruction model by taking the synthetic images I com and M as inputs, and the reconstruction model comprises a shallow feature extraction module and a synergistic dual coordination module.

[0043] The shallow feature extraction module is composed of multiple convolutional layers and multiple residual layers, and takes I com and M as input to obtain shallow features (I f , M f );

[0044] The cooperative dual coordination module is composed of two parts, namely a pixel-to-pixel fine-grained local coordination module and an image-to-image global coordination module, which take shallow features (I f , M f ) as input to obtain pixel-level features I pix and global-level features I igh respectively, and finally obtain the reconstructed image through feature fusion I out = I pix · I IGH ;

[0045] A model training unit is used to train the reconstructed model constructed by using a loss function in combination with training set data;

[0046] A reconstruction unit is used to input the synthesized image and the corresponding mask in the test set into the trained reconstruction model to obtain the reconstructed image.

[0047] The beneficial effects of the present application compared with the prior art are:

[0048] (1) The present application provides a novel cooperative dual coordination high-resolution remote sensing image large-area cloud cover missing information reconstruction method, which constructs a CDH-RecNet model from the angles of pixel-to-pixel fine-grained local coordination and image-to-image global coordination, combines SC-PA and ground feature constraint features I SSIM in the pixel-level coordination module to achieve pixel-level coordination targets and obtain I pix , combines S2AM to propose a foreground-background attention mechanism in the global-level coordination module, and coordinates the differences and same information between the foreground-background region and the target image, and combines the ground feature constraint features I SSIM to obtain I igh , which alleviates the overall inconsistency problem in the reconstructed image. Then, in the ground feature constraint module, the same ground feature in the remote sensing image often appears repeatedly at different scales and across different scales, and a structure similarity constraint function is combined to construct a ground feature constraint module to obtain I SSIM , which solves the ground feature coordination problem. Finally, through I out = I pix · II GH , the feature fusion obtains the reconstructed image I out , which realizes the large-area cloud cover missing information reconstruction target of high-resolution images.

[0049] (2) The application designs a similarity constraint function and a gradient constraint function to maintain the overall consistency of the foreground region and the background region in the synthesized image. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is an overall structure diagram of the CDH-RecNet model of the embodiment of the application;

[0051] Figure 2 is a flowchart of the missing information reconstruction step based on the CDH-RecNet of the embodiment of the application;

[0052] Figure 3 is a structure diagram of the PFLH module of the embodiment of the application;

[0053] Figure 4 is a structure diagram of the IGH module of the embodiment of the application;

[0054] Figure 5 is a missing information reconstruction result of the CDH-RecNet under 70% cloud cover of Gaofen-1 (2m) of the embodiment of the application. DETAILED DESCRIPTION

[0055] In order to facilitate those skilled in the art to understand and implement the application, Figure 2 The method flowchart is described in further detail below, and it should be understood that the embodiment described herein is only used to illustrate and introduce the application, and is not used to limit the application.

[0056] See Figure 2 The application provides a cooperative dual-coordination high-resolution remote sensing image cloud cover missing information reconstruction method, which comprises the following steps:

[0057] Step 1, obtaining Gaofen-1 (2m) paired cloud cover images I ori and cloud-free images I T based on the time proximity principle, and obtaining mask data M corresponding to the cloud and cloud shadow, and then randomly dividing into a training set, a validation set and a test set;

[0058] First, according to the revisit period of the remote sensing image, time-proximity paired Gaofen-1 (2m) cloud cover images I and cloud-free images I are obtained to construct a training data set;

[0059] In the simulation experiment, the Berlin noise algorithm is used to simulate the cloud and cloud shadow to obtain the mask data M in the simulation experiment. In the test experiment, the real cloud cover M is drawn by using the ArcGIS software. It is worth noting that 0 in M is a non-cloud area, and 1 is a cloud cover area;

[0060] Step 2, cloud shadow image I ori and cloud-free image I T ; com ;

[0061] Based on the strategy of accelerating network training by block, the obtained pairs of I ori and I T are cropped to 256x256 respectively to obtain I ori , I T and M respectively corresponding training data sets;

[0062] The model input data is the synthetic image I com and M, wherein I com is obtained by formula 1.

[0063] I com = I ori •(1-M)+I T •M (1)

[0064] Step 3, taking the synthetic image I com and M as input, constructing a reconstruction model, including a shallow feature extraction module, a collaborative double coordination module and a feature constraint module;

[0065] The shallow feature extraction module is composed of multiple convolution layers and multiple residual layers, and takes I com and M as input to obtain shallow features (I f , M f );

[0066] For the shallow feature extraction module, the module is inspired by the U-Net and Residual Network structure, composed of three convolution layers (convolution kernel size: 7x7, 4x4, 4x4) and three residual layers (convolution kernel size: 3x3, 3x3, 3x3), and takes I com and M as input to obtain shallow features (I f , M f ).

[0067] The collaborative double coordination module is composed of two parts, namely the pixel-to-pixel fine-grained local coordination module and the image-to-image overall-to-overall global coordination module, taking the shallow features (I f , M f ) as input, combining the feature constraint I SSIM before upsampling to obtain the pixel-level feature I pix and the global-level feature I igh , and finally through I out = I pix ·I IGHAfter feature fusion, a reconstructed image is obtained;

[0068] For the pixel-to-pixel fine-grained local coordination module, the pixel-to-pixel coordination structure can effectively obtain local context information, and alleviate the problems of boundary effect, spectral loss and local inconsistency in the reconstructed image. In addition, the pixel self-correction module SC-PA is different from the channel attention and spatial attention, and the SC-PA can obtain 3D pixel-by-pixel attention while reducing the computational complexity, so the pixel-to-pixel fine-grained local coordination module with a stacked structure is constructed by using the SC-PA.

[0069] The pixel-to-pixel fine-grained local coordination module PFLH takes the shallow features (I f , M f ) as input, first performs feature fusion through the first SC-PA module, then performs calculation through two 3×3 convolution layers, and then performs feature fusion with the second SC-PA module, and finally splices the features in the channel dimension with the ground object constraint features I SSIM before upsampling, and finally performs upsampling through two upsampling layers (upsampling layers) and three convolution layers (convolution kernel size: 5×5, 5×5 and 7×7) to obtain I pix .

[0070] I pix =D(PFLH(I f , M f )) (2)

[0071] PFLH is a pixel-by-pixel coordination process, and D is an upsampling process.

[0072] In the first SC-PA, the module first calculates through two 1×1 convolution branches, named branch 1 and branch 2. In branch 1, two convolution sub-branches of 1×1 and 3×3 are used, named branch 3 and branch 4. Branch 3 maps the input to the range of 0-1 through the Sigmoid function, and the features obtained by branch 4 are multiplied with the features obtained by branch 3 to obtain the final features of branch 1. Then, in branch 2, first, 1×1 convolution calculation is performed, and then 3×3 convolution calculation is performed to obtain the features of branch 2. Finally, the features of branch 1 and branch 2 are spliced in the channel dimension, and then a layer of 1×1 convolution is performed to obtain the first 3D pixel-by-pixel feature (I′ fpix , M′ fpix ). After obtaining the features of the first SC-PA module (I′ fpix , M′ fpix ), (I′ fpix , M′ fpix ) is spliced with (I f , M fAfter being added together, they are merged into the PFLH module. The calculation process of the second SC-PA module is the same as that of the first SC-PA module in terms of calculation and fusion.

[0073] For the image-to-image global coordination module, high-resolution reconstructed images are rich in geographic texture information and have dense targets, often resulting in global inconsistency. Therefore, we first borrow the image-to-image coordination structure as the backbone to extract global information, and then borrow S2AM to construct the foreground-background attention module FBAM to obtain image features I of non-reconstructed regions. bg Information change characteristics between the reconstructed area and the target image I fg Invariant features between the reconstructed region and the target image I mix The image-to-image global coordination module IGH uses shallow features (I f M f As input, features are first fused using the first FBAM module, then processed through two 3×3 convolutional layers, and then fused with features from the second FBAM module. Finally, in the layer before upsampling, features are combined with ground feature constraints I in the channel dimension. SSIM The final upsampling is achieved by concatenating two upsampling layers and three convolutional layers (kernel sizes of 5×5, 5×5, and 7×7) to obtain I. igh .

[0074] The first FBAM module is (I f M f ) as input data. The FBAM module contains two branches, namely I fg Branches and I bg Branch. In I fg Within the branch, it is further divided into I mix Branches and I L Branch. I fg In the branch, I fg First, average pooling, max pooling, and two linear layer pairs (I) are used. f M f ) Calculate the information change characteristics between the reconstructed region and the target image, and then input the obtained information change characteristics between the reconstructed region and the target image into I. mix Furthermore, invariant features between target images in the reconstructed area are calculated using average pooling, max pooling, and two linear layers. Branch I L By incorporating self-learned scaling factors α and β into the traditional residual model, the extraction of difference features between the input and target images in the reconstruction region is enhanced, thereby improving reconstruction accuracy. The improved I... LThe structure is achieved by multiplying each of the two branches by α and β, respectively, where α and β are obtained through self-learning in the model. Then, I mix and I L Feature fusion is performed by addition, and then multiplied by M to obtain I. fg Characteristics of branches. bg The branch is calculated using average pooling, max pooling, and two linear layers, and then multiplied by 1-M to obtain I. bg Characteristics of branches. Finally, I fg Branches and I bg The features obtained from the branching are obtained by addition (I′) figh M′ figh The calculation process of the internal structure of FBAM can be summarized as Equation 3. (I′) figh M′ figh ) are added together with (I) f M f The second FBAM module performs feature fusion, and its calculation process and feature fusion method are the same as those of the first FBAM module.

[0075] I FBAM =M×[I L (I fg ((I f M f ))+I mix ((I f M f ))]+(1-M)×I bg ((I f M f )) (3)

[0077] In the formula, (I f M f ) is the input feature map, I FBAM FBAM features include (I′ figh M′ figh M is a binary mask;

[0078] The ground feature constraint module consists of multiple convolutional layers, with I com Using M as input data, we obtain the ground feature constraint feature I. SSIM ;

[0079] For the ground feature constraint module, through the coordination of the PFLH and IGH modules, we obtained pixel-level and global-level features I. pix and I igh The feature constraint module is implemented using five layers of convolution (with convolution kernels of 7×7, 4×4, 4×4, 4×4, and 3×3).

[0080] Step 4, the constructed reconstruction model is trained by using a loss function;

[0081] For the loss function, different levels of feature coordination are concerned through pixel-level and global-level harmonic processing, but the spectral variation information inside the surface feature is not concerned, which is an important reason for the loss of the reconstructed image spectrum, therefore, in the loss function, a gradient loss constraint function is constructed by using a Sobel operator, as shown in the following formula 4.

[0082]

[0083] In the formula, is a gradient loss constraint function, is a Sobel operator for calculating the gradient, out is an output reconstructed image, ori is a cloud-free real image, and L1 loss is L1 loss.

[0084] The present application also adds a reconstruction loss, which forces the reconstructed image to be infinitely close to the real image, and the reconstruction loss is shown in the following formula (5).

[0085]

[0086] In the formula, is a reconstruction loss, out is an output reconstructed image, ori is a cloud-free real image, and L1 loss is L1 loss.

[0087] I pix has the advantages of low resolution and rich local context information, igh has the advantages of high resolution and global context, and they are complementary in the reconstruction task, and theoretically, satisfactory reconstruction results can be obtained. However, the same surface feature often has a repeated phenomenon in different scales and across different scales, causing the reconstructed image I out has the problem of texture and spectral inharmony of the same surface feature, therefore, a similarity constraint function is constructed as shown in formula 6:

[0088]

[0089] In the formula, S is a structural similarity function, ori is a real cloud-free image down-sampling feature, ssim is a surface feature constraint module feature, is a similarity constraint function.

[0090] In summary, the loss function used in the present research is:

[0091]

[0092] In the formula, total loss, is a structural similarity function, is a gradient loss constraint function, is a reconstruction loss constraint function, and λ1, λ2, and λ3 are weights of the respective loss functions.

[0093] Step 5, the synthetic image I com and the mask M are input into the trained reconstruction model to obtain a reconstructed image, and experimental results are shown in the following table. Figure 5 .

[0094] The Figure 5 We performed experiments on 70% cloud cover of Gaofen-2 (2m) paired data, and the reconstruction results of RFR-Net, STAN, STS-CNN, and BSN as four comparative models. From the reconstruction effect of the blue ground object in the enlarged area, the method proposed in the application is superior to the comparative methods in terms of spectral information retention, texture information retention, and overall consistency. This also proves the effectiveness of the method proposed in the application.

[0095] In specific implementation, the method proposed in the technical solution of the application can be automatically run by a person skilled in the art using computer software technology, and a system device of the method, such as a computer readable storage medium storing a computer program corresponding to the technical solution of the application and a computer device including a computer program running device, should also be within the protection scope of the application.

[0096] The embodiment of the application further provides a cooperative dual coordination remote sensing image cloud cover missing reconstruction system, comprising the following units:

[0097] A data acquisition unit is configured to acquire remote sensing paired cloud cover images I ori and cloud-free images I T based on a time proximity principle, and acquire masks M corresponding to clouds and cloud shadows, and then randomly divide the images into a training set and a test set.

[0098] A synthetic image acquisition unit is configured to acquire a synthetic image I ori based on the cloud cover images I T and the cloud-free images I com .

[0099] A model construction unit is configured to construct a reconstruction model by taking the synthetic image I com and the mask M as inputs, and the reconstruction model comprises a shallow feature extraction module and a cooperative dual coordination module.

[0100] The shallow feature extraction module is composed of multiple convolution layers and multiple residual layers, and takes the synthetic image I com and the mask M as inputs to obtain shallow features (I f , Mf );

[0101] The dual coordination module is composed of two parts, namely a pixel-to-pixel fine-grained local coordination module and an image-to-image global coordination module, to obtain pixel-level features I f and global-level features I f , respectively, by taking I pix and M igh as inputs, and finally obtaining the reconstructed image through feature fusion of I out = I pix · I IGH ;

[0102] The model training unit is configured to train the constructed reconstruction model by using a loss function in combination with training set data.

[0103] The reconstruction unit is configured to input the synthesized image and the corresponding mask in the test set into the trained reconstruction model to obtain the reconstructed image.

[0104] The specific implementation manners of the units are the same as those of the steps, and the present application will not be described.

[0105] The above-described specific embodiments further specifically describe the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above-described specific embodiments are only used to illustrate the present application, and are not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for cooperative dual coordination remote sensing image cloud cover missing reconstruction, characterized in that, The method comprises the following steps: Step 1, obtain remote sensing cloud-pair blocked images based on the time proximity principle and cloud-free images and obtain the masks corresponding to the clouds and cloud shadows and then randomly divide them into training and test sets; Step 2, cloud shadow based imagery and cloud free imagery Image acquisition composite image ; Step 3, to the synthetic image and As input, the reconstruction model is constructed, including a shallow feature extraction module and a collaborative dual coordination module; The shallow feature extraction module is composed of multiple convolutional layers and multiple residual layers, and is used to obtain shallow features and as input ; The collaborative dual coordination module consists of two parts: a pixel-to-pixel fine-grained local coordination module and an image-to-image global coordination module, using shallow features... As input, pixel-level features are obtained respectively. and global features Ultimately passed The reconstructed image is obtained after feature fusion. Step 4, combining the training set data, training the constructed reconstruction model by using a loss function; The loss function comprises a gradient loss constraint function, a reconstruction loss constraint function, and a similarity constraint function; The calculation formula of the gradient loss constraint function is as follows: wherein, is a gradient loss constraint function, is a Sobel operator for calculating gradients, is the output reconstructed image, is the cloud-free real image, subscript is an L1 loss; The calculation formula of the reconstruction loss constraint function is as follows: wherein, is a reconstruction loss, is a reconstructed image output, is a cloud-free real image, subscript is an L1 loss; The calculation formula of the similarity constraint function is as follows: In the formula, is a structural similarity function, is a real cloud-free image down-sampling feature, is a ground object constraint module feature, is a similarity constraint function; The final total loss is as follows: wherein total loss, is a structural similarity function, is a gradient loss constraint function, is a reconstruction loss constraint function, , , is a weight for each loss function; Step 5, inputting the synthetic image and the corresponding mask in the test set into the trained reconstruction model to obtain a reconstructed image.

2. The method of claim 1, wherein the method is a synergic dual coordination remote sensing image cloud-occluded missing reconstruction method. The specific calculation formula of the synthetic image is as follows: 。 3. The method of claim 1, wherein the method further comprises: determining a cloud cover area in the first image; determining a cloud cover area in the second image; and determining a cloud cover area in the third image. The shallow feature extraction module comprises a plurality of convolution layers and a plurality of residual layers.

4. The method of claim 1, wherein the method further comprises: determining a cloud cover area in the first image; determining a cloud cover area in the second image; and determining a cloud cover area in the third image. The reconstruction model further comprises a ground object constraint module, which comprises five convolutional layers, the input of which is the synthetic image and , and the output is a ground object constraint feature .

5. The collaborative dual-coordinated remote sensing image cloud occlusion reconstruction method as described in claim 4, characterized in that: The pixel-to-pixel fine-grained local coordination module includes a pixel self-correction module SC-PA, a convolution module and an up-sampling module, and the specific processing process is as follows: shallow layer features After the processing of the SC-PA module, the original input features are added as the input of the convolution module, then a SC-PA module is input again and added with the input features, and then the ground object constraint features The channel dimension is spliced, and finally the pixel-level features are output through the up-sampling module ; The SC-PA module comprises two convolution branches, the processing process of the first convolution branch is: first, processing the input feature through a convolution layer, then multiplying the outputs of two small branches each comprising a convolution layer, and then inputting the multiplied outputs into the convolution layer, the second convolution branch comprises two convolution layers, the outputs of the two convolution branches are combined and then subjected to convolution processing, finally, the original input is added to obtain the final output; The convolution module comprises two convolution layers, and the up-sampling module comprises two up-sampling layers and three convolution layers.

6. The method of claim 4, wherein the method further comprises: The image-to-image global coordination module comprises a front background attention module FBAM, a convolution module and an up-sampling module, and the specific processing process is as follows: shallow layer features After the processing of the FBAM module, the original input features are added as the input of the convolution module, then a FBAM module is input again and added with the input features, and then the ground feature constraint features are added, and finally the features are output after the up-sampling module in the channel dimension ; The FBAM module includes two branches, respectively branch and branch, in branch, is divided into branch and branch; branch, First, the information change characteristics between the reconstructed region and the target image are calculated through the average pooling layer, the maximum pooling and the two linear layers Second, the information change characteristics between the reconstructed region and the target image are input into Further, the invariant characteristics between the reconstructed region and the target image are calculated through the average pooling layer, the maximum pooling and the two linear layers Branches By improving the traditional residual, a self-learning scaling factor is added. and This improves the extraction of difference features between the input and target images in the reconstruction area, thereby enhancing reconstruction accuracy; the improved... The structure is to multiply by each of the two branches. and ,in and It is obtained through self-learning in the model; then... and Feature fusion is performed by addition, followed by multiplication. get Characteristics of branches The branch is calculated using average pooling, max pooling, and two linear layers, and then... Multiply, we get Characteristics of branches; finally, Branches and The features obtained from the branches are obtained by addition. The calculation process of the internal structure of FBAM can be summarized as follows: In the formula, is an input feature map, is an FBAM feature, is a binary mask; The convolution module comprises two convolution layers, and the up-sampling module comprises two up-sampling layers and three convolution layers.

7. A system for synergic dual coordinated remote sensing image cloud-occluded missing reconstruction, for implementing a method for synergic dual coordinated remote sensing image cloud-occluded missing reconstruction according to any one of claims 1-6, characterized in that, The method comprises the following units: A data acquisition unit is configured to acquire remote sensing pair cloud-shading images based on a time proximity principle and cloud-free images and acquire masks corresponding to the clouds and cloud shadows and then randomly divide the images into a training set and a test set a synthetic image acquisition unit configured to acquire a synthetic image based on a cloud-occluded image and a cloud-free image an image obtaining a synthetic image ; Model building unit, used to synthesize images and As input, a reconstruction model is constructed, including a shallow feature extraction module and a collaborative dual coordination module; The shallow feature extraction module is composed of multiple convolutional layers and multiple residual layers, and takes the image as input to obtain shallow features and ;​ The collaborative dual coordination module consists of two parts: a pixel-to-pixel fine-grained local coordination module and an image-to-image global coordination module, using shallow features... As input, pixel-level features are obtained respectively. and global features Ultimately passed The reconstructed image is obtained after feature fusion. The model training unit is configured to combine the training set data, and train the constructed reconstruction model by using a loss function; The reconstruction unit is configured to input the synthetic image and the corresponding mask in the test set into the trained reconstruction model to obtain a reconstructed image.

Images