Single-to-multiple temporal remote sensing image thick cloud removal method based on coarse-to-fine framework

Abstract

The application discloses a single-to-multiple time-phase remote sensing image thick cloud removing method based on a coarse-to-fine framework, and comprises the following steps: acquiring a pair of multiple time-phase remote sensing images, target images and reference images of the same place at different time phases; performing linear regression analysis on all cloud-free pixels in the target images and corresponding pixels in the reference images; performing global color transformation processing on the reference images according to the regression coefficients; taking the reference images subjected to the global color transformation processing as the input of an internal constraint-based convolutional neural network model; the internal constraint-based convolutional neural network model is designed to have a multi-scale feature extraction and fusion module and an auxiliary loss function about cloud region pixels; the output result of the model is fused with the target images by using a Poisson image fusion method to obtain an image of the thick cloud removing result which is color consistent and spatially continuous. The method can eliminate artifacts in the recovery result and accurately recover the spatial and color information of the cloud-shielded region.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method for removing thick clouds from one-to-many temporal remote sensing images based on a coarse-to-fine framework. Background Technology

[0002] Due to the rapid development of satellite technology, satellite remote sensing imagery plays a crucial role in a wide range of applications, such as target detection, land cover classification, and disaster monitoring. However, due to the influence of atmospheric conditions on satellite sensors, thick clouds are commonly used to obscure optical remote sensing images, severely hindering their subsequent applications. Therefore, removing thick clouds from remote sensing images to improve their quality and usability is both necessary and valuable.

[0003] For thick cloud removal tasks, since the ground feature information in the thick cloud region of the target image is completely lost, an additional cloudless image of the same location at a different time is usually needed as a reference. Traditional thick cloud removal methods based on local similarity pixel regression assume that the local similar pixels of each individual missing pixel are linearly related in the target image and the cloudless reference image. These methods often fail to obtain color-consistent and spatially continuous thick cloud removal results for complex and heterogeneous scenes with large areas of thick clouds.

[0004] Deep learning-based thick cloud removal methods can fully utilize the powerful nonlinear expressive capabilities of deep convolutional neural networks (CNNs), enabling them to handle images with complex terrain features and preserve the spatial continuity of these features. However, these methods require a large external image dataset for training the network for each different sensor image; without a training dataset, they cannot function properly. For thick cloud removal in single-to-multi-temporal remote sensing images, based on deep image prior theory, constraints from cloudless reference images and cloudless pixels within the target image can be used to convert random noise into a cloudless image using a convolutional neural network. Alternatively, using a reference image as input, constraints from cloudless pixels within the target image can be used to convert the reference image into a cloudless image with the same color as the target image using a convolutional neural network. However, these methods have a drawback: the restoration of cloudy region pixels lacks direct constraints, resulting in unstable restoration results and susceptibility to artifacts. Furthermore, these methods directly copy the pixels corresponding to cloud areas in the restored result into the target image to replace cloud pixels, which can easily lead to seams between the cloudless and restored regions in the final result. Summary of the Invention

[0005] The purpose of this invention is to provide a method for removing thick clouds from one-to-many temporal remote sensing images based on a coarse-to-fine framework, which solves the problems of artifacts and stitching seams in the thick cloud removal process of deep learning in the prior art, and achieves better thick cloud removal results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention provides a method for thick cloud removal in one-to-many temporal remote sensing images based on a coarse-to-fine framework, comprising the following steps:

[0008] Acquire a one-to-many temporal remote sensing image, including: a target image and a reference image at the same location but at different times; the target image is a remote sensing image with thick cloud cover; the reference image is a cloudless remote sensing image.

[0009] Perform linear regression analysis on all cloudless pixels in the target image and their corresponding pixels in the reference image;

[0010] Based on the regression coefficients of the linear regression analysis, the reference image is subjected to global color transformation processing;

[0011] The reference image, after global color transformation, is used as input to a convolutional neural network model based on internal constraints. The convolutional neural network model based on internal constraints is designed with a multi-scale feature extraction and fusion module and an auxiliary loss function for pixels in the cloud region.

[0012] The output of the convolutional neural network model based on internal constraints and the target image are fused using the Poisson image fusion method to obtain an image with consistent color and spatial continuity after thick cloud removal.

[0013] Furthermore, after acquiring a one-to-many temporal remote sensing image, the method further includes: acquiring a binary cloud mask image in the target image.

[0014] Further, based on the regression coefficients from the linear regression analysis, a global color transformation is performed on the reference image; including:

[0015] Using the obtained regression coefficients, a global color transformation is performed on the reference image to obtain a coarse, cloudless result C with colors close to the target image.

[0016] C = a·R + b (1)

[0017] Where R is the reference image, and the regression coefficients a and b are calculated using the least squares method:

[0018]

[0019]

[0020] Among them, R cfp and T cfp These are the pixels in the same cloudless region in the reference image R and the target image T, respectively. This indicates the operation of calculating the mean.

[0021] Furthermore, the processing procedure of the convolutional neural network model based on internal constraints includes:

[0022] The reference image C, after global color transformation, is used as the input image; a fake target image F is generated through a CNN network consisting of convolutional layers, activation function layers, a multi-scale feature extraction and fusion module, convolutional layers, and normalization layers.

[0023] The network is optimized by using the pixel loss of the cloudless region in the target image T and the pixel loss of the corresponding cloud region in the processed reference image C, so that the generated result gradually approaches the true value of the target image T.

[0024] Furthermore, the multi-scale feature extraction and fusion module extracts multi-scale features through three independent branches, each branch consisting of four residual blocks;

[0025] The kernel sizes for the three independent branches are 1×1, 3×3, and 5×5, respectively.

[0026] The features output from the three independent branches are aggregated together through a connection layer and then input into a CAM layer for feature fusion.

[0027] Furthermore, the auxiliary loss function for the pixels in the cloud region includes:

[0028] The pixels in the cloudless regions of the target image T are constrained by the network, and the loss of the cloudless pixels is L. cfp Recorded as:

[0029] L cfp =||(1-M)⊙(TF)||1 (4)

[0030] Where ||·||1 is the L1 norm, ⊙ is the pixel dot product; M is the binary cloud mask image, where 1 represents cloud pixels and 0 represents cloudless pixels;

[0031] Using pixels in the cloud region of the reference image C after global color transformation, an auxiliary loss L for the cloud region pixels is designed. cp :

[0032] L cp =||M⊙(CF)||1 (5)

[0033] The overall objective function is as follows:

[0034] L = L cfp +λL cp (6)

[0035] Here, λ is a positive weight value used to control the relative importance of the two losses.

[0036] Furthermore, the output of the convolutional neural network model based on internal constraints and the target image are fused using the Poisson image fusion method; including:

[0037] Seamless cloud removal results are generated using the Poisson image fusion formula, which includes a gradient constraint and a Dirichlet boundary condition:

[0038]

[0039] Where T′ represents the final fusion result. Here, Ω represents the gradient operator, and Ω is the cloudy region. denoted as the outer boundary of the clouded region; F represents the image generated by the convolutional neural network model based on internal constraints.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention provides a method for thick cloud removal in single-to-multi-temporal remote sensing images based on a coarse-to-fine framework. This method can remove thick clouds from single-to-multi-temporal remote sensing images without requiring additional training datasets. Furthermore, this invention introduces a multi-scale convolutional feature extraction and fusion module, along with an auxiliary loss term related to cloud regions, into the local refinement network. This eliminates artifacts in the restored results and accurately recovers the spatial and color information of cloud-occluded areas. Finally, this invention uses a Poisson image fusion method to obtain thick cloud removal results that are color-consistent and spatially continuous. Attached Figure Description

[0042] Figure 1 A flowchart of a method for removing thick clouds from one-to-many temporal remote sensing images based on a coarse-to-fine framework, provided for embodiments of the present invention;

[0043] Figure 2 A schematic diagram illustrating the principle of a method for removing thick clouds from one-to-many temporal remote sensing images based on a coarse-to-fine framework, provided in an embodiment of the present invention.

[0044] Figure 3 A schematic diagram of the network structure and loss composition used in the local refining step provided in an embodiment of the present invention;

[0045] Figures 4a-4d Four Landsat 8 images of thick clouds in visible light;

[0046] Figures 5a-5d yes Figures 4a-4d Corresponding multi-temporal cloudless reference images;

[0047] Figures 6a-6d yes Figures 4a-4dThe resulting image after thick cloud removal. Detailed Implementation

[0048] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0049] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] Reference Figure 1 As shown, the present invention provides a method for removing thick clouds from one-to-many temporal remote sensing images based on a coarse-to-fine framework, comprising the following steps:

[0052] S10. Acquire a one-to-many temporal remote sensing image, including: a target image and a reference image at the same location but at different times; the target image is a remote sensing image with thick cloud cover; the reference image is a cloudless remote sensing image.

[0053] S20. Perform linear regression analysis on all cloudless pixels in the target image and their corresponding pixels in the reference image; perform global color transformation on the reference image based on the regression coefficients of the linear regression analysis.

[0054] S30. The reference image after global color transformation is used as the input of the convolutional neural network model based on internal constraints; the convolutional neural network model based on internal constraints is designed with a multi-scale feature extraction and fusion module and an auxiliary loss function for pixels in the cloud region.

[0055] S40. The output of the convolutional neural network model based on internal constraints and the target image are fused using the Poisson image fusion method to obtain an image with consistent color and spatial continuity of thick cloud removal result.

[0056] This invention utilizes all cloud-free pixels in the target image and their corresponding pixels in the reference image to perform linear regression. The calculated regression coefficients are then used to perform a global color transformation on the reference image, yielding a coarse result with colors similar to the target image. Then, a convolutional neural network (CNN) based on internal constraints is used to refine the coarse result. Specifically, this invention incorporates a multi-scale feature extraction and fusion module (MFEFM) and an auxiliary loss term related to cloud regions to eliminate artifacts. Finally, this invention uses a Poisson image fusion method to generate a seamless thick cloud removal result.

[0057] The technical solution of the present invention will be illustrated more clearly by the following examples:

[0058] The technical principle of this invention is as follows: Figure 2 As shown. The hardware used, such as the computer configuration, includes an Intel Core i7-6700K processor, an Nvidia GeForce GTX 1080 graphics processor with a clock speed of 4.0GHz, 16GB of memory, and an operating system of Ubuntu 20.04. The thick cloud removal method of this invention is implemented, for example, through a program developed using the Python language. The CNN-based local refinement is implemented using the PyTorch framework. This invention removes thick clouds from single-to-multi-temporal remote sensing images in a coarse-to-fine manner, gradually obtaining a cloud-free result with consistent color and spatial continuity. Specifically, it includes the following steps:

[0059] Step 1: Data Preparation

[0060] The method proposed in this invention is for thick cloud removal from single-to-multi-temporal remote sensing images. For a remote sensing image obscured by thick clouds, denoted as target image T, since the ground feature information in the thick cloud region of T is completely lost, a cloud-free image from the same location but a different temporal phase is needed to assist in the thick cloud removal; this is denoted as reference image R. Furthermore, this invention only restores the ground feature information in the cloud-covered regions of T, without altering the ground feature information in the cloud-free regions. Therefore, a binary cloud mask corresponding to T is also required, denoted as M. M can be obtained manually or automatically using existing cloud detection methods; this disclosure does not limit this approach.

[0061] Step 2: Global Color Transformation

[0062] The target image T and the reference image R were acquired at different times, and due to the different imaging conditions, they usually have color differences. Therefore, firstly, linear regression is performed based on the clear, cloud-free pixels in T and the corresponding pixels in R. Then, the obtained regression coefficients are used to perform a global color transformation on R to obtain a coarse, cloud-free result C with a color close to T.

[0063] C = a·R + b (1)

[0064] Where R is the reference image, and the regression coefficients a and b are calculated using the least squares method:

[0065]

[0066]

[0067] Among them, R cfp and T cfp These are the pixels in the same cloudless region in the reference image R and the target image T, respectively. This indicates the operation of calculating the mean.

[0068] Step 3: Local Refinement Based on CNN

[0069] C and T have generally similar colors, but there are still significant color differences in certain local areas. Therefore, in this embodiment, a CNN network is used to refine C locally, resulting in a more accurate and spatially continuous refined cloudless image F.

[0070] Among them, the proposed CNN-based refined model is as follows: Figure 3 As shown, image C is used as input, and a fake target image F is generated through a CNN network containing an MFEFM module. The pixel loss L in the cloudless region of image T is utilized. cfp And the pixel loss L in the corresponding cloud region of C cp The network is optimized to gradually approach the true value of T. Details of the network structure, MFEFM, and loss function are described below.

[0071] 1) Network Structure

[0072] The network structure proposed in this invention is as follows: Figure 3 As shown, the input data is first transformed into the feature space through a convolutional layer and a ReLU activation layer; then, multi-scale residual convolutional blocks are used for further feature extraction, and the features are fused through a channel attention module (CAM); finally, a convolutional layer is used to generate the output, which is then normalized using the sigmoid function. Throughout the network, all convolutional layers have a stride of 1 and 32 channels; except for the multi-scale feature extraction and fusion module, all convolutional kernels are 3x3 in size.

[0073] 2) Multi-scale feature extraction and fusion

[0074] Due to the diversity of ground features and clouds in remote sensing images, the restoration of cloud regions typically relies on contextual information at different scales. Therefore, a multi-scale feature extraction and fusion module is introduced into the network. Feature extraction is performed using convolutional kernels of different scales to obtain features with different receptive fields. Three independent branches are used to extract multi-scale features, each consisting of four residual blocks with convolutional kernel sizes of 1x1, 3x3, and 5x5, respectively. These features are then aggregated. Considering that the restoration of different images and cloud distributions emphasizes different scales, a CAM (Configuration-Based Feature Fusion) module is used for feature fusion.

[0075] CAM first performs global mean pooling on the input feature map, then uses two 1x1 convolution operations, and then obtains the scaling factor for each channel through the sigmoid function, and uses it to multiply the input multi-scale feature map, thereby rescaling the feature values ​​of each channel.

[0076] 3) Loss Function

[0077] The network's goal is to generate a cloudless image F from C that is as close as possible to T. Therefore, the pixels in the cloudless regions of T can be used to constrain the network, and the loss for the cloudless pixels is L. cfp It can be written as:

[0078] L cfp =||(1-M)⊙(TF)||1 (4)

[0079] Where ||·||1 is the L1 norm, ⊙ is the pixel dot product, and M is the binary cloud mask image, where 1 represents a cloud pixel and 0 represents a cloudless pixel.

[0080] The objective of this invention is to recover cloud pixels in T, but it is impossible to obtain constraints on cloud regions from T. However, observations show that pixels in C containing cloud regions are somewhat similar to the ground truth in T, and artifacts are rarely observed. Therefore, utilizing pixels in C containing cloud regions, an auxiliary loss L for cloud region pixels is designed. cp :

[0081] L cp =||M⊙(CF)||1 (5)

[0082] Our overall objective function is as follows:

[0083] L = L cfp +λL cp (6)

[0084] Here, λ is a positive weight value used to control the relative importance of the two losses. Considering that C is not the true value, the training process is divided into two stages, L... cp Use only in the first stage.

[0085] 4) Training process

[0086] The overall objective function was optimized using stochastic gradient descent (6). Before training, the input image was normalized to [0, 1]. The weights of the convolution kernel were initialized with a Gaussian function with a mean of 0 and a variance of 0.01, and the bias was initialized with a fixed value of 0.01. The Adam optimization algorithm was used, with the exponential decay rate of the first moment estimation set to 0.9, the exponential decay rate of the second moment estimation set to 0.999, and the learning rate set to 0.01. The entire training process consisted of 600 iterations, with 400 iterations in the first stage and λ=2 in the second stage; and 200 iterations in the second stage and λ=0 in the third stage. After training, the output of the last iteration was the locally optimized, refined cloudless result F.

[0087] Step 4: Poisson Image Fusion

[0088] If the corresponding pixels are directly cloned from F to replace the cloudy pixels in T, it may result in a seam between the cloudy and cloudless areas in the restored result. Therefore, in this invention, Poisson Image Fusion (PIF) is used to generate a seamless declouding result. PIF is mathematically expressed as Equation (7), which includes a gradient constraint and a Dirichlet boundary condition.

[0089]

[0090] Where T′ represents the final fusion result. Here, Ω represents the gradient operator, and Ω is the cloudy region. Let T' be the outer boundary of the clouded region; F is the image generated by the convolutional neural network model based on internal constraints. A global optimization process is used to solve equation (7), and the resulting solution T' guarantees that it has a gradient as close as possible to F (i.e., image detail) on the clouded region, and has a consistent intensity with T on the boundary of the clouded region, thus obtaining a satisfactory cloudless result with accurate and consistent brightness. Therefore, T' is the final thick cloud removal result obtained by this method.

[0091] like Figures 4a-4d Four Landsat 8 images of thick clouds in visible light. Figures 5a-5d yes Figures 4a-4dCorresponding multi-temporal cloudless reference images; this invention utilizes the linear regression coefficients between pixels in the same cloudless region in the target image and the reference image to perform a global color transformation on the reference image, obtaining a coarse restoration result. Then, a convolutional neural network based on internal constraints is used to refine the coarse result, which does not require training with an external training dataset. This invention improves the network's performance by designing a multi-scale convolutional feature extraction and fusion module in the network and introducing an auxiliary loss related to cloud regions. After training with internal data, the network finally outputs a more accurate cloudless result. Finally, this invention uses the Poisson image fusion method to fuse the refined result with the target image, thereby obtaining a thick cloud removal result with consistent color and spatial continuity; as shown. Figures 6a-6d yes Figures 4a-4d The resulting image after thick cloud removal.

[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for thick cloud removal in single-to-multi-temporal remote sensing images based on a coarse-to-fine framework, characterized in that, Includes the following steps: Acquire a one-to-many temporal remote sensing image, including: a target image and a reference image at the same location but at different times; the target image is a remote sensing image with thick cloud cover; the reference image is a cloudless remote sensing image. Linear regression analysis is performed on all cloudless pixels in the target image and their corresponding pixels in the reference image; based on the regression coefficients of the linear regression analysis, a global color transformation is performed on the reference image. The reference image, after global color transformation, is used as input to a convolutional neural network model based on internal constraints. The convolutional neural network model based on internal constraints is designed with a multi-scale feature extraction and fusion module and an auxiliary loss function for pixels in the cloud region. The output of the convolutional neural network model based on internal constraints and the target image are fused using the Poisson image fusion method to obtain an image with consistent color and spatial continuity after thick cloud removal. Specifically, based on the regression coefficients of linear regression analysis, a global color transformation process is performed on the reference image; including: Using the obtained regression coefficients, a global color transformation is performed on the reference image to obtain a coarse, cloudless result C with colors close to the target image. (1) Where R is the reference image, and the regression coefficients a and b are calculated using the least squares method: (2) (3) Among them, R cfp and T cfp These are the pixels in the same cloudless region in the reference image R and the target image T, respectively. This indicates the operation of calculating the mean; The processing steps of a convolutional neural network model based on internal constraints include: The reference image C, after global color transformation, is used as the input image; a fake target image F is generated through a CNN network consisting of convolutional layers, activation function layers, a multi-scale feature extraction and fusion module, convolutional layers, and normalization layers. The network is optimized by using the pixel loss of the cloudless region in the target image T and the pixel loss of the corresponding cloud region in the processed reference image C, so that the generated result gradually approaches the true value of the target image T. The multi-scale feature extraction and fusion module extracts multi-scale features through three independent branches, each branch consisting of four residual blocks; the convolution kernel sizes of the three independent branches are respectively... and The features output from the three independent branches are aggregated together through a connection layer and then input into a CAM layer for feature fusion.

2. The method for thick cloud removal in single-to-multi-temporal remote sensing images based on a coarse-to-fine framework according to claim 1, characterized in that, After acquiring a one-to-many temporal remote sensing image, the method further includes: acquiring a binary cloud mask image in the target image.

3. The method for thick cloud removal in single-to-multi-temporal remote sensing images based on a coarse-to-fine framework according to claim 1, characterized in that, The auxiliary loss function for pixels in the cloud region includes: The pixels in the cloudless regions of the target image T are constrained by the network, and the loss of the cloudless pixels is L. cfp Recorded as: (4) in, It is an L1 norm. is the pixel dot product; M is the binary cloud mask image, where 1 represents cloud pixels and 0 represents no-cloud pixels; Using pixels in the cloud region of the reference image C after global color transformation, an auxiliary loss L for the cloud region pixels is designed. cp : (5) The overall objective function is as follows: (6) in, It is a positive weight value used to control the relative importance of the two losses.

4. The method for thick cloud removal in single-to-multi-temporal remote sensing images based on a coarse-to-fine framework according to claim 1, characterized in that, The Poisson image fusion method is used to fuse the output of the convolutional neural network model based on internal constraints with the target image; including: Seamless cloud removal results are generated using the Poisson image fusion formula, which includes a gradient constraint and a Dirichlet boundary condition: (7) in, For the final fusion result, For gradient operators, The area is cloud-covered. denoted as the outer boundary of the clouded region; F represents the image generated by the convolutional neural network model based on internal constraints.

Images