Dual network driven self-supervised thick cloud removal method and system for multi-temporal remote sensing images

Abstract

The invention relates to the field of image processing, and proposes a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method and system. A dual-network architecture cloud removal model is designed, and the spatial range of cloud components is constrained according to an initial model. The cloud removal initial model adaptively generates a mask to accurately separate a clean area from a cloudy area according to the mask. Then, an image component network and a cloud component network are used to capture the complex features of the image component and the cloud component, respectively, thereby further improving the accuracy and robustness of the separation. In addition, because of the separate capture of cloud components, the problem of insufficient capture capability of clouds of different forms is effectively avoided. An image component network is constructed according to a guided feature generator and a spatial feature restorer, and local detail features are effectively captured. At the same time, the destruction of spatial information is avoided, and the authenticity of the image after cloud removal is further improved. The invention improves the accuracy of the remote sensing image thick cloud removal method.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method and system. Background Art

[0002] With the rapid development of remote sensing satellite earth observation technology, the quality of captured remote sensing images is constantly improving in both spatial resolution and spectral resolution, and plays an important role in various fields such as mineral exploration, military reconnaissance, environmental monitoring and urban planning. However, since the annual average cloud coverage of the earth is as high as 66%, this greatly affects the quality of remote sensing image observation and the implementation of various downstream tasks. Therefore, cloud removal of remote sensing images is a crucial link.

[0003] In the existing technology, traditional cloud removal methods are difficult to adapt to large areas of cloud coverage, resulting in the inability to fill large gaps and poor detail recovery capabilities. Cloud removal methods based on basic model optimization are also unable to meet image quality requirements due to their insufficient adaptability to complex cloud forms, heavy reliance on model assumptions and high computational costs. Due to changes in time and space, the size, distribution, and thickness of clouds are also different. Existing deep models cannot meet the ability to capture clouds of different forms and can only characterize global and local features separately.

[0004] Therefore, how to design a thick cloud removal method for remote sensing images to meet the ability to capture clouds of different forms and improve the accuracy of cloud removal. Summary of the invention

[0005] Based on this, the present invention proposes a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method and system. A dual-network architecture cloud removal model is designed. According to the initial model, the spatial range of the cloud component is constrained. The cloud removal initial model adaptively generates a mask to accurately separate the clean area from the cloudy area according to the mask. Then, the image component network and the cloud component network are used to capture the complex features of the image component and the cloud component, respectively, thereby further improving the accuracy and robustness of the separation. In addition, because the cloud component is captured separately, the problem of insufficient ability to capture clouds of different forms is effectively avoided. The image component network is constructed according to the guided feature generator and the spatial feature restorer, and the local detail features are effectively captured. At the same time, the destruction of spatial information is avoided, and the authenticity of the image after cloud removal is further improved. The present invention improves the accuracy of the remote sensing image thick cloud removal method.

[0006] The present invention proposes a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method, comprising:

[0007] Acquiring remote sensing image data and performing preprocessing, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series;

[0008] Separating a clean area image and a cloudy area image in the remote sensing image data according to an initial cloud removal model, wherein the initial cloud removal model adaptively generates a mask to divide the clean area and the cloudy area according to the mask;

[0009] capturing image component features in the clean region image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer;

[0010] capturing cloud component features in the cloud region image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with skip connections;

[0011] A final cloud-removed image is acquired according to the image component features and the cloud component features.

[0012] In summary, according to the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method, a dual-network architecture cloud removal model is designed, the spatial range of the cloud component is constrained according to the initial model, and the cloud removal initial model adaptively generates a mask to accurately separate the clean area from the cloudy area according to the mask, and then the image component network and the cloud component network are used to capture the complex features of the image component and the cloud component, respectively, thereby further improving the accuracy and robustness of the separation. In addition, because of the separate capture of the cloud component, the problem of insufficient ability to capture clouds of different forms is effectively avoided. The image component network is constructed according to the guided feature generator and the spatial feature restorer, and the local detail features are effectively captured. At the same time, the destruction of spatial information is avoided, and the authenticity of the image after cloud removal is further improved. The present invention improves the accuracy of the remote sensing image thick cloud removal method. Specifically, remote sensing image data is acquired and preprocessed, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series, and a clean area image and a cloud area image in the remote sensing image data are separated according to an initial cloud removal model, wherein the initial cloud removal model adaptively generates a mask to divide the clean area and the cloud area according to the mask, and accurately separates the clean area from the cloud area, thereby improving the accuracy and effectiveness of overall cloud removal, and capturing image component features in the clean area image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer, and while retaining the spatial information of the image, the image detail information is enriched, thereby making the cloud-removed image more in line with the actual situation, and capturing cloud component features in the cloud area image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with jump connections, and cloud components are captured separately, thereby effectively avoiding the problem of insufficient ability to capture clouds of different forms, and obtaining a final cloud-removed image according to the image component features and the cloud component features. The present invention improves the accuracy of the thick cloud removal method for remote sensing images.

[0013] Furthermore, the step of separating the clean area image and the cloudy area image in the remote sensing image data according to the cloud removal initial model specifically includes:

[0014] The cloud removal initialization model separates clean area images and cloud area images in remote sensing image data;

[0015] The specific algorithm of the cloud removal initial model is as follows:

[0016] ,

[0017] in, represents the observed image contaminated by clouds, represents a binary mask, represents a clean image, represents the cloud component, represents the Hadamard product.

[0018] Furthermore, the step of capturing the image component features in the clean area image according to the image component network specifically includes:

[0019] Inputting the clean region image into an image component network, wherein the image component network includes a guided feature generator and a spatial feature restorer;

[0020] The clean area image is input as a guide image into the guide feature generator, and the guide feature generator generates multi-scale guide features according to an attention mechanism;

[0021] The spatial feature restorer restores spatial detail information of the multi-scale guided features to obtain image component features;

[0022] The specific algorithm of the image component feature is as follows:

[0023] ,

[0024] in, represents the image component features, represents the image component network, represents the image component network parameters, represents a random tensor, Represents a guide image.

[0025] Furthermore, the guiding feature generator specifically includes:

[0026] The guided feature generator is a series of encoding blocks and a series of decoding blocks with skip connections;

[0027] The guided feature generator convolves the guided image according to the time series to obtain the initial input features;

[0028] The encoding block series encodes the initial input features according to the spatial attention mechanism to obtain multiple encoding features. The specific algorithm of the encoding block series is as follows:

[0029] ,

[0030] in, represents the encoding features of the next encoding layer, Indicates the layer number, represents a series of coded blocks, Represents the encoding features of the current layer;

[0031] The coding features are decoded according to the decoding block series to obtain multiple decoding features, each decoding layer in the decoding block series has a unique corresponding coding layer, the coding block series corresponds to the layers in the decoding block series in reverse order, and the last coding layer in the reverse order corresponds to the initial decoding layer. The specific algorithm of the decoding block series is as follows:

[0032] ,

[0033] in, represents the decoding features of the next decoding layer, Indicates the layer number, represents a series of decoded blocks, represents the decoded features of the current layer, represents a skip connection, Indicates the encoding features of the encoding layer corresponding to the current decoding layer.

[0034] Furthermore, the spatial feature restorer specifically includes:

[0035] The spatial feature restorer includes a guided feature attention module and a feature compensation module;

[0036] Except for the initial decoding layer, each decoding layer has a unique corresponding guided feature attention module;

[0037] Except for the last coding layer, each coding layer has a unique corresponding feature compensation module;

[0038] The guided feature attention module generates a decoding feature weight according to a gating mechanism, and the branch of the guided feature attention module for generating the decoding feature weight includes a two-dimensional convolutional layer, a LeakyRelu activation function layer, and a sigmoid activation function layer;

[0039] Multiplying the decoded features with the intermediate features of the spatial feature restorer according to the decoded feature weights to align the output image of the guided feature attention module and the guided image of the guided feature generator;

[0040] The specific algorithm of the guided feature attention module is as follows:

[0041]

[0042] in, represents the initial input features that guide the feature attention module, represents the guided feature attention module, represents a random tensor, represents the encoding features of the final encoding layer, represents the output feature of the guided feature attention module of the current layer, Represents the output features of the feature compensation module of the current layer, Represents the decoded features of the current layer;

[0043] The feature compensation module performs weighted fusion of the deep spatial features and the high-level decoding features, wherein the high-level decoding features are decoding features other than the decoding features of the initial decoding layer;

[0044] The specific algorithm of the feature compensation module is as follows:

[0045] ,

[0046] in, Represents the output features of the feature compensation module of the current layer, represents the feature compensation module, represents the output feature of the guided feature attention module of the previous layer, Indicates the encoding features of the encoding layer corresponding to the current decoding layer.

[0047] Furthermore, the step of capturing cloud component features in the cloud area image according to the cloud component network specifically includes:

[0048] Capturing cloud component features in images of cloudy areas based on a cloud component network;

[0049] The cloud component network is a U-Net structure, and network parameters are optimized according to a self-supervision mechanism. The cloud component network includes an encoder and a decoder with jump connections, the encoder includes a convolution layer, a downsampling layer, and a nonlinear activation function layer, and the decoder includes a convolution layer, an upsampling layer, and a nonlinear activation function layer;

[0050] The specific algorithm of the cloud component network is as follows:

[0051] ,

[0052] in, represents the cloud component characteristics, represents the cloud component network, represents the cloud component network parameters, represents random noise.

[0053] Furthermore, the step of obtaining the final cloud-removed image according to the image component features and the cloud component features specifically includes:

[0054] The final de-clouded image is obtained according to the image component features and the cloud component features. The specific algorithm for obtaining the final de-clouded image is as follows:

[0055] ,

[0056] ,

[0057] Among them, min means minimization function, represents the image component network parameters, represents the cloud component network parameters, represents a binary mask, represents the observed image contaminated by clouds, represents the Hadamard product, represents the image component network, represents a random tensor, represents the guide image, represents the cloud component network, represents random noise, represents the Frobenius norm, represents the nuclear norm, represents the 2,1 norm, and represents the regularization parameter, Represents a clean image.

[0058] The present invention proposes a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal system, comprising:

[0059] A preprocessing module, used for acquiring remote sensing image data and performing preprocessing, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series;

[0060] An initial separation module, used for separating the clean area image and the cloudy area image in the remote sensing image data according to the cloud removal initial model, wherein the cloud removal initial model adaptively generates a mask to divide the clean area and the cloudy area according to the mask;

[0061] An image component module, configured to capture image component features in the clean region image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer;

[0062] A cloud component module, configured to capture cloud component features in the cloud area image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with skip connections;

[0063] A result generation module is used to obtain a final de-clouded image according to the image component features and the cloud component features.

[0064] The present invention also provides a storage medium, which stores one or more programs. When the programs are executed by a processor, the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method as described above is implemented.

[0065] The present invention also provides a computer device, the computer device comprising a memory and a processor, wherein:

[0066] The memory is used to store computer programs;

[0067] When the processor is used to execute the computer program stored in the memory, the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flow chart of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention;

[0069] Figure 2 A flow chart of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the second embodiment of the present invention;

[0070] Figure 3 A schematic diagram of the structure of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal system proposed in the third embodiment of the present invention;

[0071] Figure 4 A schematic diagram of the coding layer structure of the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention;

[0072] Figure 5 A schematic diagram of the decoding layer structure of the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention;

[0073] Figure 6 A schematic diagram of the structure of the guided feature attention module of the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention;

[0074] Figure 7 A schematic diagram of the structure of a feature compensation module of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention;

[0075] Figure 8 This is a graph showing comparative experimental results of the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the second embodiment of the present invention.

[0076] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0077] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0078] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0080] See also Figure 1 , which is a flow chart of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the first embodiment of the present invention, the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method comprises steps S01 to S05, wherein:

[0081] Step S01: Acquire remote sensing image data and perform preprocessing;

[0082] Step S02: Separating clean area images and cloud area images in remote sensing image data according to the initial cloud removal model;

[0083] It should be noted that the cloud removal initial model in this embodiment separates the clean area image and the cloud area image in the remote sensing image data;

[0084] The specific algorithm of the cloud removal initial model is as follows:

[0085] ,

[0086] in, represents the observed image contaminated by clouds, represents a binary mask, represents a clean image, represents the cloud component, Represents the Hadamard product.

[0087] Step S03: capturing image component features in the clean area image according to the image component network;

[0088] It should be noted that in this embodiment, the clean area image is input into the image component network, and the image component network includes a guided feature generator and a spatial feature restorer;

[0089] The clean area image is input as a guide image into the guide feature generator, and the guide feature generator generates multi-scale guide features according to an attention mechanism;

[0090] The spatial feature restorer restores spatial detail information of the multi-scale guided features to obtain image component features;

[0091] The specific algorithm of the image component feature is as follows:

[0092] ,

[0093] in, represents the image component features, represents the image component network, represents the image component network parameters, represents a random tensor, Indicates the guiding image;

[0094] The guided feature generator is a series of encoding blocks and a series of decoding blocks with skip connections;

[0095] For the detailed structure of a single coding layer in a coding block series, please refer to Figure 4 For the detailed structure of a single decoding layer in a decoding block series, please refer to Figure 5 ;

[0096] The guided feature generator convolves the guided image according to the time series to obtain the initial input features;

[0097] The encoding block series encodes the initial input features according to the spatial attention mechanism to obtain multiple encoding features. The specific algorithm of the encoding block series is as follows:

[0098] ,

[0099] in, represents the encoding features of the next encoding layer, Indicates the layer number, represents a series of coded blocks, Represents the encoding features of the current layer;

[0100] The coding features are decoded according to the decoding block series to obtain multiple decoding features, each decoding layer in the decoding block series has a unique corresponding coding layer, the coding block series corresponds to the layers in the decoding block series in reverse order, and the last coding layer in the reverse order corresponds to the initial decoding layer. The specific algorithm of the decoding block series is as follows:

[0101] ,

[0102] in, represents the decoding features of the next decoding layer, Indicates the layer number, represents a series of decoded blocks, represents the decoded features of the current layer, represents a skip connection, Indicates the encoding features of the encoding layer corresponding to the current decoding layer;

[0103] The spatial feature restorer includes a guided feature attention module and a feature compensation module;

[0104] The specific structure of the guided feature attention module can be found in Figure 6 For the specific structure of the feature compensation module, please refer to Figure 7 ;

[0105] Except for the initial decoding layer, each decoding layer has a unique corresponding guided feature attention module;

[0106] Except for the last coding layer, each coding layer has a unique corresponding feature compensation module;

[0107] The guided feature attention module generates a decoding feature weight according to a gating mechanism, and the branch of the guided feature attention module for generating the decoding feature weight includes a two-dimensional convolutional layer, a LeakyRelu activation function layer, and a sigmoid activation function layer;

[0108] Multiplying the decoded features with the intermediate features of the spatial feature restorer according to the decoded feature weights to align the output image of the guided feature attention module and the guided image of the guided feature generator;

[0109] The specific algorithm of the guided feature attention module is as follows:

[0110]

[0111] in, represents the initial input features that guide the feature attention module, represents the guided feature attention module, represents a random tensor, represents the encoding features of the final encoding layer, Represents the output features of the guided feature attention module of the current layer, Represents the output features of the feature compensation module of the current layer, Represents the decoded features of the current layer;

[0112] The feature compensation module performs weighted fusion of the deep spatial features and the high-level decoding features, wherein the high-level decoding features are decoding features other than the decoding features of the initial decoding layer;

[0113] The specific algorithm of the feature compensation module is as follows:

[0114] ,

[0115] in, Represents the output features of the feature compensation module of the current layer, represents the feature compensation module, represents the output feature of the guided feature attention module of the previous layer, Indicates the encoding features of the encoding layer corresponding to the current decoding layer.

[0116] Step S04: capturing cloud component features in the cloud area image according to the cloud component network;

[0117] It should be noted that in this embodiment, cloud component features in the cloud area image are captured according to the cloud component network;

[0118] The cloud component network is a U-Net structure, and network parameters are optimized according to a self-supervision mechanism. The cloud component network includes an encoder and a decoder with jump connections, the encoder includes a convolution layer, a downsampling layer, and a nonlinear activation function layer, and the decoder includes a convolution layer, an upsampling layer, and a nonlinear activation function layer;

[0119] The specific algorithm of the cloud component network is as follows:

[0120] ,

[0121] in, represents the cloud component characteristics, represents the cloud component network, represents the cloud component network parameters, represents random noise.

[0122] Step S05: obtaining a final cloud-free image according to the image component features and the cloud component features;

[0123] It should be noted that in this embodiment, the final de-clouded image is obtained according to the image component features and the cloud component features. The specific algorithm for obtaining the final de-clouded image is as follows:

[0124] ,

[0125] ,

[0126] Among them, min means minimization function, represents the image component network parameters, represents the cloud component network parameters, represents a binary mask, represents the observed image contaminated by clouds, represents the Hadamard product, represents the image component network, represents a random tensor, represents the guide image, represents the cloud component network, represents random noise, represents the Frobenius norm, represents the nuclear norm, represents the 2,1 norm, and represents the regularization parameter, Represents a clean image.

[0127] In summary, according to the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method, a dual-network architecture cloud removal model is designed, the spatial range of the cloud component is constrained according to the initial model, and the cloud removal initial model adaptively generates a mask to accurately separate the clean area from the cloudy area according to the mask, and then the image component network and the cloud component network are used to capture the complex features of the image component and the cloud component, respectively, thereby further improving the accuracy and robustness of the separation. In addition, because of the separate capture of the cloud component, the problem of insufficient ability to capture clouds of different forms is effectively avoided. The image component network is constructed according to the guided feature generator and the spatial feature restorer, and the local detail features are effectively captured. At the same time, the destruction of spatial information is avoided, and the authenticity of the image after cloud removal is further improved. The present invention improves the accuracy of the remote sensing image thick cloud removal method. Specifically, remote sensing image data is acquired and preprocessed, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series, and a clean area image and a cloud area image in the remote sensing image data are separated according to an initial cloud removal model, wherein the initial cloud removal model adaptively generates a mask to divide the clean area and the cloud area according to the mask, and accurately separates the clean area from the cloud area, thereby improving the accuracy and effectiveness of overall cloud removal, and capturing image component features in the clean area image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer, and while retaining the spatial information of the image, the image detail information is enriched, thereby making the cloud-removed image more in line with the actual situation, and capturing cloud component features in the cloud area image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with jump connections, and cloud components are captured separately, thereby effectively avoiding the problem of insufficient ability to capture clouds of different forms, and obtaining a final cloud-removed image according to the image component features and the cloud component features. The present invention improves the accuracy of the thick cloud removal method for remote sensing images.

[0128] See also Figure 2 , which is a flow chart of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method proposed in the second embodiment of the present invention, the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method comprises steps S11 to S17, wherein:

[0129] Step S11: Acquire remote sensing image data and perform preprocessing;

[0130] Step S12: using a declouding initial model to separate clean area images and cloud area images in the remote sensing image data;

[0131] Step S13: inputting the clean area image into the image component network, and inputting the clean area image into the guided feature generator as the guided image. The guided feature generator generates multi-scale guided features according to the attention mechanism, and the spatial feature restorer restores the spatial detail information of the multi-scale guided features to obtain image component features.

[0132] Step S14: the guide feature generator performs convolution processing on the guide image according to the time series to obtain initial input features, the encoding block series encodes the initial input features according to the spatial attention mechanism to obtain multiple encoding features, and the decoding block series decodes the encoding features to obtain multiple decoding features;

[0133] Step S15: the guided feature attention module generates a decoding feature weight according to the gating mechanism, and multiplies and weights the decoding feature with the intermediate feature of the spatial feature restorer according to the decoding feature weight to align the output image of the guided feature attention module and the guided image of the guided feature generator, and the feature compensation module performs weighted fusion of the deep spatial feature and the high-level decoding feature;

[0134] Step S16: capturing cloud component features in the cloud area image according to the cloud component network;

[0135] Step S17: obtaining a final cloud-free image according to the image component features and the cloud component features;

[0136] It should be noted that in this embodiment, the present invention is compared with the remote sensing image thick cloud removal method in the prior art. The data set used in the experiment is the Farmland data set, the evaluation index 1 is the PSNR evaluation index, the evaluation index 2 is the SSIM, and the evaluation index 3 is the CC evaluation index. The comparison data results of the present invention and the remote sensing image thick cloud removal method in the prior art are as follows Table 1:

[0137] Table 1

[0138]

[0139] According to the data in Table 1 above, it can be seen that the method proposed in the present invention has achieved the best results in all indicators under different numbers of spectral bands. Compared with the existing traditional method Mosaicing, the method of the present invention has improved the PSNR evaluation index by more than 10dB, compared with the existing pure model method, it has improved by 7-8dB, and compared with the existing pure network method MT, it has improved by more than 5 dB. It can be seen that the present invention has made significant progress compared with the existing technology;

[0140] See also Figure 8 , Figure 8The following is a comparison chart of the experimental results. It can be seen that the two model-based methods TVLRSDC and RTCR both have very obvious blurring in the cloud-covered area, while the image restoration-based method Mosaicing is obtained by averaging the pixels in the clean phase area at the corresponding cloud position. Therefore, when the time interval between different time nodes is large or the surface difference changes greatly, the recovery result of the cloud-covered area will appear to be out of tune with the surrounding map. The visual effect of the pure network method MT has many details that cannot be restored, resulting in large-scale blurring. The STLR-DP and LRRSSN methods add model constraints on the basis of the network. However, some obvious artifacts still appear in the boundary area of ​​the cloud. In comparison, the method of the present invention shows a more ideal effect. It can not only restore the detail information better, but also has no obvious artifacts, which is a significant improvement.

[0141] In summary, according to the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method, a dual-network architecture cloud removal model is designed, the spatial range of the cloud component is constrained according to the initial model, and the cloud removal initial model adaptively generates a mask to accurately separate the clean area from the cloudy area according to the mask, and then the image component network and the cloud component network are used to capture the complex features of the image component and the cloud component, respectively, thereby further improving the accuracy and robustness of the separation. In addition, because of the separate capture of the cloud component, the problem of insufficient ability to capture clouds of different forms is effectively avoided. The image component network is constructed according to the guided feature generator and the spatial feature restorer, and the local detail features are effectively captured. At the same time, the destruction of spatial information is avoided, and the authenticity of the image after cloud removal is further improved. The present invention improves the accuracy of the remote sensing image thick cloud removal method. Specifically, remote sensing image data is acquired and preprocessed, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series, and a clean area image and a cloud area image in the remote sensing image data are separated according to an initial cloud removal model, wherein the initial cloud removal model adaptively generates a mask to divide the clean area and the cloud area according to the mask, and accurately separates the clean area from the cloud area, thereby improving the accuracy and effectiveness of overall cloud removal, and capturing image component features in the clean area image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer, and while retaining the spatial information of the image, the image detail information is enriched, thereby making the cloud-removed image more in line with the actual situation, and capturing cloud component features in the cloud area image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with jump connections, and cloud components are captured separately, thereby effectively avoiding the problem of insufficient ability to capture clouds of different forms, and obtaining a final cloud-removed image according to the image component features and the cloud component features. The present invention improves the accuracy of the thick cloud removal method for remote sensing images.

[0142] See also Figure 3 , which is a schematic diagram of the structure of a dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal system proposed in the third embodiment of the present invention, and the system includes:

[0143] A preprocessing module 10, used to acquire and preprocess remote sensing image data, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series;

[0144] An initial separation module 20 is used to separate the clean area image and the cloudy area image in the remote sensing image data according to the cloud removal initial model, and the cloud removal initial model adaptively generates a mask to divide the clean area and the cloudy area according to the mask;

[0145] An image component module 30, configured to capture image component features in the clean region image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer;

[0146] A cloud component module 40, configured to capture cloud component features in the cloud region image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with skip connections;

[0147] The result generating module 50 is used to obtain a final de-clouded image according to the image component features and the cloud component features.

[0148] The present invention also proposes a computer storage medium on which one or more programs are stored, which, when executed by a processor, implements the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method.

[0149] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method.

[0150] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0152] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method, characterized in that: include: Acquiring remote sensing image data and performing preprocessing, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series; Separating a clean area image and a cloudy area image in the remote sensing image data according to an initial cloud removal model, wherein the initial cloud removal model adaptively generates a mask to divide the clean area and the cloudy area according to the mask; capturing image component features in the clean region image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer; The step of capturing the image component features in the clean area image according to the image component network specifically includes: Inputting the clean region image into an image component network, wherein the image component network includes a guided feature generator and a spatial feature restorer; The clean area image is input as a guide image into the guide feature generator, and the guide feature generator generates multi-scale guide features according to an attention mechanism; The spatial feature restorer restores spatial detail information of the multi-scale guided features to obtain image component features; The specific algorithm of the image component feature is as follows: ， in, represents the image component features, represents the image component network, represents the image component network parameters, represents a random tensor, Indicates the guiding image; capturing cloud component features in the cloud region image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with skip connections; The step of capturing cloud component features in the cloud area image according to the cloud component network specifically includes: Capturing cloud component features in images of cloudy areas based on a cloud component network; The cloud component network is a U-Net structure, and network parameters are optimized according to a self-supervision mechanism. The cloud component network includes an encoder and a decoder with jump connections, the encoder includes a convolution layer, a downsampling layer, and a nonlinear activation function layer, and the decoder includes a convolution layer, an upsampling layer, and a nonlinear activation function layer; The specific algorithm of the cloud component network is as follows: ， in, represents the cloud component characteristics, represents the cloud component network, represents the cloud component network parameters, represents random noise; A final cloud-removed image is acquired according to the image component features and the cloud component features.

2. The dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method according to claim 1 is characterized in that: The step of separating the clean area image and the cloud area image in the remote sensing image data according to the cloud removal initial model specifically includes: The cloud removal initialization model separates clean area images and cloud area images in remote sensing image data; The specific algorithm of the cloud removal initial model is as follows: ， in, represents the observed image contaminated by clouds, represents a binary mask, represents a clean image, represents the cloud component, represents the Hadamard product.

3. The dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method according to claim 1 is characterized in that: The guiding feature generator specifically comprises: The guided feature generator is a series of encoding blocks and a series of decoding blocks with skip connections; The guided feature generator convolves the guided image according to the time series to obtain the initial input features; The encoding block series encodes the initial input features according to the spatial attention mechanism to obtain multiple encoding features. The specific algorithm of the encoding block series is as follows: ， in, represents the encoding features of the next encoding layer, represents the layer number, represents a series of coded blocks, Represents the encoding features of the current layer; The coding features are decoded according to the decoding block series to obtain multiple decoding features, each decoding layer in the decoding block series has a unique corresponding coding layer, the coding block series corresponds to the layers in the decoding block series in reverse order, and the last coding layer in the reverse order corresponds to the initial decoding layer. The specific algorithm of the decoding block series is as follows: ， in, represents the decoding features of the next decoding layer, represents the layer number, represents a series of decoded blocks, represents the decoded features of the current layer, represents a skip connection, Indicates the encoding features of the encoding layer corresponding to the current decoding layer.

4. The dual network driven self-supervised multi-temporal remote sensing image thick cloud removal method according to claim 1, characterized in that: The spatial feature restorer specifically comprises: The spatial feature restorer includes a guided feature attention module and a feature compensation module; Except for the initial decoding layer, each decoding layer has a unique corresponding guided feature attention module; Except for the last coding layer, each coding layer has a unique corresponding feature compensation module; The guided feature attention module generates a decoding feature weight according to a gating mechanism, and the branch of the guided feature attention module for generating the decoding feature weight includes a two-dimensional convolutional layer, a LeakyRelu activation function layer, and a sigmoid activation function layer; Multiplying the decoded features with the intermediate features of the spatial feature restorer according to the decoded feature weights to align the output image of the guided feature attention module and the guided image of the guided feature generator; The specific algorithm of the guided feature attention module is as follows: in, represents the initial input features that guide the feature attention module, represents the guided feature attention module, represents a random tensor, represents the encoding features of the final encoding layer, Represents the output features of the guided feature attention module of the current layer, Represents the output features of the feature compensation module of the current layer, Represents the decoded features of the current layer; The feature compensation module performs weighted fusion of the deep spatial features and the high-level decoding features, wherein the high-level decoding features are decoding features other than the decoding features of the initial decoding layer; The specific algorithm of the feature compensation module is as follows: ， in, Represents the output features of the feature compensation module of the current layer, represents the feature compensation module, represents the output feature of the guided feature attention module of the previous layer, Indicates the encoding features of the encoding layer corresponding to the current decoding layer.

5. The dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method according to claim 1, characterized in that: The step of obtaining the final cloud-removed image according to the image component features and the cloud component features specifically includes: The final de-clouded image is obtained according to the image component features and the cloud component features. The specific algorithm for obtaining the final de-clouded image is as follows: ， ， Among them, min means minimization function, represents the image component network parameters, represents the cloud component network parameters, represents a binary mask, represents the observed image contaminated by clouds, represents the Hadamard product, represents the image component network, represents a random tensor, represents the guide image, represents the cloud component network, represents random noise, represents the Frobenius norm, represents the nuclear norm, represents the 2,1 norm, and represents the regularization parameter, Represents a clean image.

6. A dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal system, characterized in that: include: A preprocessing module, used for acquiring remote sensing image data and performing preprocessing, wherein the remote sensing image data includes a plurality of remote sensing images arranged in a time series; An initial separation module, used for separating the clean area image and the cloudy area image in the remote sensing image data according to the cloud removal initial model, wherein the cloud removal initial model adaptively generates a mask to divide the clean area and the cloudy area according to the mask; An image component module, configured to capture image component features in the clean region image according to an image component network, wherein the image component network is constructed according to a guided feature generator and a spatial feature restorer; The step of capturing the image component features in the clean area image according to the image component network specifically includes: Inputting the clean region image into an image component network, wherein the image component network includes a guided feature generator and a spatial feature restorer; The clean area image is input as a guide image into the guide feature generator, and the guide feature generator generates multi-scale guide features according to an attention mechanism; The spatial feature restorer restores spatial detail information of the multi-scale guided features to obtain image component features; The specific algorithm of the image component feature is as follows: ， in, represents the image component features, represents the image component network, represents the image component network parameters, represents a random tensor, Indicates the guiding image; A cloud component module, configured to capture cloud component features in the cloud area image according to a cloud component network, wherein the cloud component network is constructed according to an encoder-decoder structure with skip connections; The step of capturing cloud component features in the cloud area image according to the cloud component network specifically includes: Capturing cloud component features in images of cloudy areas based on a cloud component network; The cloud component network is a U-Net structure, and network parameters are optimized according to a self-supervision mechanism. The cloud component network includes an encoder and a decoder with jump connections, the encoder includes a convolution layer, a downsampling layer, and a nonlinear activation function layer, and the decoder includes a convolution layer, an upsampling layer, and a nonlinear activation function layer; The specific algorithm of the cloud component network is as follows: ， in, represents the cloud component characteristics, represents the cloud component network, represents the cloud component network parameters, represents random noise; A result generation module is used to obtain a final de-clouded image according to the image component features and the cloud component features.

7. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by the processor, implement the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method as described in any one of claims 1 to 5.

8. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, it implements the dual-network driven self-supervised multi-temporal remote sensing image thick cloud removal method described in any one of claims 1-5.

Images