A remote sensing image cloud removal method based on image fusion feature learning and reconstruction

Abstract

The present invention relates to a remote sensing image cloud removal method based on image fusion feature learning and reconstruction, comprising: obtaining a cloud image, multiple reference images, and cloud and cloud shadow masks corresponding one-to-one to the cloud image and the reference images; determining a first reference image; replacing pixels of clouds and cloud shadows in the cloud image with pixels at corresponding positions in the first reference image to obtain a first fused image; cropping and classifying the first fused image to obtain a first image set and a second image set; processing the second image set to obtain a third image set; obtaining an image reconstruction data set based on the third image set and the second image set; training an image reconstruction model based on the image reconstruction data set to obtain a trained image reconstruction model; superimposing each cropped image in the first image set and its cloud and cloud shadow masks, and inputting the superimposed images into the trained image reconstruction model to obtain a predicted image set; and obtaining a final cloud-removed image based on the second image set and the predicted image set.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image cloud removal method based on image fusion feature learning and reconstruction. Background Art

[0002] Remote sensing imagery is highly intuitive and contains rich surface information, making it widely used in disaster prevention and mitigation, resource surveys, agricultural surveys, and military applications. However, due to limitations in their imaging mechanisms, remote sensing images are susceptible to cloud interference, severely limiting their use. Clouds have complex effects on remote sensing images: thin clouds weaken surface radiation, causing image distortion and blurring; thick clouds obscure surface information and cast shadows on the surface.

[0003] Existing research on cloud removal from remote sensing images still focuses on a single type of thin clouds or thick clouds. It is impossible to remove thin clouds, thick clouds and cloud shadows in remote sensing images at the same time. In addition, the original surface information characteristics are difficult to accurately restore. There are large color differences before and after removal and poor overall consistency. Summary of the Invention

[0004] (1) Technical issues to be resolved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a remote sensing image cloud removal method based on image fusion feature learning and reconstruction, which solves the technical problems that the existing research on remote sensing image cloud removal is still focused on a single type of thin clouds or thick clouds, and it is impossible to remove thin clouds, thick clouds and cloud shadows in remote sensing images at the same time. In addition, the original surface information features are difficult to accurately restore, and there are large color differences before and after removal and poor overall consistency.

[0006] (2) Technical solution

[0007] In order to achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] The embodiment of the present invention provides a remote sensing image cloud removal method based on image fusion feature learning and reconstruction, comprising:

[0009] S1. Obtaining a cloud image and multiple reference images taken by a remote sensing satellite for a designated area; and cloud and cloud shadow masks corresponding to the cloud image and the multiple reference images, respectively; the cloud image is an image taken by a remote sensing satellite for the designated area at a first moment, and clouds are displayed in the cloud image; the reference image is an image taken by a remote sensing satellite for the designated area at a second moment, and the second moment is different from the first moment;

[0010] S2. determining a first reference image from the plurality of reference images based on cloud and cloud shadow masks corresponding one-to-one to the cloud image and the plurality of reference images;

[0011] S3, replacing the pixels of clouds and cloud shadows in the cloud image with pixels at positions corresponding to the pixels of clouds and cloud shadows in the first reference image, to obtain a first fused image;

[0012] S4. Cropping the first fused image to obtain multiple cropped images, and classifying the cropped images to obtain a first image set and a second image set;

[0013] S5. Process the second image set to obtain a third image set;

[0014] S6. Acquire an image reconstruction dataset based on the third image set and the second image set;

[0015] S7. Training a pre-acquired image reconstruction model based on the image reconstruction dataset to obtain a trained image reconstruction model;

[0016] S8. Superimpose each cropped image in the first image set and its cloud and cloud shadow masks, and input the superimposed images into the trained image reconstruction model to obtain a predicted image set;

[0017] S9. Obtain a final cloud-removed image corresponding to the cloud-containing image based on the second image set and the predicted image set.

[0018] Preferably,

[0019] The first reference image is a reference image with the smallest cloud ratio among the reference images satisfying formula (1);

[0020] The cloud ratio corresponding to the reference image is the ratio of the number of clouds and cloud image pixels in the reference image to the number of all pixels in the reference image.

[0021] The formula (1) is:

[0022]

[0023] P cloud Represents the set of pixel locations of clouds and cloud shadows in a cloud image;

[0024] P Reference A collection of pixel locations representing clouds and cloud shadows in the reference image.

[0025] Preferably, the S4 specifically includes:

[0026] S41: Cropping the first fused image to obtain multiple cropped images;

[0027] The size of each cropped image is 256*256;

[0028] S42: Determine whether each cropped image contains a pixel in the first reference image, and obtain a determination result;

[0029] S43. Classify the cropped images based on the judgment result corresponding to each cropped image to obtain a first image set and a second image set;

[0030] The first image set includes cropped images that are determined to contain pixels in the first reference image;

[0031] The second image set includes cropped images that are determined not to contain pixels in the first reference image.

[0032] Preferably, the S5 specifically includes:

[0033] S51, obtaining image regions corresponding to each cropped image in the second image set in the first reference image;

[0034] S52, obtaining the cloud ratio in each image area;

[0035] The cloud ratio in the image area is the ratio of the number of clouds and cloud image pixels in the image area to the number of all image pixels in the image area;

[0036] S53: Based on the cloud ratio in the image area, the cropped images in the second image set corresponding to the image area are processed according to a preset strategy to obtain a third image set;

[0037] The preset strategies include:

[0038] If the cloud ratio in the image area is greater than 0% and less than 100%, the pixels in the non-cloud area of ​​the image area will be used to replace the pixels in the corresponding positions in the corresponding cropped image to obtain the corresponding replaced cropped image;

[0039] The non-cloud region in the image region is obtained based on the cloud and cloud shadow mask of the first reference image;

[0040] If the cloud ratio in the image area is equal to 0, a cloud and cloud shadow mask image is randomly selected from the cloud area mask set, and the pixels with cloud areas in the cloud and cloud shadow mask image are used to replace the pixels at the corresponding positions in the cropped image corresponding to the image area to obtain the corresponding replaced cropped image;

[0041] The cloud region mask set is a plurality of cloud and cloud shadow mask images obtained by cropping the cloud and cloud shadow masks of the cloud image according to a size of 256*256.

[0042] Preferably, the image reconstruction dataset in S6 includes multiple pairs of images;

[0043] Each pair of images includes a replaced cropped image in the third image set and a corresponding cropped image in the second image set.

[0044] Preferably, the S7 specifically includes:

[0045] S71. Based on the image reconstruction dataset, obtain a fusion mask image corresponding to the replaced cropped image in the third image set of each pair of images in the image reconstruction dataset;

[0046] When the replaced cropped image is obtained by randomly selecting a cloud and cloud shadow mask image from the cloud region mask set and replacing pixels at corresponding positions in the cropped image corresponding to the image region with pixels having cloud regions in the cloud and cloud shadow mask image, the fused mask image corresponding to the replaced cropped image is the cloud and cloud shadow mask image;

[0047] When the replaced cropped image is obtained by replacing pixels at corresponding positions in the corresponding cropped image with pixels in a non-cloud area in the image area, the fused mask image corresponding to the replaced cropped image is the non-cloud area in the image area;

[0048] S72, using each replaced cropped image in the third image set and the fused mask image corresponding to the replaced cropped image as input images of the pre-acquired image reconstruction model, and using the cropped image in the second image set corresponding to the replaced cropped image as a target image of the pre-acquired image reconstruction model, and inputting the images into the pre-acquired image reconstruction model; the pre-acquired image reconstruction model is trained based on the input images and the target images to obtain a trained image reconstruction model;

[0049] The predicted image set includes predicted images obtained by predicting each cropped image in the first image set by the trained image reconstruction model.

[0050] Preferably,

[0051] All cropped images in the second image set and all predicted images in the predicted image set are spliced ​​together to obtain a final cloud-removed image corresponding to the cloud-containing image.

[0052] Preferably,

[0053] The image reconstruction model includes a generator and a discriminator;

[0054] Wherein, the generator includes a plurality of feature extraction modules;

[0055] The feature extraction module is obtained by removing the layer normalization structure on the basis of the ConvNeXt network feature extraction module, replacing the GELU activation function with the Leaky RELU activation function, and removing the regularization structure.

[0056] Preferably,

[0057] The discriminator in the image reconstruction model is a hierarchical Markov discriminator;

[0058] Each level in the hierarchical Markov discriminator is provided with an output for determining whether the input image is real or fake;

[0059] The hierarchical Markov discriminator discriminates the similarity between the input image and the target image through the outputs corresponding to all levels during the training process, and determines the total loss value of the hierarchical Markov discriminator.

[0060] Preferably,

[0061] The hierarchical Markov discriminator uses formula (2) to determine the total loss value of the hierarchical Markov discriminator;

[0062] The formula (2) is

[0063]

[0064] loss d is the total loss value of the hierarchical Markov discriminator;

[0065] i represents the i-th level of the hierarchical Markov discriminator; i = [1, 2, 3, 4];

[0066] loss outputi is the loss value output by the i-th layer;

[0067] λ i is the weight of the pre-set output loss value of the i-th layer.

[0068] (3) Beneficial effects

[0069] The beneficial effects of the present invention are as follows: a remote sensing image cloud removal method based on image fusion feature learning and reconstruction of the present invention replaces the pixels of clouds and cloud shadows in the cloud image with pixels at positions corresponding to the pixels of the clouds and cloud shadows in the first reference image to obtain a first fused image, then the first fused image is cropped to obtain multiple cropped images, and the cropped images are classified to obtain a first image set and a second image set, and then the second image set is processed to obtain a third image set, and then, based on the third image set and the second image set, an image reconstruction data set is obtained; based on the image reconstruction data set, a pre-acquired image reconstruction model is trained to obtain a trained image reconstruction model; compared with the prior art, the trained image reconstruction model in the present invention is obtained by training the image reconstruction data set, and therefore can not only learn the conversion features of the image from a cloud-free image to a cloud-free area of ​​a cloud-containing image, but also learn the spatial distribution features of clouds in the cloud-containing image, so that the original surface information of the cloud and cloud shadow area can be accurately restored, the original features of the image can be restored, and the overall consistency of the cloud removal result can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to the present invention;

[0071] Figure 2 Schematic diagram of the structure of the generator of the image reconstruction model of the present invention;

[0072] Figure 3 Schematic diagram of the structure of the discriminator of the image reconstruction model of the present invention;

[0073] Figure 4 is a schematic diagram of a cloud image in this embodiment;

[0074] Figure 5 Schematic diagram of the final cloud-removed image corresponding to the cloud-containing image in this embodiment. DETAILED DESCRIPTION

[0075] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0076] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0077] See also Figure 1This embodiment provides a remote sensing image cloud removal method based on image fusion feature learning and reconstruction, including:

[0078] S1. Obtain a cloud image and multiple reference images taken by a remote sensing satellite for a designated area; and cloud and cloud shadow masks corresponding to the cloud image and the multiple reference images, respectively; the cloud image is an image taken by a remote sensing satellite for the designated area at a first moment, and clouds are displayed in the cloud image; the reference image is an image taken by a remote sensing satellite for the designated area at a second moment; and the second moment is different from the first moment.

[0079] In practical applications, cloud cover causes the loss of surface information, and the lost information cannot be generated out of thin air. Homologous remote sensing images are a reliable and easy-to-use source of supplementary information.

[0080] S2. Determine a first reference image from the plurality of reference images according to the cloud and cloud shadow masks corresponding one-to-one to the cloud image and the plurality of reference images.

[0081] In this embodiment, the reference image has no imaging time constraints. Any image taken by a remote sensing satellite for a specified area at any time can be selected as a reference image. This is because the cloud removal method proposed in this embodiment can effectively solve the problem of changes in imaging targets and imaging environments caused by different imaging times, weakening the selection conditions of the reference image and improving the applicability of the method.

[0082] In this embodiment, the first reference image is a reference image having the smallest cloud ratio among the reference images satisfying formula (1).

[0083] The cloud ratio corresponding to the reference image is the ratio of the number of clouds and cloud image elements in the reference image to the number of all pixels in the reference image.

[0084] The formula (1) is:

[0085]

[0086] P cloud A collection of pixel locations representing clouds and cloud shadows in a cloud image.

[0087] P Reference A collection of pixel locations representing clouds and cloud shadows in the reference image.

[0088] That is, the determined first reference image does not overlap with the clouds and cloud image elements of the cloud image. In this embodiment, Sentinel-2 images (Sentinel-2 is a satellite launched by the European Space Agency (ESA) and is primarily used for Earth observation and monitoring. The satellite was successfully launched on June 23, 2015, and is primarily used to collect surface information and provide high-resolution image data) and their cloud and cloud shadow masks are selected as experimental data.

[0089] S3. Replace the pixels of the cloud and cloud shadow in the cloud image with pixels at positions corresponding to the pixels of the cloud and cloud shadow in the first reference image to obtain a first fused image.

[0090] In this embodiment, the surface information lost in the cloud image is preliminarily restored through image fusion.

[0091] S4. Crop the first fused image to obtain multiple cropped images, and classify the cropped images to obtain a first image set and a second image set.

[0092] In the practical application of this embodiment, the S4 specifically includes:

[0093] S41: Crop the first fused image to obtain multiple cropped images.

[0094] The size of each cropped image is 256*256 (here refers to the number of pixels).

[0095] S42: Determine whether each cropped image contains a pixel in the first reference image, and obtain a determination result.

[0096] S43 . Classify the cropped images based on the judgment result corresponding to each cropped image to obtain a first image set and a second image set.

[0097] The first image set includes cropped images that are judged to contain pixels in the first reference image.

[0098] The second image set includes cropped images that are determined not to contain pixels in the first reference image.

[0099] S5. Process the second image set to obtain a third image set.

[0100] Wherein, the S5 specifically includes:

[0101] S51 , obtaining image regions corresponding to each cropped image in the second image set in the first reference image.

[0102] S52: Obtain the cloud ratio in each image area.

[0103] The cloud ratio in the image region is the ratio of the number of clouds and cloud image pixels in the image region to the number of all image pixels in the image region.

[0104] S53 : Based on the cloud ratio in the image area, the cropped images corresponding to the image area in the second image set are processed according to a preset strategy to obtain a third image set.

[0105] The preset strategies include:

[0106] If the cloud ratio in the image area is greater than 0% and less than 100%, the pixels in the non-cloud area of ​​the image area will be used to replace the pixels at the corresponding positions in the corresponding cropped image to obtain the corresponding replaced cropped image.

[0107] The non-cloud region in the image region is obtained based on the cloud and cloud shadow mask of the first reference image.

[0108] If the cloud ratio in the image area is equal to 0, a cloud and cloud shadow mask image is randomly selected from the cloud area mask set, and the pixels with cloud areas in the cloud and cloud shadow mask image are used to replace the pixels at the corresponding positions in the cropped image corresponding to the image area to obtain the corresponding replaced cropped image.

[0109] The cloud region mask set is a plurality of cloud and cloud shadow mask images obtained by cropping the cloud and cloud shadow masks of the cloud image according to a size of 256*256.

[0110] In this embodiment, if the cloud ratio in the image area is equal to 0, a cloud and cloud shadow mask image is randomly selected from the cloud area mask set. The purpose is to construct a third image set that can not only learn the image conversion characteristics from a cloud-free image to a cloud-free area of ​​a cloud image, but also learn the spatial distribution characteristics of clouds in the cloud image.

[0111] In this embodiment, two different strategies are used to process the different cloud and cloud shadow contents in the corresponding image region of the first reference image. Specifically, when the cloud content in the image region is greater than 0% and less than 100%, pixels in the non-cloud region of the image region are used to replace pixels in the corresponding position in the corresponding cropped image. Furthermore, when the cloud content in the image region is equal to 0, a cloud and cloud shadow mask image is randomly selected from the cloud region mask set, and pixels in the cloud region of the cloud and cloud shadow mask image are used to replace pixels in the corresponding position in the cropped image corresponding to the image region, thereby obtaining the corresponding replaced cropped image. This facilitates learning the distribution characteristics of clouds and cloud shadows in images with clouds, while also fully utilizing the information of cloud-free areas in images with clouds.

[0112] S6. Acquire an image reconstruction dataset based on the third image set and the second image set.

[0113] The image reconstruction dataset in S6 includes multiple pairs of images.

[0114] Each pair of images includes a replaced cropped image in the third image set and a corresponding cropped image in the second image set.

[0115] S7. Based on the image reconstruction dataset, the pre-acquired image reconstruction model is trained to obtain a trained image reconstruction model. In this embodiment, the trained image reconstruction model enables the reconstructed image to maintain overall consistency, restore the original features of the cloud image, and retain clearer detailed texture information.

[0116] The S7 specifically includes:

[0117] S71. Based on the image reconstruction dataset, obtain a fusion mask image corresponding to the replaced cropped image in the third image set of each pair of images in the image reconstruction dataset.

[0118] Among them, when the replaced cropped image is obtained by randomly selecting a cloud and cloud shadow mask image from the cloud area mask set, and using the pixels with cloud areas in the cloud and cloud shadow mask image to replace the pixels at corresponding positions in the cropped image corresponding to the image area, the fused mask image corresponding to the replaced cropped image is the cloud and cloud shadow mask image.

[0119] When the replaced cropped image is obtained by replacing the pixels at corresponding positions in the corresponding cropped image with the pixels in the non-cloud area of ​​the image area, the fused mask image corresponding to the replaced cropped image is the non-cloud area in the image area.

[0120] S72. Each replaced cropped image in the third image set and the fused mask image corresponding to the replaced cropped image are used as input images of the pre-acquired image reconstruction model, and the cropped image in the second image set corresponding to the replaced cropped image is used as the target image of the pre-acquired image reconstruction model, and are input into the pre-acquired image reconstruction model. The pre-acquired image reconstruction model is trained based on the input image and the target image to obtain a trained image reconstruction model.

[0121] The predicted image set includes predicted images obtained by predicting each cropped image in the first image set by the trained image reconstruction model.

[0122] The image reconstruction model is a designed multi-scale feature fusion image reconstruction generates adversarial network (MSFFIR-GAN) including a generator and a discriminator.

[0123] The generator uses multiple feature extraction modules and upsampling feature fusion modules to gradually extract and integrate feature information at different depths and scales to enhance the model's contextual awareness and robustness. The feature extraction module is based on the ConvNeXt network feature extraction module, removing the layer normalization structure, replacing the GELU activation function with the LeakyRELU activation function, and removing the regularization structure.

[0124] The discriminator uses a designed hierarchical Markov discriminator, which measures the similarity between the reconnected image and the target image at different depths through four different levels of output, helping to make the generated image have clearer detail information.

[0125] Wherein, the generator includes multiple feature extraction modules.

[0126] The feature extraction module is obtained by removing the layer normalization structure on the basis of the ConvNeXt network feature extraction module, replacing the GELU activation function with the Leaky RELU activation function, and removing the regularization structure.

[0127] In this embodiment, the image reconstruction model is actually a multi-scale feature fusion image reconstruction generative adversarial network (MSFFIR-GAN). Figure 2 As shown, the generator and discriminator of the image reconstruction model.

[0128] See also Figure 3 , the discriminator in the image reconstruction model is a hierarchical Markov discriminator.

[0129] Each level in the hierarchical Markov discriminator is provided with an output for determining whether the input image is true or false.

[0130] The hierarchical Markov discriminator discriminates the similarity between the input image and the target image through the outputs corresponding to all levels during the training process, and determines the total loss value of the hierarchical Markov discriminator.

[0131] The hierarchical Markov discriminator uses formula (2) to determine the total loss value of the hierarchical Markov discriminator.

[0132] The formula (2) is:

[0133]

[0134] loss d is the total loss value of the hierarchical Markov discriminator.

[0135] i represents the i-th level of the hierarchical Markov discriminator; i=[1, 2, 3, 4].

[0136] loss outputi is the loss value output by the i-th layer.

[0137] λ i is the weight of the pre-set output loss value of the i-th layer.

[0138] S8. After superimposing each cropped image and its cloud and cloud shadow masks in the first image set, input the superimposed images into the trained image reconstruction model to obtain a predicted image set.

[0139] S9. Obtain a final cloud-removed image corresponding to the cloud-containing image based on the second image set and the predicted image set.

[0140] In this embodiment, all the cropped images in the second image set and all the predicted images in the predicted image set are spliced ​​together to obtain the cloud image (e.g. Figure 4 The final cloud removal image corresponding to Figure 5 .

[0141] In this embodiment, a remote sensing image cloud removal method based on image fusion feature learning and reconstruction is provided. Cloud and cloud shadow pixels in a cloud image are replaced with pixels in a first reference image corresponding to the cloud and cloud shadow pixels to obtain a first fused image. The first fused image is then cropped to obtain multiple cropped images, which are then classified to obtain a first image set and a second image set. The second image set is then processed to obtain a third image set. An image reconstruction dataset is then obtained based on the third image set and the second image set. A pre-acquired image reconstruction model is trained based on the image reconstruction dataset to obtain a trained image reconstruction model. Compared to the prior art, the trained image reconstruction model in the present invention, because it is trained using the image reconstruction dataset, can not only learn the transformation features of an image from a cloud-free image to a cloud-free area in a cloud image, but also learn the spatial distribution features of clouds in a cloud image. Therefore, the original surface information of the cloud and cloud shadow areas can be accurately restored, the original features of the image can be restored, and the overall consistency of the cloud removal results can be ensured.

[0142] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0144] It should be noted that, in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims enumerating several means, several of these means may be embodied by one and the same hardware. The use of the words first, second, third etc. is for convenience only and does not indicate any order. These words may be understood as part of the component name.

[0145] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0146] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0147] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention shall also include such modifications and variations.

Claims

1. A remote sensing image cloud removal method based on image fusion feature learning and reconstruction, characterized in that: include: S1. Obtain cloud images and multiple reference images taken by remote sensing satellites for a designated area; and cloud and cloud shadow masks corresponding one-to-one to the cloud image and multiple reference images, respectively; the cloud image is an image taken by a remote sensing satellite at a first moment for a designated area, and clouds are displayed in the cloud image; the reference image is an image taken by a remote sensing satellite at a second moment for the designated area, and the second moment is different from the first moment; S2. determining a first reference image from the plurality of reference images based on cloud and cloud shadow masks corresponding one-to-one to the cloud image and the plurality of reference images; S3, replacing the pixels of clouds and cloud shadows in the cloud image with pixels at positions corresponding to the pixels of clouds and cloud shadows in the first reference image, to obtain a first fused image; S4. Cropping the first fused image to obtain multiple cropped images, and classifying the cropped images to obtain a first image set and a second image set; S5. Process the second image set to obtain a third image set; S6. Acquire an image reconstruction dataset based on the third image set and the second image set; S7. Training a pre-acquired image reconstruction model based on the image reconstruction dataset to obtain a trained image reconstruction model; S8. Superimpose each cropped image in the first image set and its cloud and cloud shadow masks, and input the superimposed images into the trained image reconstruction model to obtain a predicted image set; S9. Obtain a final cloud-removed image corresponding to the cloud-containing image based on the second image set and the predicted image set.

2. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 1 is characterized in that: The first reference image is a reference image with the smallest cloud ratio among the reference images satisfying formula (1); The cloud ratio corresponding to the reference image is the ratio of the number of clouds and cloud image pixels in the reference image to the number of all pixels in the reference image. The formula (1) is: P cloud Represents the set of pixel locations of clouds and cloud shadows in a cloud image; P Reference A collection of pixel locations representing clouds and cloud shadows in the reference image.

3. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 2 is characterized in that: The S4 specifically includes: S41: Cropping the first fused image to obtain multiple cropped images; The size of each cropped image is 256*256; S42: Determine whether each cropped image contains a pixel in the first reference image, and obtain a determination result; S43. Classify the cropped images based on the judgment result corresponding to each cropped image to obtain a first image set and a second image set; The first image set includes cropped images that are determined to contain pixels in the first reference image; The second image set includes cropped images that are determined not to contain pixels in the first reference image.

4. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 3 is characterized in that: The S5 specifically includes: S51, obtaining image regions corresponding to each cropped image in the second image set in the first reference image; S52, obtaining the cloud ratio in each image area; The cloud ratio in the image area is the ratio of the number of clouds and cloud image pixels in the image area to the number of all image pixels in the image area; S53: Based on the cloud ratio in the image area, the cropped images in the second image set corresponding to the image area are processed according to a preset strategy to obtain a third image set; The preset strategies include: If the cloud ratio in the image area is greater than 0% and less than 100%, the pixels in the non-cloud area of ​​the image area will be used to replace the pixels in the corresponding positions in the corresponding cropped image to obtain the corresponding replaced cropped image; The non-cloud region in the image region is obtained based on the cloud and cloud shadow mask of the first reference image; If the cloud ratio in the image area is equal to 0, a cloud and cloud shadow mask image is randomly selected from the cloud area mask set, and the pixels with cloud areas in the cloud and cloud shadow mask image are used to replace the pixels at the corresponding positions in the cropped image corresponding to the image area to obtain the corresponding replaced cropped image; The cloud region mask set is a plurality of cloud and cloud shadow mask images obtained by cropping the cloud and cloud shadow masks of the cloud image according to a size of 256*256.

5. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 4 is characterized in that: The image reconstruction dataset in S6 includes multiple pairs of images; Each pair of images includes a replaced cropped image in the third image set and a corresponding cropped image in the second image set.

6. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 5 is characterized in that: The S7 specifically includes: S71. Based on the image reconstruction dataset, obtain a fusion mask image corresponding to the replaced cropped image in the third image set of each pair of images in the image reconstruction dataset; When the replaced cropped image is obtained by randomly selecting a cloud and cloud shadow mask image from the cloud region mask set and replacing pixels at corresponding positions in the cropped image corresponding to the image region with pixels having cloud regions in the cloud and cloud shadow mask image, the fused mask image corresponding to the replaced cropped image is the cloud and cloud shadow mask image; When the replaced cropped image is obtained by replacing pixels at corresponding positions in the corresponding cropped image with pixels in a non-cloud area in the image area, the fused mask image corresponding to the replaced cropped image is the non-cloud area in the image area; S72, using each replaced cropped image in the third image set and the fused mask image corresponding to the replaced cropped image as input images of the pre-acquired image reconstruction model, and using the cropped image in the second image set corresponding to the replaced cropped image as a target image of the pre-acquired image reconstruction model, and inputting the images into the pre-acquired image reconstruction model; the pre-acquired image reconstruction model is trained based on the input images and the target images to obtain a trained image reconstruction model; The predicted image set includes predicted images obtained by predicting each cropped image in the first image set by the trained image reconstruction model.

7. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 6 is characterized in that: All cropped images in the second image set and all predicted images in the predicted image set are spliced ​​together to obtain a final cloud-removed image corresponding to the cloud-containing image.

8. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 7 is characterized in that: The image reconstruction model includes a generator and a discriminator; Wherein, the generator includes a plurality of feature extraction modules; The feature extraction module is obtained by removing the layer normalization structure on the basis of the ConvNeXt network feature extraction module, replacing the GELU activation function with the Leaky RELU activation function, and removing the regularization structure.

9. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 8 is characterized in that: The discriminator in the image reconstruction model is a hierarchical Markov discriminator; Each level in the hierarchical Markov discriminator is provided with an output for determining whether the input image is real or fake; The hierarchical Markov discriminator discriminates the similarity between the input image and the target image through the outputs corresponding to all levels during the training process, and determines the total loss value of the hierarchical Markov discriminator.

10. The remote sensing image cloud removal method based on image fusion feature learning and reconstruction according to claim 9, characterized in that: The hierarchical Markov discriminator uses formula (2) to determine the total loss value of the hierarchical Markov discriminator; The formula (2) is loss d is the total loss value of the hierarchical Markov discriminator; i represents the i-th level of the hierarchical Markov discriminator; i = [1, 2, 3, 4]; loss outputi is the loss value output by the i-th layer; λ i is the weight of the pre-set output loss value of the i-th layer.

Images