Sample image generation method, training method and device of cloud removal model

Abstract

This disclosure provides a sample image generation method, a cloud removal model training method, and an apparatus, relating to the technical fields of image processing and deep learning. The specific implementation includes: acquiring an original image and a thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds; extracting basic cloud component data from the thin cloud image; for each band of the original image, correcting the basic cloud component data according to the spectral correction factor corresponding to that band and the compensation coefficient for the thin cloud image, generating cloud radiative component data corresponding to that band; for each band of the original image, generating a single-band clouded image corresponding to that band based on the single-band original image corresponding to that band, the cloud radiative component data corresponding to that band, and a preset atmospheric light value; and combining the single-band clouded images corresponding to each band to obtain a sample image corresponding to the original image.

Description

Technical Field

[0001] This disclosure relates to the technical fields of image processing and deep learning, and in particular to a sample image generation method and apparatus, a cloud removal model training method and apparatus, a remote sensing image processing method, apparatus, electronic device, storage medium and computer program product. Background Technology

[0002] In the field of remote sensing, deep learning-based methods for thin cloud restoration in remote sensing images typically rely on a large number of high-quality thin cloud sample images. Therefore, acquiring thin cloud sample images is a crucial foundation for the research and application of thin cloud restoration technology. However, due to the randomness and time-sensitivity of thin cloud distribution, and limitations imposed by actual acquisition conditions, it is often difficult to obtain thin cloud sample images that perfectly match a specific ground feature scene. Therefore, constructing high-quality thin cloud sample images is of paramount importance. Summary of the Invention

[0003] This disclosure provides a sample image generation method and apparatus, a cloud removal model training method and apparatus, a remote sensing image processing method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of this disclosure, a method for generating a sample image is provided, comprising: acquiring an original image and a thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds; extracting cloud basic component data from the thin cloud image; for each band of the original image, correcting the cloud basic component data according to the spectral correction factor corresponding to that band and the compensation coefficient for the thin cloud image, thereby generating cloud radiative component data corresponding to that band; for each band of the original image, generating a single-band clouded image corresponding to that band according to the single-band original image corresponding to that band, the cloud radiative component data corresponding to that band, and a preset atmospheric light value; and combining the single-band clouded images corresponding to each band to obtain a sample image corresponding to the original image, wherein the sample image is an optical remote sensing image containing thin clouds.

[0005] According to another aspect of this disclosure, a method for training a cloud removal model is provided, comprising: acquiring a sample image, the sample image being obtained according to the sample image generation method described above; performing cloud removal processing on the sample image using an initial cloud removal model to obtain a repaired image; and adjusting the parameters of the initial cloud removal model based on the repaired image and the original image corresponding to the sample image to obtain a cloud removal model, the cloud removal model being used to perform thin cloud removal processing on the image to be processed.

[0006] According to another aspect of this disclosure, a remote sensing image processing method is provided, comprising: acquiring an image to be processed, wherein the image to be processed is an optical remote sensing image containing a thin cloud region; segmenting the image to be processed into a plurality of first image blocks, each first image block having a first resolution; performing cloud removal processing on the plurality of first image blocks respectively using a cloud removal model to obtain a plurality of first restored image blocks, and obtaining an initial restored image based on the plurality of first restored image blocks, wherein the cloud removal model is trained according to the cloud removal model training method described above; segmenting the initial restored image into a plurality of second image blocks, each second image block having a second resolution, and the second segmentation... The resolution differs from the first resolution; multiple second image patches are resampled to obtain multiple resampled second image patches, each with the first resolution; cloud removal is performed on the multiple resampled second image patches using a cloud removal model to obtain multiple second restored image patches, and a second restored image is obtained based on the multiple second restored image patches; in response to determining that the second restored image does not contain thin cloud regions, the second restored image is used as a reference restored image, and with the reference restored image as a reference, the initial restored image is color-corrected based on a wavelet transform algorithm to obtain the target restored image corresponding to the image to be processed.

[0007] According to another aspect of this disclosure, a sample image generation apparatus is provided, comprising: a first acquisition module for acquiring an original image and a thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds; an extraction module for extracting cloud basic component data from the thin cloud image; a first generation module for correcting the cloud basic component data for each band of the original image according to the spectral correction factor corresponding to that band and the compensation coefficient for the thin cloud image, thereby generating cloud radiation component data corresponding to that band; a second generation module for generating a single-band clouded image for each band of the original image according to the single-band original image corresponding to that band, the cloud radiation component data corresponding to that band, and a preset atmospheric light value; and a combination module for combining the single-band clouded images corresponding to each band to obtain a sample image corresponding to the original image, wherein the sample image is an optical remote sensing image containing thin clouds.

[0008] According to another aspect of this disclosure, a training apparatus for a cloud removal model is provided, comprising: a second acquisition module for acquiring sample images, the sample images being obtained from the sample image generation apparatus described above; a processing module for performing cloud removal processing on the sample images using an initial cloud removal model to obtain a restored image; and a parameter tuning module for tuning the initial cloud removal model based on the restored image and the original image corresponding to the sample image to obtain a cloud removal model, the cloud removal model being used to perform thin cloud removal processing on the image to be processed.

[0009] According to another aspect of this disclosure, a remote sensing image processing method is provided, comprising: a third acquisition module for acquiring an image to be processed, wherein the image to be processed is an optical remote sensing image containing a thin cloud region; a first segmentation module for segmenting the image to be processed into a plurality of first image blocks, each first image block having a first resolution; a first restoration module for performing cloud removal processing on the plurality of first image blocks respectively using a cloud removal model to obtain a plurality of first restored image blocks, and obtaining an initial restored image based on the plurality of first restored image blocks; wherein the cloud removal model is trained using a training device for the cloud removal model described above; and a second segmentation module for segmenting the initial restored image into a plurality of second image blocks, each second image block having a second resolution, and The second resolution is different from the first resolution; the resampling module is used to resample multiple second image blocks respectively to obtain multiple resampled second image blocks, each of the resampled second image blocks having a first resolution; the second restoration module is used to perform cloud removal processing on multiple resampled second image blocks respectively using the cloud removal model to obtain multiple second restored image blocks, and obtain a second restored image based on the multiple second restored image blocks; the correction module is used to, in response to determining that the second restored image does not contain thin cloud regions, use the second restored image as a reference restored image, and use the reference restored image as a reference to perform color correction processing on the initial restored image based on the wavelet transform algorithm to obtain a target restored image corresponding to the image to be processed.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the sample image generation method, the cloud removal model training method, and the remote sensing image processing method as described above.

[0011] According to another aspect of this disclosure, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the sample image generation method, the cloud removal model training method, and the remote sensing image processing method as described above.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the sample image generation method, the cloud removal model training method, and the remote sensing image processing method as described above. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0014] Figure 1 This is a flowchart of a sample image generation method according to an embodiment of the present disclosure;

[0015] Figure 2 This is a flowchart of a training method for a cloud-free model according to an embodiment of the present disclosure;

[0016] Figure 3A The schematic diagram illustrates the structure of the initial cloud removal model;

[0017] Figure 3B The schematic diagram illustrates the structure of each feature-refined feedforward submodule;

[0018] Figure 3C The schematic diagram illustrates the structure of the attention submodule;

[0019] Figure 4 This is a flowchart of a remote sensing image processing method according to an embodiment of the present disclosure;

[0020] Figure 5 This is a block diagram of a sample image generation apparatus according to an embodiment of the present disclosure;

[0021] Figure 6 This is a block diagram of a training apparatus for a cloud-free model according to an embodiment of the present disclosure;

[0022] Figure 7 This is a block diagram of a remote sensing image processing apparatus according to embodiments of the present disclosure;

[0023] Figure 8 This is a block diagram of an electronic device suitable for implementing a sample image generation method, a cloud removal model training method, and a remote sensing image processing method according to embodiments of the present disclosure. Detailed Implementation

[0024] The technical solutions of the present disclosure will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure are within the scope of protection of the present disclosure.

[0025] It should be noted that the sequence numbers of each operation in the following methods are for descriptive purposes only and should not be construed as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed exactly in the order shown. Furthermore, in the description of this disclosure, the terms "first," "second," etc. (if present) are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0026] The technical solutions disclosed herein involve data collection, storage, use, processing, transmission, provision, disclosure, and application, all of which comply with the relevant laws and regulations of the relevant countries and regions and do not violate public order and good morals. Furthermore, in the technical solutions disclosed herein, authorization or consent from the data owner is obtained before acquiring or collecting any relevant data.

[0027] Figure 1 This is a flowchart of a sample image generation method according to an embodiment of the present disclosure.

[0028] like Figure 1 As shown, the sample image generation method 100 includes operations S110 to S150.

[0029] In operation S110, the raw image and thin cloud image are acquired.

[0030] In this embodiment of the disclosure, the original image is a cloudless ground feature image containing multiple bands, such as red, green, blue, and near-infrared bands. The thin cloud image is a single-band optical remote sensing image containing thin clouds. The thin cloud image is used to provide thin cloud signals for the generation process of the sample image. In some embodiments, the thin cloud image may be, for example, an optical remote sensing image acquired from the sea surface. The relatively uniform and non-redundant ground feature features of the sea surface background make it easy to extract thin cloud signals, thereby improving the accuracy of thin cloud simulation. It is understood that the thin cloud image in this embodiment of the disclosure is not limited to sea surface images, but may also be an optical remote sensing image containing thin clouds acquired from other ground feature types (such as deserts, grasslands, etc.), as long as it contains extractable thin cloud signals.

[0031] In this embodiment of the disclosure, the original image can be processed by band-by-band thin cloud simulation based on the thin cloud image to obtain single-band cloud images corresponding to each band, and sample images can be obtained accordingly. This process is described below.

[0032] In operation S120, cloud base component data is extracted from thin cloud images.

[0033] In this embodiment of the disclosure, extracting cloud fundamental component data from a thin cloud image includes the following steps: First, determining the minimum pixel value in the thin cloud image. Then, subtracting the minimum pixel value from each pixel value in the thin cloud image to obtain a difference image. After obtaining the difference image, setting the pixel values ​​in the difference image that are less than or equal to a preset threshold to zero, thereby obtaining the cloud fundamental component data. Here, the preset threshold can be set to 0, for example. This cloud fundamental component data is used to characterize the pure cloud radiation signal retained after removing background signals (such as sea surface) from the thin cloud image.

[0034] In operation S130, for each band of the original image, the basic component data of the cloud layer is corrected according to the spectral correction factor corresponding to that band and the compensation coefficient for the thin cloud image, and the cloud radiation component data corresponding to that band is generated.

[0035] In this embodiment of the disclosure, during the extraction of cloud fundamental component data from a thin cloud image, cloud edges or thinner cloud regions may be misclassified as background and set to zero, resulting in a loss of some cloud signal energy. To compensate for this energy loss, this embodiment of the disclosure utilizes a compensation coefficient for the thin cloud image to perform energy compensation on the cloud fundamental component data, thereby maintaining the consistency of cloud signal strength.

[0036] In this embodiment of the disclosure, the compensation coefficient can be generated based on the basic cloud component data and thin cloud image.

[0037] For example, the above compensation coefficient can be calculated using the following formula.

[0038]

[0039] In the above formula, I represents the compensation coefficient. c This represents a thin cloud image, where the total number of pixels in the thin cloud image is E*H, I c (e,h) represents the pixel value at coordinates (e,h) in the thin cloud image, where e and h represent the row and column indices in the thin cloud image, respectively. e = 1, 2, ..., E, h = 1, 2, ..., H, and E and H are both integers greater than 1. dc This represents the basic component data of the cloud layer, which is a matrix of the same size as the thin cloud image (matrix size E*H). dc (e,h) represents the basic cloud component value at position (e,h) in the matrix, where e represents the row index of the matrix, e=1,2,…,E. h represents the column index of the matrix, h=1,2,…,H. This is an indicator function.

[0040] According to the atmospheric scattering law, in the visible and near-infrared (VNIR) spectral range, the cloud reflectance ρ c The relationship with the center wavelength λ of the band can be expressed as: γ is a scattering parameter related to cloud particle size, used to describe the relationship between cloud reflectivity and wavelength in the visible-near-infrared band.

[0041] Based on the above rules, for any two VNIR bands i and j, the cloud reflectivity between them satisfies the following relationship:

[0042]

[0043] In the above formula, , These are the cloud reflectances for band i and band j, respectively. , These are the center wavelengths of band i and band j, respectively.

[0044] The ratio of the center wavelengths between band j and band i is denoted as I. n ,Right now:

[0045]

[0046] The above formula can then be expressed in the following form:

[0047]

[0048] Based on this physical law, the embodiments of this disclosure introduce a spectral correction factor. It is used to perform spectral correction on the basic component data of clouds to adjust the intensity ratio of cloud signals in a single band between different bands, so that the cloud radiation component data corresponding to each band in the final generation conforms to the physical law of cloud reflectivity changing with wavelength, thereby avoiding spectral distortion of sample images.

[0049] In this embodiment of the disclosure, for each band in the original image, the cloud reflectance ratio between the reference band and the band can be calculated based on the ratio between the center wavelength of the reference band and the center wavelength of the band and the preset scattering parameter γ, and the cloud reflectance ratio can be used as the spectral correction factor corresponding to the band.

[0050] In this embodiment of the invention, the scattering parameter γ can be randomly selected within the range of [0,2] to simulate the cloud reflectivity characteristics under different cloud particle sizes. Furthermore, for each band in the original image, the band with the highest reflectivity of the thin cloud within the VNIR spectral range (typically the blue band) can be used as a reference band. Then, the center wavelength of this reference band is applied to calculate the spectral correction factor corresponding to each band.

[0051] It should be noted that, in practical applications, to achieve better spectral correction, for each band in the original image, a preset empirical value for the cloud reflectance ratio can be used instead of the cloud reflectance ratio calculated based on the ratio between the center wavelength of the reference band and the center wavelength of the current band. For example, the empirical values ​​for the cloud reflectance ratios corresponding to the blue, green, red, and near-infrared bands in the original image are 0.976, 0.912, 0.834, and 0.652, respectively. These empirical values ​​for cloud reflectance ratios also follow the aforementioned physical laws.

[0052] After obtaining the aforementioned compensation coefficients and spectral correction factors corresponding to each band, energy compensation and spectral correction can be performed on the cloud base component data using the compensation coefficients and spectral correction factors corresponding to each band, thereby obtaining the cloud radiation component data corresponding to each band. For example, in this embodiment of the disclosure, the cloud radiation component data corresponding to each band can be calculated using the following formula.

[0053]

[0054] In the above formula, I ci This represents the cloud radiation component data corresponding to each band. I represents the compensation coefficient. dc This represents the basic component data of the cloud layer. This indicates the spectral correction factor corresponding to each band.

[0055] In this embodiment, the intensity of the cloud signal can be recovered by using a compensation coefficient to perform energy compensation on the cloud base component data. Furthermore, by using spectral correction factors corresponding to each band to correct the intensity of the energy-compensated cloud base component data for the corresponding bands, the final generated cloud radiation component data for each band conforms to physical laws. This achieves the transformation of the cloud signal from a geometric replication to a simulation of spectral characteristics, ensuring that the subsequently generated sample images not only visually represent thin clouds but also maintain spectral consistency with thin clouds in real satellite imagery. This, in turn, guarantees the generalization ability and accuracy of the cloud removal model trained based on these sample images in practical applications.

[0056] In operation S140, for each band of the original image, a single-band cloud image corresponding to that band is generated based on the original single-band image corresponding to that band, the cloud radiation component data corresponding to that band, and the preset atmospheric light value.

[0057] The process of generating single-band cloud images corresponding to each band is explained below.

[0058] First, the cloud radiation component data corresponding to each band are inverted to obtain the cloud transmission weights corresponding to each band.

[0059]

[0060] In the above formula, I represents the cloud transmission weight corresponding to each band. ci This represents the cloud radiation component data corresponding to each band, where B is the gray value matrix corresponding to the maximum gray value, and the size of this gray value matrix is ​​equal to that of I. ci Consistent. For example, when the original image is an 8-bit optical remote sensing image, the maximum grayscale value is 255.

[0061] Next, based on the original single-band images and cloud transmission weights corresponding to each band, the ground radiation component data transmitted through the cloud layer corresponding to each band are obtained.

[0062]

[0063] In the above formula, I ol This represents the ground object radiation component data transmitted through clouds corresponding to each band, I cf This represents the original single-band image corresponding to each band, and ⨀ indicates element-wise multiplication.

[0064] Next, based on the cloud radiation component data corresponding to each band and the preset atmospheric light value, the cloud scattered radiation component data corresponding to each band is determined.

[0065] In this embodiment of the disclosure, the atmospheric light value is a preset value. For example, the atmospheric light value is 0.95.

[0066] For example, the cloud scattering radiation component data corresponding to each band can be calculated according to the following formula.

[0067]

[0068] In the above formula, I aci I represents the cloud scattering radiation component data corresponding to each band. ci This represents the cloud radiation component data corresponding to each band, where A represents the atmospheric light value.

[0069] Next, the ground object radiation component data transmitted through the cloud layer and the cloud-scattered radiation component data corresponding to each band are superimposed to obtain the single-band cloud image corresponding to each band.

[0070]

[0071] In the above formula, I sc This indicates the cloud-covered images corresponding to each band.

[0072] In operation S150, the single-band cloud images corresponding to each band are combined to obtain a sample image corresponding to the original image.

[0073] In this embodiment of the disclosure, the sample image is an optical remote sensing image containing thin clouds, which can be used as a training sample to train the initial cloud removal model.

[0074] Figure 2 This is a flowchart of a training method for a cloud-free model according to an embodiment of the present disclosure.

[0075] like Figure 2 As shown, the training method 200 for the cloud removal model includes operations S210 to S230.

[0076] In operation S210, a sample image is acquired.

[0077] In this embodiment of the disclosure, the sample image is an optical remote sensing image containing a thin cloud region. This sample image is obtained according to the sample image generation method described in the above embodiments, and will not be repeated here.

[0078] In operation S220, the initial declouding model is used to process the sample image to remove clouds, resulting in a repaired image.

[0079] Figure 3A A schematic diagram illustrating the structure of the initial cloud removal model is shown below. Figure 3A The process of generating a repaired image using an initial declouding model is illustrated by way of example.

[0080] like Figure 3A As shown, the initial cloud removal model includes a shallow feature extraction network, an encoding network, a decoding network, a feature refinement network, and a reconstruction network.

[0081] Shallow feature extraction networks are used to extract shallow features from sample images to obtain initial features. Shallow feature extraction networks typically include, for example, 3x3 convolutional layers.

[0082] The encoding network comprises M cascaded downsampling modules, where M is an integer greater than 1. Each downsampling module includes N stacked feature refinement feedforward submodules and downsampling submodules, used to refine the input features through N levels before performing downsampling, where N is an integer greater than 0. The N feature refinement feedforward submodules all employ the same structure, performing layer-by-layer feature refinement on the input features. For example, the downsampling submodules can use 3x3 convolutional layers to implement the downsampling process.

[0083] The decoding network comprises M cascaded upsampling modules. Each upsampling module includes N stacked adaptive feature enhancement submodules and upsampling submodules, used to perform N levels of adaptive feature enhancement on the input features before upsampling. The N adaptive feature enhancement submodules all employ the same structure, performing layer-by-layer adaptive feature enhancement on the input features. For example, the upsampling submodule can use a 3x3 convolutional layer to implement the upsampling process.

[0084] The feature refinement network comprises L stacked adaptive enhancement modules, where L is a positive integer, used to perform L levels of adaptive feature enhancement processing on the features output by the decoding network to obtain deep features. All L adaptive enhancement modules employ the same structure to implement layer-by-layer adaptive feature enhancement processing.

[0085] The reconstruction network is used to generate a residual image by refining the depth features of the network output. This residual image contains pixel-level difference information from the sample image (i.e., an optical remote sensing image containing thin cloud regions) to the restored image (i.e., an optical remote sensing image without thin clouds). The restored image can be obtained based on the residual image and the sample image. In some embodiments, the reconstruction network may include, for example, 3x3 convolutional layers.

[0086] Based on the initial cloud removal model described above, the repaired image can be obtained through the following steps.

[0087] First, a shallow feature extraction network is used to extract shallow features from the sample image to obtain initial features.

[0088] Next, the initial features are downsampled at M levels using M downsampling modules in the encoding network to obtain M levels of downsampled features.

[0089] According to embodiments of this disclosure, the initial features are downsampled at M levels using M downsampling modules in the coding network to obtain M-level downsampled features. This includes: for the m-th downsampling module in the M downsampling modules, where m is a positive integer less than or equal to M, using N stacked feature refinement feedforward sub-modules in the m-th downsampling module to perform N-level feature refinement on the input features of the m-th downsampling module to obtain the N-level refined features; and using the downsampling sub-modules in the m-th downsampling module to perform downsampling on the N-level refined features to obtain the m-level downsampled features. Wherein, when m equals 1, the initial features are used as the input features of the m-th downsampling module, and when m is greater than 1 and less than or equal to M, the downsampled features of the (m-1)-th level are used as the input features of the m-th downsampling module.

[0090] The following is for reference. Figure 3A The process of obtaining downsampled features at M levels is illustrated by an example.

[0091] like Figure 3A As shown, after the initial features are input into the encoding network, the initial features are first refined through N levels using the N stacked feature refinement feedforward submodules in the first downsampling module, resulting in the refined features of the Nth level. Then, the refined features of the Nth level are downsampled using the downsampling submodule in the first downsampling module, resulting in the downsampled features of the first level.

[0092] Next, the N stacked feature refinement feedforward submodules in the second downsampling module are used to refine the downsampled features of the first level through N levels of feature refinement, resulting in another refined feature of the Nth level. Then, the downsampling submodules in the second downsampling module are used to downsample the other refined feature of the Nth level, resulting in the downsampled features of the second level.

[0093] By analogy, the downsampling features of the Mth level can be obtained.

[0094] Next, using the M upsampling modules in the decoding network based on the downsampling features of the M levels, the downsampling features of the Mth level are subjected to M-level upsampling processing to obtain the upsampling features of the Mth level.

[0095] According to embodiments of this disclosure, the downsampling features of the Mth level are upsampled at M levels based on downsampling features of M levels using M upsampling modules in the decoding network to obtain the upsampling features of the Mth level. This includes: for the mth upsampling module among the M upsampling modules, where m is a positive integer less than or equal to M, fusing the input features of the mth upsampling module with the downsampling features of the (M-m+1)th level to obtain the fused features of the mth level; and utilizing N stacked adaptive feature enhancers in the mth upsampling module. The module performs N-level adaptive feature enhancement processing on the fused features of the m-th level to obtain the first adaptive enhanced feature of the N-th level; the upsampling sub-module in the m-th upsampling module performs upsampling processing on the first adaptive enhanced feature of the N-th level to obtain the upsampled feature of the m-th level; wherein, when m equals 1, the downsampled feature of the M-th level is used as the input feature of the m-th upsampling module, and when m is greater than 1 and less than or equal to M, the upsampled feature of the (m-1)-th level is used as the input feature of the m-th upsampling module.

[0096] The following is for reference. Figure 3A The process of obtaining the upsampled features of the Mth level is illustrated by an example.

[0097] like Figure 3A As shown, after the downsampled features of the Mth level are input into the decoding network, the downsampled features of the Mth level are first fused with the downsampled features of the Mth level output by the Mth downsampled module in the encoding network. Figure 3A Sino-Israeli symbols (This represents the fusion process), resulting in the fused features of the first level. Then, using N stacked adaptive feature enhancement submodules in the first upsampling module, the fused features of the first level are subjected to N levels of adaptive feature enhancement processing to obtain the first adaptive enhanced features of the Nth level. Afterwards, the first adaptive enhanced features of the Nth level are upsampled using the upsampling submodule in the first upsampling module to obtain the upsampled features of the first level.

[0098] Next, the upsampled features of the first level are fused with the downsampled features of the (M-1)th level output by the (M-1)th downsampled module in the encoding network to obtain the fused features of the second level. Then, using the N stacked adaptive feature enhancement submodules in the second upsampled module, the fused features of the second level are subjected to N levels of adaptive feature enhancement processing to obtain another first adaptive enhanced feature of the Nth level. Afterwards, the upsampled submodules in the second upsampled module are used to upsample the other first adaptive enhanced feature of the Nth level to obtain the upsampled features of the second level.

[0099] Next, the upsampled features of the second level are fused with the downsampled features of the (M-2)th level output by the (M-2)th downsampled module in the encoding network to obtain the fused features of the third level. Then, using the N stacked adaptive feature enhancement submodules in the third upsampled module, the fused features of the third level are subjected to N levels of adaptive feature enhancement processing to obtain another first adaptive enhanced feature of the Nth level. Afterwards, the upsampled submodules in the third upsampled module are used to upsample this first adaptive enhanced feature of the Nth level to obtain the upsampled features of the third level.

[0100] By analogy, the upsampled features of the Mth level can be obtained.

[0101] Next, the upsampled features of the Mth layer are processed using a feature refinement network to obtain deep features.

[0102] In this embodiment of the disclosure, L stacked adaptive feature enhancement modules in the feature refinement network can be used to perform L-level adaptive feature enhancement processing on the upsampled features of the M-th layer to obtain the second adaptive enhanced feature of the L-th layer. Then, the second adaptive enhanced feature of the L-th layer is determined as a depth feature.

[0103] Next, the deep features are processed using a reconstruction network to obtain a residual image, and the repaired image is obtained based on the residual image and the sample image.

[0104] For example, a 3x3 convolutional layer is used to convolve the depth features output by the feature refinement network to obtain a residual image. Then, the residual image and the sample image are added together to obtain the repaired image corresponding to the sample image.

[0105] In operation S230, the parameters of the initial declouding model are adjusted based on the original image corresponding to the repaired image and the sample image to obtain the declouding model.

[0106] In this embodiment, the original image corresponding to the sample image is a cloudless optical remote sensing image containing multiple bands, as described in the above embodiments. The sample loss can be determined based on the difference between the restored image and the original image. For example, the sample loss can be calculated using a mean squared error loss function or a cross-entropy loss function. Then, the sample loss is used to adjust the model parameters of the initial declouding model to obtain a declouding model. This declouding model is used to perform thin cloud removal processing on the image to be processed.

[0107] Figure 3B The schematic diagram illustrates the structure of each feature refinement feedforward submodule in the m-th downsampling module of the encoding network.

[0108] like Figure 3BAs shown, each feature refinement feedforward submodule includes a first segmentation unit, a convolutional unit, a stitching unit, a first channel transform layer, a second segmentation unit, a deep convolutional layer, a first fusion unit, a second channel transform layer, and a first residual connection unit. The first fusion unit can use... Figure 3B In The first residual connection unit can be represented by a symbol. Figure 3B In Symbols are used to represent them.

[0109] The following example uses the nth (where n is a positive integer less than or equal to N) feature refinement feedforward submodule out of N feature refinement feedforward submodules. (Refer to...) Figure 3B This example illustrates the process of using the nth feature to refine the feedforward submodule to generate the refined features of the nth level.

[0110] First, the input features of the nth feature refinement feedforward submodule are divided into multiple sub-input features along the channel dimension using the first segmentation unit in the nth feature refinement feedforward submodule.

[0111] In this embodiment of the disclosure, when n equals 1, the input features of the m-th downsampling module can be used as the input features of the n-th feature refinement feedforward submodule. When n is greater than 1 and less than or equal to N, the refinement features of the (n-1)-th level are used as the input features of the n-th feature refinement feedforward submodule.

[0112] After obtaining the input features of the nth feature refinement feedforward submodule, the first segmentation unit is used to segment the input features of the nth feature refinement feedforward submodule into G sub-input features along the channel dimension. The number of channels of each sub-input feature is 1 / G of the number of channels of the input features of the nth feature refinement feedforward submodule, where G is an integer greater than 1.

[0113] It should be noted that, in some other embodiments, before processing the input features of the nth feature refinement feedforward submodule using the first segmentation unit, the input features of the nth feature refinement feedforward submodule can be first normalized using a first normalization layer to obtain the first normalized features. Then, the first normalized features are input into the first segmentation unit for processing to obtain multiple sub-input features.

[0114] Next, the convolutional units in the nth feature refinement feedforward submodule are used to perform partial depth convolution processing on multiple sub-input features to obtain multiple first sub-convolutional features.

[0115] In this embodiment, the convolutional unit performs feature extraction through partial depth-wise convolution. Specifically, for each of the G sub-input features, the convolutional unit divides the sub-input feature into a first channel part and a second channel part along the channel dimension. The first channel part is then subjected to depth-wise convolution, and the processed first channel part is concatenated with the second channel part along the channel dimension to obtain the first sub-convolutional feature corresponding to that sub-input feature. This embodiment effectively extracts useful features from each sub-input feature by performing partial depth-wise convolution on each sub-input feature.

[0116] Next, the concatenation unit in the nth feature refinement feedforward submodule is used to concatenate multiple first sub-convolutional features along the channel dimension to obtain enhanced features.

[0117] Next, the enhanced features are processed by the first channel transform layer in the nth feature refinement feedforward submodule to obtain the first transform feature.

[0118] In this embodiment, the first channel transformation layer is, for example, a 1*1 convolutional layer, used to expand the number of channels. By using the first channel transformation layer to perform a linear transformation on the enhanced features, the channel dimension of the enhanced features is expanded, thereby enhancing the expressive power of the features.

[0119] Next, the second segmentation unit in the nth feature refinement feedforward submodule is used to split the first transform feature into two parts along the channel dimension, resulting in the first sub-transform feature and the second sub-transform feature.

[0120] In this embodiment, the first transformation feature can be divided into two parts along the channel dimension using a second segmentation unit, resulting in a first sub-transformation feature and a second sub-transformation feature. The first sub-transformation feature is left unprocessed, while the second sub-transformation feature is used for subsequent spatial feature extraction.

[0121] Next, the deep convolutional layer in the nth feature refinement feedforward submodule is used to perform deep convolution on the second sub-transformation feature to obtain the second sub-convolutional feature.

[0122] In this embodiment, the deep convolutional layer can be, for example, a 3x3 deep convolutional layer. By performing channel-by-channel convolution on the second sub-transformation features using the deep convolutional layer, computational complexity can be significantly reduced while capturing local spatial information. The aforementioned second sub-convolutional features serve as spatial gating signals, used for subsequent dynamic weighting of the first sub-transformation features, thereby achieving the filtering of channel information from the first sub-transformation features.

[0123] Next, the first fusion unit in the nth feature refinement feedforward submodule is used to fuse the first sub-transformation feature and the second sub-convolution feature to obtain the first gated feature.

[0124] In this embodiment, the first fusion sub-unit is used to element-wise multiply the first sub-transform feature and the second sub-convolution feature to selectively enhance or suppress the channel information of the first sub-transform feature, thereby obtaining the first gated feature. This first gated feature retains both the global information of the channel dimension and incorporates the local correlation of the spatial dimension, thus improving the effectiveness and relevance of feature representation.

[0125] Next, the second channel transformation layer in the nth feature refinement feedforward submodule is used to process the first gated feature to obtain the second transformation feature.

[0126] In this embodiment, the second channel transformation layer is, for example, a 1*1 convolutional layer, used to restore the number of channels of the first gated feature to be consistent with the number of channels of the input feature of the nth feature refinement feedforward submodule. This facilitates subsequent residual connections.

[0127] Next, the second transformed feature and the input feature of the nth feature refinement feedforward submodule are fused using the first residual connection unit in the nth feature refinement feedforward submodule to obtain the refined feature of the nth level.

[0128] In some embodiments, each adaptive feature enhancement submodule in the m-th upsampling module described above includes an attention submodule and a feature refinement feedforward submodule.

[0129] According to embodiments of this disclosure, the above-described method of using N stacked adaptive feature enhancement sub-modules in the m-th upsampling module to perform N-level adaptive feature enhancement processing on the fusion features of the m-th level to obtain the first adaptive enhanced feature of the N-th level includes: for the n-th adaptive feature enhancement sub-module in the N adaptive feature enhancement sub-modules, where n is a positive integer less than or equal to N, processing the input features of the n-th adaptive feature enhancement sub-module using the attention sub-module in the n-th adaptive feature enhancement sub-module to obtain the first attention fusion feature; and using the feature refinement feedforward sub-module in the n-th adaptive feature enhancement sub-module to refine the first attention fusion feature to obtain the first adaptive enhanced feature of the n-th level; wherein, when n equals 1, the fusion features of the m-th level are used as the input features of the n-th adaptive feature enhancement sub-module, and when n is greater than 1 and less than or equal to N, the first adaptive enhanced feature of the (n-1)-th level is used as the input features of the n-th adaptive feature enhancement sub-module.

[0130] Figure 3C The schematic diagram illustrates the structure of the attention submodule.

[0131] like Figure 3C As shown, the attention submodule includes a first feature mapping layer, a second feature mapping layer, a third feature mapping layer, a softmax activation layer, a ReLU squared activation layer, a gating unit, a first linear mapping layer, a second linear mapping layer, a third channel transform layer, and a second residual connection unit.

[0132] The following is combined Figure 3C The example illustrates the process of using the attention submodule in the nth adaptive feature enhancement submodule to process the input features of the nth adaptive feature enhancement submodule to obtain the first attention fusion feature.

[0133] First, the input features of the nth adaptive feature enhancement submodule are processed using the first feature mapping layer, the second feature mapping layer, and the third feature mapping layer, respectively, to obtain query features, key features, and value features.

[0134] In this embodiment, the first, second, and third feature mapping layers all employ the same convolutional structure, for example, they each include a sequentially connected 1x1 convolutional layer and a 3x3 depthwise convolutional layer. Specifically, the first feature mapping layer first uses a 1x1 convolutional layer to linearly map the input features of the nth adaptive feature enhancement submodule. Subsequently, a 3x3 depthwise convolution is performed on the mapped input features of the nth adaptive feature enhancement submodule to generate query features. Similarly, the second and third feature mapping layers process the input features of the nth adaptive feature enhancement submodule in the same way, generating key features and value features accordingly.

[0135] It should be noted that, in some other embodiments, before processing the input features of the nth adaptive feature enhancement submodule using the first, second, and third feature mapping layers described above, the input features of the nth adaptive feature enhancement submodule can first be normalized using the second normalization layer to obtain the second normalized features. Then, the second normalized features are input into the first, second, and third feature mapping layers for further processing.

[0136] Next, a dense attention weight is generated using a Softmax activation layer based on query features and key features.

[0137] In this embodiment of the disclosure, the Softmax activation layer can be implemented using the Softmax activation function. For example, the dense attention weights can be calculated using the following formula.

[0138]

[0139] In the above formula, W DSA This represents dense attention weights, Softmax() represents the Softmax activation function, Q represents the query feature, and K represents the key feature. This represents the learnable scaling parameters.

[0140] Next, a ReLU squared activation layer is used to generate sparse attention weights based on query features and key features.

[0141] In this embodiment of the disclosure, the ReLU squared activation layer may employ ReLU. 2 Activation function implementation. For example, sparse attention weights can be calculated using the following formula.

[0142]

[0143] In the above formula, W SSA ReLU represents sparse attention weights. 2 () represents ReLU 2 The activation function, where Q represents the query feature and K represents the key feature, This represents the learnable scaling parameters.

[0144] Next, the second gating feature is obtained by using the gating unit based on the sparse attention weights.

[0145] In this embodiment of the disclosure, the gating unit processes the sparse attention weights through the GELU activation function to obtain a second gating feature for modulation.

[0146] For example, the second gating feature can be determined by the following formula.

[0147]

[0148] In the above formula, F SG Denotes the second gating feature, GELU() represents the GELU activation function, and W SSA ⨀ represents the sparse attention weights and ⨀ represents the Hamada product.

[0149] Next, the second gated feature is linearly mapped using the first linear mapping layer and the second linear mapping layer respectively to obtain the scaling factor and the offset factor.

[0150] In this embodiment, the first linear mapping layer and the second linear mapping layer employ the same structure, for example, both being 1x1 convolutional layers. The first linear mapping layer is used to linearly map the second gated feature, generating a scaling factor. The second linear mapping layer is used to linearly map the second gated feature, generating an offset factor. The scaling factor and offset factor are used for subsequent pixel-level modulation of the dense attention weights.

[0151] For example, the scaling factor and offset factor can be determined by the following formula.

[0152]

[0153]

[0154] In the above formula, Indicates the scaling factor. F represents the offset factor. SG Indicates the second gating feature. This represents the learnable weight parameters of the first linear mapping layer. This represents the learnable weight parameters of the second linear mapping layer.

[0155] Next, the dense attention weights are modulated at the pixel level according to the scaling factor and the offset factor to obtain adaptive attention weights. The adaptive attention weights are then used to perform weighted fusion of the value features to obtain weighted attention features.

[0156] For example, weighted attention features can be determined using the following formula.

[0157]

[0158]

[0159] In the above formula, W ASSA Indicates adaptive attention weights, Indicates the scaling factor. W represents the offset factor. DSA F represents the dense attention weights. A ∠ represents the weighted attention feature, V represents the value feature, and ⨀ represents the Hamada product.

[0160] In this embodiment of the disclosure, the dense attention weights are modulated at the pixel level using scaling factors and offset factors to obtain adaptive attention weights. The adaptive attention weights are then used to perform weighted fusion of value features. This can effectively suppress noise interference from irrelevant regions while retaining important feature interactions, thereby improving the feature selection capability of the attention mechanism and ultimately enhancing the representation effectiveness of weighted attention features.

[0161] Next, the weighted attention features are processed using the third channel transformation layer to obtain the third transformation features.

[0162] In this embodiment, the third channel transformation layer, for example, is a 1*1 convolutional layer, used to linearly map the weighted attention features to restore the number of channels of the weighted attention features to be consistent with the number of channels of the input features of the nth adaptive feature enhancement submodule. This facilitates subsequent residual connections.

[0163] Next, the second residual connection unit is used to fuse the input features of the third transform feature and the nth adaptive feature enhancement submodule to obtain the first attention fusion feature. The second residual connection unit can use... Figure 3C In Symbols are used to represent them.

[0164] According to embodiments of this disclosure, the feature refinement feedforward submodule in each adaptive feature enhancement submodule has the same structure as the feature refinement feedforward submodule described in the above embodiments. Furthermore, the process of refining the first attention fusion features using the feature refinement feedforward submodule in the nth adaptive feature enhancement submodule is similar to the process of generating the refined features of the nth level using the nth feature refinement feedforward submodule described in the above embodiments; therefore, it will not be repeated here for brevity.

[0165] According to embodiments of this disclosure, each adaptive feature enhancement module in the feature refinement network described above includes an attention submodule and a feature refinement feedforward submodule.

[0166] In this embodiment of the disclosure, the L-level adaptive feature enhancement processing is performed on the upsampled features of the M-th level using L stacked adaptive feature enhancement modules to obtain the second adaptive enhanced feature of the L-th level. This includes: for the l-th adaptive feature enhancement module among the L adaptive feature enhancement modules, where l is a positive integer less than or equal to L, the input features of the l-th adaptive feature enhancement module are processed using the attention submodule in the l-th adaptive feature enhancement module to obtain the second attention fusion feature; and the second attention fusion feature is refined using the feature refinement feedforward submodule in the l-th adaptive feature enhancement module to obtain the second adaptive enhanced feature of the l-th level. Wherein, when l equals 1, the upsampled features of the M-th level are used as the input features of the l-th adaptive feature enhancement module; when l is greater than 1 and less than or equal to L, the second adaptive enhanced feature of the (l-1)-th level is used as the input features of the l-th adaptive feature enhancement module.

[0167] In this embodiment, the attention submodule in the l-th adaptive feature enhancement module has the same structure as the attention submodule in the n-th adaptive feature enhancement module. Furthermore, the process of processing the input features of the l-th adaptive feature enhancement module using the attention submodule in the l-th adaptive feature enhancement module is similar to the process of processing the input features of the n-th adaptive feature enhancement module using the attention submodule in the n-th adaptive feature enhancement module; therefore, it will not be described in detail here for the sake of brevity.

[0168] Similarly, the feature refinement feedforward submodule in the l-th adaptive feature enhancement module has the same structure as the feature refinement feedforward submodule. Furthermore, the process of refining the second attention fusion features using the feature refinement feedforward submodule in the l-th adaptive feature enhancement module is similar to the process described in the above embodiments of generating the refined features of the nth level using the n-th feature refinement feedforward submodule; therefore, it will not be elaborated further here to save space.

[0169] Figure 4 This is a flowchart of a remote sensing image processing method according to an embodiment of the present disclosure.

[0170] like Figure 4 As shown, the remote sensing image processing method 400 includes operations S410 to S470.

[0171] In operation S410, an image to be processed is acquired. The image to be processed is an optical remote sensing image containing thin cloud regions.

[0172] In operation S420, the image to be processed is divided into multiple first image blocks, each first image block having a first resolution.

[0173] Given that the images to be processed are typically large in size, direct global processing would incur significant computational overhead. Therefore, the image can be pre-divided into multiple image blocks, and then cloud removal can be performed on each block. This approach significantly reduces the computational complexity of the cloud removal process and improves the effectiveness of thin cloud removal.

[0174] For example, the image to be processed can be divided into multiple first image blocks of size 512*512, each first image block having a first resolution.

[0175] In operation S430, the cloud removal model is used to perform cloud removal processing on multiple first image blocks to obtain multiple first repaired image blocks, and an initial repaired image is obtained based on the multiple first repaired image blocks.

[0176] For example, multiple 512*512 size first image blocks are input into the cloud removal model for cloud removal processing to obtain multiple first restored image blocks. Then, the multiple first restored image blocks are stitched together to obtain the initial restored image. The above-mentioned cloud removal model is trained according to the training method of the cloud removal model described in the above embodiments, and will not be repeated here.

[0177] The initial restored image obtained after the initial cloud removal process may still contain areas of thin cloud that have not been removed. To improve the restoration quality, the initial restored image can be further divided into multiple second image blocks with a second resolution, and then each second image block can be subjected to cloud removal processing.

[0178] In operation S440, the initial repaired image is divided into multiple second image blocks, each second image block having a second resolution, and the second resolution is different from the first resolution.

[0179] In the embodiments of this disclosure, the block size of the second image block can be flexibly set according to actual application requirements. For example, a block size of 4096*4096 can be used. In other embodiments, a smaller block size, such as 256*256, can also be used to adapt to different processing precision or computing resource constraints.

[0180] In operation S450, multiple second image blocks are resampled to obtain multiple resampled second image blocks, each of which has a first resolution.

[0181] In this embodiment of the disclosure, in order to maintain the consistency of the input resolution, each second image block is resampled to convert its resolution from the second resolution to the first resolution, resulting in multiple resampled second image blocks.

[0182] In operation S460, the cloud removal model is used to perform cloud removal processing on multiple resampled second image blocks to obtain multiple second repaired image blocks, and a second repaired image is obtained based on the multiple second repaired image blocks.

[0183] Similarly, the cloud removal model described above is used to remove clouds from multiple resampled second image patches to obtain multiple second restored image patches. Then, the multiple second restored image patches are stitched together to obtain the second restored image.

[0184] In operation S470, in response to determining that the second restored image does not contain thin cloud regions, the second restored image is used as a reference restored image, and with the reference restored image as a reference, the initial restored image is color-corrected based on the wavelet transform algorithm to obtain the target restored image corresponding to the image to be processed.

[0185] In this embodiment, if it is determined that the second restored image does not contain thin cloud regions, then the second restored image is determined as a reference restored image. In other embodiments, if it is determined that the second restored image still contains thin cloud regions that have not been removed, the second restored image can be further segmented into multiple third image blocks, each third image block having a third resolution, and the third resolution being different from the second resolution. Then, the multiple third image blocks are resampled to obtain multiple resampled third image blocks, each resampled third image block having a first resolution. Then, a cloud removal model is used to perform cloud removal processing on the multiple resampled third image blocks to obtain multiple third restored image blocks, and a third restored image is obtained based on the multiple third restored image blocks. If the third restored image does not contain thin cloud regions, then the third restored image is determined as a reference restored image. If the third restored image still contains thin cloud regions that have not been removed, then the above operation is repeated until a reference restored image without thin cloud regions is obtained.

[0186] The aforementioned reference restoration image serves as the baseline source for color correction. Wavelet transform is used to guide the color adjustment of the initial restoration image in the frequency domain, enabling it to obtain accurate colors consistent with the reference image while maintaining texture details, ultimately generating a high-quality target restoration image.

[0187] The following example illustrates the process of color correction for the initial restored image based on the wavelet transform algorithm.

[0188] For example, wavelet decomposition is performed on both the initial restored image and the reference restored image to decompose them into low-frequency subbands representing color and illumination information and high-frequency subbands representing texture and edge information, respectively. Then, the low-frequency subbands of the reference restored image and the high-frequency subbands of the initial restored image are added together, and an inverse wavelet transform is performed to reconstruct the image, thereby generating a target restored image with accurate colors and natural visual appearance while preserving the original texture details.

[0189] Figure 5 This is a block diagram of a sample image generation apparatus according to an embodiment of the present disclosure.

[0190] like Figure 5 As shown, the sample image generation device 500 includes: a first acquisition module 510, an extraction module 520, a first generation module 530, a second generation module 540, and a combination module 550.

[0191] The first acquisition module 510 is used to acquire the original image and the thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds.

[0192] The extraction module 520 is used to extract basic cloud component data from thin cloud images.

[0193] The first generation module 530 is used to correct the basic component data of the cloud layer for each band of the original image according to the spectral correction factor corresponding to the band and the compensation coefficient for the thin cloud image, and generate the cloud radiation component data corresponding to the band.

[0194] The second generation module 540 is used to generate a single-band cloud image corresponding to each band of the original image, based on the single-band original image corresponding to that band, the cloud radiation component data corresponding to that band, and the preset atmospheric light value.

[0195] The combination module 550 is used to combine the single-band cloud images corresponding to each band to obtain a sample image corresponding to the original image. The sample image is an optical remote sensing image containing thin clouds.

[0196] Figure 6 This is a block diagram of a training apparatus for a cloud-free model according to an embodiment of the present disclosure.

[0197] like Figure 6 As shown, the training device 600 for the cloud-free model includes: a second acquisition module 610, a processing module 620, and a parameter tuning module 630.

[0198] The second acquisition module 610 is used to acquire a sample image, which is obtained from the sample image generation device described above.

[0199] The processing module 620 is used to perform cloud removal processing on the sample image using the initial cloud removal model to obtain the repaired image.

[0200] The parameter tuning module 630 is used to tune the initial declouding model based on the original image corresponding to the repaired image and the sample image to obtain a declouding model, which is used to perform declouding processing on the image to be processed.

[0201] Figure 7 This is a block diagram of a remote sensing image processing apparatus according to an embodiment of the present disclosure.

[0202] like Figure 7 As shown, the remote sensing image processing device 700 includes: a third acquisition module 710, a first segmentation module 720, a first repair module 730, a second segmentation module 740, a resampling module 750, a second repair module 760, and a correction module 770.

[0203] The third acquisition module 710 is used to acquire the image to be processed, which is an optical remote sensing image containing thin cloud regions.

[0204] The first segmentation module 720 is used to segment the image to be processed into multiple first image blocks, each first image block having a first resolution.

[0205] The first restoration module 730 is used to perform cloud removal processing on multiple first image blocks using a cloud removal model to obtain multiple first restored image blocks, and to obtain an initial restored image based on the multiple first restored image blocks; wherein, the cloud removal model is trained by the training device of the cloud removal model described above.

[0206] The second segmentation module 740 is used to segment the initial repaired image into multiple second image blocks, each second image block having a second resolution, and the second resolution being different from the first resolution.

[0207] The resampling module 750 is used to resample multiple second image blocks respectively to obtain multiple resampled second image blocks, each of which has a first resolution.

[0208] The second repair module 760 is used to perform cloud removal processing on multiple resampled second image blocks using a cloud removal model to obtain multiple second repair image blocks, and to obtain a second repaired image based on the multiple second repair image blocks.

[0209] The correction module 770 is used to respond to determining that the second restored image does not contain thin cloud regions, to use the second restored image as a reference restored image, and to perform color correction processing on the initial restored image based on the wavelet transform algorithm, using the reference restored image as a reference, to obtain the target restored image corresponding to the image to be processed.

[0210] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module in the device embodiment are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiment, and will not be repeated here.

[0211] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a sample image generation method, a cloud removal model training method, and a remote sensing image processing method according to embodiments of the present disclosure.

[0212] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0213] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0214] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0215] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the sample image generation method, the cloud removal model training method, and the remote sensing image processing method according to the embodiments of this disclosure.

[0216] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0217] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the sample image generation method, the cloud removal model training method, and the remote sensing image processing method provided in the embodiments of this disclosure.

[0218] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0219] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0220] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0221] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0222] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0223] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0224] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for generating a sample image, comprising: Acquire the original image and the thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds; Extract the basic cloud component data from the thin cloud image; For each band of the original image, the cloud base component data is corrected according to the spectral correction factor corresponding to the band and the compensation coefficient for the thin cloud image, to generate cloud radiation component data corresponding to the band. For each band of the original image, a single-band cloud image corresponding to the band is generated based on the original single-band image corresponding to the band, the cloud radiation component data corresponding to the band, and the preset atmospheric light value. The single-band cloud images corresponding to each band are combined to obtain a sample image corresponding to the original image. The sample image is an optical remote sensing image containing thin clouds.

2. The method of claim 1, wherein, The spectral correction factor corresponding to the band is determined in the following way: The cloud reflectivity ratio between the reference band and the band is determined based on the ratio between the center wavelength of the reference band and the center wavelength of the band, and the preset scattering parameters. The cloud reflectance ratio is used as the spectral correction factor corresponding to the band.

3. The method of claim 2, wherein, The scattering parameter is a value greater than or equal to 0 and less than or equal to 2.

4. The method of any one of claims 1 to 3, wherein, The step of generating a single-band cloud image corresponding to the band based on the original single-band image corresponding to the band, the cloud radiation component data corresponding to the band, and the preset atmospheric light value includes: The cloud radiation component data corresponding to the band is inverted to obtain the cloud transmission weight corresponding to the band. Based on the original single-band image corresponding to the band and the cloud transmission weight corresponding to the band, the ground object radiation component data transmitted through the cloud corresponding to the band is obtained. Based on the cloud radiation component data corresponding to the band and the atmospheric light value, determine the cloud scattered radiation component data corresponding to the band; The ground object radiation component data transmitted through the cloud layer corresponding to the band is superimposed with the cloud-scattered radiation component data corresponding to the band to obtain a single-band cloud image corresponding to the band.

5. The method of any one of claims 1 to 4, wherein, The extraction of cloud layer basic component data from the thin cloud image includes: Determine the minimum pixel value in the thin cloud image; Subtract the minimum pixel value from each pixel value in the thin cloud image to obtain a difference image; The pixel values ​​in the difference image that are less than or equal to a preset threshold are set to zero to obtain the basic component data of the cloud layer.

6. The method of any one of claims 1 to 5, wherein, The compensation coefficient is generated based on the cloud base component data and the thin cloud image.

7. A method for training a cloud-free model, comprising: A sample image is obtained, wherein the sample image is generated by the sample image generation method according to any one of claims 1 to 6; The sample image is processed to remove clouds using the initial cloud removal model to obtain the repaired image; The initial declouding model is adjusted based on the repaired image and the original image corresponding to the sample image to obtain a declouding model, which is used to perform declouding on the image to be processed.

8. The method of claim 7, wherein, The initial cloud removal model includes a shallow feature extraction network, an encoding network, a decoding network, a feature refinement network, and a reconstruction network. The encoding network includes M downsampling modules, and the decoding network includes M upsampling modules, where M is an integer greater than 1. The process of removing clouds from the sample image using the initial cloud removal model to obtain the repaired image includes: The shallow feature extraction network is used to extract shallow features from the sample image to obtain initial features; The initial features are downsampled at M levels using M downsampling modules in the encoding network to obtain M levels of downsampled features; Using the M upsampling modules in the decoding network based on the downsampling features of M levels, the downsampling features of the Mth level are subjected to M-level upsampling processing to obtain the upsampling features of the Mth level. The upsampled features of the Mth layer are processed using the feature refinement network to obtain deep features; The deep features are processed using the reconstruction network to obtain a residual image, and the repaired image is obtained based on the residual image and the sample image.

9. The method of claim 8, wherein, Each downsampling module includes N stacked feature refinement feedforward sub-modules and downsampling sub-modules, where N is an integer greater than 0; the step of using the M downsampling modules in the encoding network to perform M-level downsampling processing on the initial features to obtain M-level downsampling features includes: for the m-th downsampling module among the M downsampling modules, where m is a positive integer less than or equal to M, By utilizing the N stacked feature refinement feedforward sub-modules in the m-th downsampling module, the input features of the m-th downsampling module are subjected to N levels of feature refinement processing to obtain the refined features of the N-th level. The refined features of the Nth level are downsampled using the downsampling submodule in the m-th downsampling module to obtain the downsampled features of the m-th level. Wherein, when m equals 1, the initial feature is used as the input feature of the m-th downsampling module; when m is greater than 1 and less than or equal to M, the downsampling feature of the (m-1)-th level is used as the input feature of the m-th downsampling module.

10. The method of claim 9, wherein, Each of the aforementioned feature refinement feedforward submodules includes a first segmentation unit, a convolution unit, a stitching unit, a first channel transform layer, a second segmentation unit, a deep convolution layer, a first fusion unit, a second channel transform layer, and a first residual connection unit; The step of utilizing N stacked feature refinement feedforward submodules in the m-th downsampling module to perform N levels of feature refinement processing on the input features of the m-th downsampling module to obtain the N-th level of refined features includes: targeting the n-th feature refinement feedforward submodule in the N feature refinement feedforward submodules, where n is a positive integer less than or equal to N. The input features of the nth feature refinement feedforward submodule are divided into multiple sub-input features along the channel dimension using the first segmentation unit in the nth feature refinement feedforward submodule. The convolutional unit in the nth feature refinement feedforward submodule is used to perform partial depth convolution processing on the multiple sub-input features to obtain multiple first sub-convolutional features; The concatenation unit in the nth feature refinement feedforward submodule concatenates multiple first sub-convolutional features along the channel dimension to obtain enhanced features; The enhanced feature is processed by the first channel transform layer in the nth feature refinement feedforward submodule to obtain the first transform feature; The first transformation feature is split into two parts along the channel dimension by using the second segmentation unit in the nth feature refinement feedforward submodule to obtain the first sub-transformation feature and the second sub-transformation feature. The second sub-transform feature is obtained by performing a deep convolution on the second sub-convolution feature using the deep convolutional layer in the nth feature refinement feedforward submodule; The first fusion unit in the nth feature refinement feedforward submodule is used to fuse the first sub-transform feature and the second sub-convolution feature to obtain the first gated feature; The first gated feature is processed by the second channel transformation layer in the nth feature refinement feedforward submodule to obtain the second transformation feature; The second transformed feature and the input feature of the nth feature refinement feedforward submodule are fused using the first residual connection unit in the nth feature refinement feedforward submodule to obtain the refinement feature of the nth level. Specifically, when n equals 1, the input feature of the m-th downsampling module is used as the input feature of the n-th feature refinement feedforward submodule; when n is greater than 1 and less than or equal to N, the refinement feature of the (n-1)-th level is used as the input feature of the n-th feature refinement feedforward submodule.

11. The method according to claim 8, wherein, Each of the upsampling modules includes N stacked adaptive feature enhancement submodules and upsampling submodules, where N is an integer greater than 0; The step of using M upsampling modules in the decoding network to perform M-level upsampling processing on the downsampling features of the Mth level based on M-level downsampling features to obtain the upsampling features of the Mth level includes: for the m-th upsampling module among the M upsampling modules, where m is a positive integer less than or equal to M, The input features of the m-th upsampling module are fused with the downsampling features of the M-m+1-th level to obtain the fused features of the m-th level. By utilizing N stacked adaptive feature enhancement sub-modules in the m-th upsampling module, the fused features of the m-th level are subjected to N-level adaptive feature enhancement processing to obtain the first adaptive enhanced feature of the N-th level. The upsampling submodule in the m-th upsampling module is used to upsample the first adaptive enhancement feature of the N-th level to obtain the upsampled feature of the m-th level. Wherein, when m equals 1, the downsampling feature of the Mth level is used as the input feature of the mth upsampling module; when m is greater than 1 and less than or equal to M, the upsampling feature of the (m-1)th level is used as the input feature of the mth upsampling module.

12. The method according to claim 11, wherein, Each of the adaptive feature enhancement submodules includes an attention submodule and a feature refinement feedforward submodule; The step of using N stacked adaptive feature enhancement submodules in the m-th upsampling module to perform N-level adaptive feature enhancement processing on the fused features of the m-th level to obtain the first adaptive enhanced feature of the N-th level includes: targeting the n-th adaptive feature enhancement submodule among the N adaptive feature enhancement submodules, where n is a positive integer less than or equal to N. The input features of the nth adaptive feature enhancement submodule are processed by the attention submodule in the nth adaptive feature enhancement submodule to obtain the first attention fusion feature; The first attention fusion feature is refined using the feature refinement feedforward submodule in the nth adaptive feature enhancement submodule to obtain the first adaptive enhancement feature at the nth level. Specifically, when n equals 1, the fusion feature of the m-th level is used as the input feature of the n-th adaptive feature enhancement submodule; when n is greater than 1 and less than or equal to N, the first adaptive enhancement feature of the (n-1)-th level is used as the input feature of the n-th adaptive feature enhancement submodule.

13. The method according to claim 12, wherein, The attention submodule includes a first feature mapping layer, a second feature mapping layer, a third feature mapping layer, a Softmax activation layer, a ReLU squared activation layer, a gating unit, a first linear mapping layer, a second linear mapping layer, a third channel transform layer, and a second residual connection unit. The process of using the attention submodule in the nth adaptive feature enhancement submodule to process the input features of the nth adaptive feature enhancement submodule to obtain the first attention fusion feature includes: The input features of the nth adaptive feature enhancement submodule are processed by the first feature mapping layer, the second feature mapping layer, and the third feature mapping layer, respectively, to obtain query features, key features, and value features; The Softmax activation layer is used to generate dense attention weights based on the query features and the key features; The ReLU squared activation layer is used to generate sparse attention weights based on the query features and the key features; The second gating feature is obtained using the gating unit based on the sparse attention weights; The second gated feature is linearly mapped using the first linear mapping layer and the second linear mapping layer respectively to obtain the scaling factor and the offset factor; The dense attention weights are pixel-level modulated according to the scaling factor and the offset factor to obtain adaptive attention weights, and the value features are weighted and fused using the adaptive attention weights to obtain weighted attention features. The weighted attention features are processed using the third channel transformation layer to obtain the third transformation features; The third transformation feature and the input feature of the nth adaptive feature enhancement submodule are fused using the second residual connection unit to obtain the first attention fusion feature.

14. The method according to any one of claims 8 to 13, wherein, The feature refinement network comprises L stacked adaptive feature enhancement modules, where L is an integer greater than 0; The process of using the feature refinement network to process the upsampled features of the Mth layer to obtain deep features includes: The upsampled features of the Mth level are subjected to L-level adaptive feature enhancement processing using L stacked adaptive feature enhancement modules to obtain the second adaptive enhanced features of the Lth level. The second adaptive enhancement feature of the Lth level is determined as the depth feature.

15. The method according to claim 14, wherein, Each of the adaptive feature enhancement modules includes an attention submodule and a feature refinement feedforward submodule; The step of using L stacked adaptive feature enhancement modules to perform L-level adaptive feature enhancement processing on the upsampled features of the Mth level to obtain the second adaptive enhanced feature of the Lth level includes: for the l-th adaptive feature enhancement module among the L adaptive feature enhancement modules, where l is a positive integer less than or equal to L, The attention submodule in the l-th adaptive feature enhancement module is used to process the input features of the l-th adaptive feature enhancement module to obtain the second attention fusion feature; The second attention fusion feature is refined using the feature refinement feedforward submodule in the l-th adaptive feature enhancement module to obtain the second adaptive enhancement feature at the l-th level. Specifically, when l equals 1, the upsampled feature of the Mth level is used as the input feature of the lth adaptive feature enhancement module; when l is greater than 1 and less than or equal to L, the second adaptive enhancement feature of the (l-1)th level is used as the input feature of the lth adaptive feature enhancement module.

16. A remote sensing image processing method, comprising: Acquire an image to be processed, wherein the image to be processed is an optical remote sensing image containing a thin cloud region; The image to be processed is divided into multiple first image blocks, each of which has a first resolution; The cloud removal model is used to process multiple first image blocks to obtain multiple first restored image blocks, and an initial restored image is obtained based on the multiple first restored image blocks. The cloud removal model is trained by the training method of the cloud removal model according to any one of claims 7 to 15. The initial repaired image is divided into multiple second image blocks, each second image block having a second resolution, and the second resolution being different from the first resolution; Multiple second image blocks are resampled to obtain multiple resampled second image blocks, each of which has a first resolution; The cloud removal model is used to remove clouds from multiple resampled second image blocks to obtain multiple second restored image blocks, and a second restored image is obtained based on the multiple second restored image blocks; In response to determining that the second restored image does not contain thin cloud regions, the second restored image is used as a reference restored image, and with the reference restored image as a reference, the initial restored image is subjected to color correction processing based on the wavelet transform algorithm to obtain the target restored image corresponding to the image to be processed.

17. A sample image generation apparatus, comprising: The first acquisition module is used to acquire the original image and the thin cloud image, wherein the original image is a cloudless ground feature image containing multiple bands, and the thin cloud image is a single-band optical remote sensing image containing thin clouds. The extraction module is used to extract basic cloud component data from the thin cloud image; The first generation module is used to correct the cloud basic component data for each band of the original image according to the spectral correction factor corresponding to the band and the compensation coefficient for the thin cloud image, and generate cloud radiation component data corresponding to the band. The second generation module is used to generate a single-band cloud image corresponding to each band of the original image, based on the single-band original image corresponding to the band, the cloud radiation component data corresponding to the band, and the preset atmospheric light value. The combination module is used to combine single-band cloud images corresponding to each band to obtain a sample image corresponding to the original image. The sample image is an optical remote sensing image containing thin clouds.

18. A training device for a cloud-free model, comprising: The second acquisition module is used to acquire a sample image, which is obtained by the sample image generation device according to claim 17; The processing module is used to perform cloud removal processing on the sample image using the initial cloud removal model to obtain the repaired image; The parameter tuning module is used to tune the initial cloud removal model based on the repaired image and the original image corresponding to the sample image to obtain a cloud removal model, which is used to perform thin cloud removal processing on the image to be processed.

19. A remote sensing image processing apparatus, comprising: The third acquisition module is used to acquire the image to be processed, which is an optical remote sensing image containing a thin cloud region. The first segmentation module is used to segment the image to be processed into multiple first image blocks, each of which has a first resolution. The first restoration module is used to perform cloud removal processing on multiple first image blocks using a cloud removal model to obtain multiple first restored image blocks, and to obtain an initial restored image based on the multiple first restored image blocks; wherein, the cloud removal model is trained by the cloud removal model training device according to claim 18. The second segmentation module is used to segment the initial repaired image into multiple second image blocks, each second image block having a second resolution, and the second resolution being different from the first resolution; The resampling module is used to resample multiple second image blocks respectively to obtain multiple resampled second image blocks, each of the resampled second image blocks having a first resolution; The second repair module is used to perform cloud removal processing on multiple resampled second image blocks using the cloud removal model to obtain multiple second repair image blocks, and to obtain a second repaired image based on the multiple second repair image blocks. The correction module is used to, in response to determining that the second restored image does not contain thin cloud regions, take the second restored image as a reference restored image, and, with the reference restored image as a reference, perform color correction processing on the initial restored image based on a wavelet transform algorithm to obtain a target restored image corresponding to the image to be processed.

20. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the sample image generation method according to any one of claims 1 to 6, or the cloud removal model training method according to any one of claims 7 to 15, or the remote sensing image processing method according to claim 16.

21. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the sample image generation method according to any one of claims 1 to 6, or the cloud removal model training method according to any one of claims 7 to 15, or the remote sensing image processing method according to claim 16.

22. A computer program product comprising a computer program that, when executed by a processor, implements the sample image generation method according to any one of claims 1 to 6, or the cloud removal model training method according to any one of claims 7 to 15, or the remote sensing image processing method according to claim 16.

Images