Remote sensing image cloud removing method and system based on attention mechanism and diffusion model

Abstract

The application discloses a remote sensing image cloud removing method and system based on an attention mechanism and a diffusion model, and the method comprises the following steps: acquiring a remote sensing image, and performing gray scale conversion on the remote sensing image to obtain a gray scale remote sensing image; performing weighted processing on the remote sensing image through a high-gray-value attention mechanism and the gray scale remote sensing image to obtain a weighted remote sensing image; inputting the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks; and performing merging processing on all the denoised image blocks to obtain a cloud-removed remote sensing image. It can be seen that, by using the block-based denoising diffusion model combined with the high-gray-value attention mechanism, different attention weights are allocated to regions with different cloud contents in the remote sensing image, so that the problems of inaccurate cloud region positioning and long reasoning time are avoided, and the cloud removing effect of the remote sensing image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a remote sensing image cloud removal method and system based on an attention mechanism and a diffusion model. BACKGROUND

[0002] Remote sensing image information plays a crucial role in earth observation tasks due to its extensive coverage, efficient data acquisition capability, and diverse data types. However, in practical applications, natural factors such as cloud coverage and atmospheric conditions often hinder the effective information extraction of remote sensing images, leading to image data distortion and affecting subsequent analysis and application.

[0003] Traditional remote sensing image cloud removal methods, such as physical model-based cloud removal methods, interpolation-based methods, and deep learning-based methods, have achieved certain results, but still face many challenges. For example, physical model methods are computationally complex and difficult to adapt to complex and variable cloud conditions; interpolation methods are prone to loss of image details, affecting image quality; and deep learning-based methods, such as generative adversarial networks (GANs) and variational autoencoders (VAEs), although perform well, still face issues such as insufficient detail preservation in generated images, large data requirements, and poor training stability when dealing with complex cloud types such as thick clouds and cloud shadows. Therefore, there is a need to solve the defects of existing technologies. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a remote sensing image cloud removal method and system based on an attention mechanism and a diffusion model, which uses a block-based denoising diffusion model combined with a high gray value attention mechanism, assigns different attention weights to regions with different cloud contents in the remote sensing image, avoids inaccurate cloud region positioning and long reasoning time, and improves the remote sensing image cloud removal effect.

[0005] To solve the above technical problems, the present application discloses a remote sensing image cloud removal method based on an attention mechanism and a diffusion model, which comprises:

[0006] Obtain a remote sensing image and perform gray scale conversion on the remote sensing image to obtain a gray scale remote sensing image;

[0007] Perform weighted processing on the remote sensing image through a high gray value attention mechanism and the gray scale remote sensing image to obtain a weighted remote sensing image;

[0008] Input the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks;

[0009] Merge all the denoised image blocks to obtain a cloud-removed remote sensing image.

[0010] As an optional implementation, in the first aspect of the present application, the weighting processing of the remote sensing image by the high gray value attention mechanism and the gray remote sensing image comprises:

[0011] The high gray value region is extracted from the gray remote sensing image by using a learnable gray threshold, and a high gray value region mask is obtained;

[0012] The attention weight matrix is generated by using the high gray value region mask and the remote sensing image;

[0013] The remote sensing image is weighted processed by using the attention weight matrix, and a weighted remote sensing image is obtained.

[0014] As an optional implementation, in the first aspect of the present application, the attention weight matrix is generated by using the high gray value region mask and the remote sensing image comprises:

[0015] The high gray value region mask is multiplied by the remote sensing image to obtain a high gray value feature;

[0016] The high gray value feature and the remote sensing image are respectively subjected to convolution layer and batch normalization layer, then element-wise combined, and subjected to non-linear function mapping by using a first activation function, to obtain a combined feature after non-linear mapping;

[0017] The combined feature after non-linear mapping is subjected to convolution normalization operation, and data is mapped by using a second activation function, to obtain an attention weight matrix.

[0018] As an optional implementation, in the first aspect of the present application, the remote sensing image is weighted processed by using the attention weight matrix, and a weighted remote sensing image is obtained, which comprises:

[0019] The attention weight matrix is multiplied by the remote sensing image to obtain a weighted remote sensing image.

[0020] As an optional implementation, in the first aspect of the present application, the weighting processing of the remote sensing image by the high gray value attention mechanism and the gray remote sensing image comprises:

[0021] An initial noise image is randomly sampled from a standard normal distribution, and the size of the initial noise image is the same as that of the weighted remote sensing image;

[0022] The initial noise image and the weighted remote sensing image are cropped according to a preset position dictionary, to obtain a plurality of noise image blocks and a plurality of degraded image blocks, wherein there is an overlapping area between two adjacent noise image blocks;

[0023] performing noise estimation on the noise image block and the degradation image block by using a conditional diffusion model to obtain estimated noise;

[0024] performing noise mean calculation on the estimated noise and the accumulated weight to obtain average estimated noise;

[0025] updating a plurality of the noise image blocks according to the average estimated noise to obtain a plurality of denoised image blocks.

[0026] As an optional implementation, in the first aspect of the present application, the noise mean calculation on the estimated noise and the accumulated weight to obtain average estimated noise comprises:

[0027] performing element-level division on the estimated noise and the accumulated weight to obtain average estimated noise.

[0028] As an optional implementation, in the first aspect of the present application, the merging processing of all the denoised image blocks to obtain a cloud-removed remote sensing image comprises:

[0029] merging all the denoised image blocks, wherein a weighted average method is used for the estimated noise of the overlapping area to ensure smooth transition between the merged denoised image blocks.

[0030] The second aspect of the present application discloses a remote sensing image cloud removal system based on an attention mechanism and a diffusion model, which comprises:

[0031] a gray image acquisition module, which is configured to acquire a remote sensing image and perform gray conversion on the remote sensing image to obtain a gray remote sensing image;

[0032] a weighted image acquisition module, which is configured to perform weighted processing on the remote sensing image by using a high-gray-value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image;

[0033] a denoising module, which is configured to input the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks;

[0034] a merging module, which is configured to perform merging processing on all the denoised image blocks to obtain a cloud-removed remote sensing image.

[0035] As an optional implementation, in the second aspect of the present application, the weighted image acquisition module performs weighted processing on the remote sensing image by using a high-gray-value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image, which comprises:

[0036] extract a high gray value region from the gray remote sensing image by using a learnable gray threshold to obtain a high gray value region mask;

[0037] generate an attention weight matrix by using the high gray value region mask and the remote sensing image;

[0038] perform weighted processing on the remote sensing image by using the attention weight matrix to obtain a weighted remote sensing image.

[0039] As an optional implementation, in the second aspect of the present application, the weighted image acquisition module generates an attention weight matrix by using the high gray value region mask and the remote sensing image, which includes:

[0040] multiply the high gray value region mask and the remote sensing image to obtain a high gray value feature;

[0041] perform element-wise merging on the high gray value feature and the remote sensing image after passing through a convolution layer and a batch normalization layer respectively, and perform nonlinear function mapping by using a first activation function to obtain a merged feature after nonlinear mapping;

[0042] perform convolution normalization operation on the merged feature after nonlinear mapping, and map the data by using a second activation function to obtain an attention weight matrix.

[0043] As an optional implementation, in the second aspect of the present application, the weighted image acquisition module performs weighted processing on the remote sensing image by using the attention weight matrix to obtain a weighted remote sensing image, which includes:

[0044] multiply the attention weight matrix and the remote sensing image to obtain a weighted remote sensing image.

[0045] As an optional implementation, in the second aspect of the present application, the denoising module inputs the weighted remote sensing image into a diffusion model to perform denoising processing to obtain a plurality of denoised image blocks, which includes:

[0046] randomly sample an initial noise image from a standard normal distribution, the size of the initial noise image being the same as that of the weighted remote sensing image;

[0047] crop the initial noise image and the weighted remote sensing image according to a preset position dictionary to obtain a plurality of noise image blocks and a plurality of degraded image blocks, wherein there is an overlapping area between two adjacent noise image blocks;

[0048] perform noise estimation on the noise image blocks and the degraded image blocks by using a conditional diffusion model to obtain estimated noise;

[0049] According to the estimated noise and the cumulative weight, noise mean value calculation is performed to obtain an average estimated noise;

[0050] According to the average estimated noise, a plurality of the noise image blocks are updated to obtain a plurality of denoised image blocks.

[0051] As an optional implementation, in the second aspect of the present application, the denoising module performs noise mean value calculation according to the estimated noise and the cumulative weight to obtain an average estimated noise, which includes:

[0052] The estimated noise is divided by the cumulative weight at an element level to obtain the average estimated noise.

[0053] As an optional implementation, in the second aspect of the present application, the merging module performs merging processing on all the denoised image blocks to obtain a cloud-removed remote sensing image, which includes:

[0054] All the denoised image blocks are merged, wherein the estimated noise of the overlapping area is obtained by using a weighted average method to ensure smooth transition between the merged denoised image blocks.

[0055] The third aspect of the present application discloses a remote sensing image cloud removal device based on an attention mechanism and a diffusion model, which includes:

[0056] A memory storing executable program codes;

[0057] A processor coupled with the memory;

[0058] The processor invokes the executable program codes stored in the memory to perform part or all steps of the remote sensing image cloud removal method based on the attention mechanism and the diffusion model disclosed in the first aspect of the present application.

[0059] The fourth aspect of the present application discloses a computer storage medium storing computer instructions, which are invoked to perform part or all steps of the remote sensing image cloud removal method based on the attention mechanism and the diffusion model disclosed in the first aspect of the present application.

[0060] Compared with the prior art, the present application has the following beneficial effects:

[0061] By introducing the high gray value attention mechanism, the attention to the high gray value area in the image can be effectively enhanced. By suppressing or filtering out the low gray value area, the model is allowed to automatically identify the thick cloud area, better learn the feature information of the cloud, allocate more attention weight to the thick cloud area in the reconstruction process, and prioritize the reconstruction thereof; secondly, on the basis of the above, combined with the block-based denoising diffusion model, by using the guided denoising process for smooth noise estimation of overlapping image blocks in the inference process, the size-agnostic image restoration is realized, the image reconstruction process is more stable compared with other generation models, and the detail texture and color restoration process is more accurate, effectively solving the problems of weak self-adaptive ability, long inference time and poor thick cloud area restoration effect of the existing remote sensing image cloud removal method. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0063] Figure 1 is a flowchart of a remote sensing image cloud removal method based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application;

[0064] Figure 2 is a structural block diagram of a remote sensing image cloud removal system based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application;

[0065] Figure 3 is a structural diagram of a remote sensing image cloud removal device based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application;

[0066] Figure 4 is a block-based denoising probability diffusion model network structure diagram in the embodiments of the present application;

[0067] Figure 5 is a high gray value attention mechanism network architecture diagram in the embodiments of the present application;

[0068] Figure 6 is a comparison diagram of the cloud removal effect of a remote sensing image cloud removal method based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application on a cloudy image sample 1 and the effect of other latest cloud removal models;

[0069] Figure 7 is a comparison diagram of the cloud removal effect of a remote sensing image cloud removal method based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application on a cloudy image sample 2 and the effect of other latest cloud removal models. DETAILED DESCRIPTION

[0070] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0071] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or end.

[0072] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a separate or alternative embodiment in isolation from other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0073] The application discloses a remote sensing image cloud removal method and system based on an attention mechanism and a diffusion model, which uses a block-based denoising diffusion model combined with a high gray value attention mechanism, allocates different attention weights to different regions with different cloud contents in the remote sensing image, avoids inaccurate cloud region positioning, long reasoning time and other problems, and improves the remote sensing image cloud removal effect. The following will be described in detail.

[0074] Embodiment one

[0075] Please refer to Figure 1 , Figure 1 is a flowchart of a remote sensing image cloud removal method based on an attention mechanism and a diffusion model disclosed by the embodiments of the present application. Among them, Figure 1 The described method is applied to a remote sensing image cloud removal device based on an attention mechanism and a diffusion model. The cloud removal device can be a corresponding cloud removal terminal, cloud removal equipment or server, and the server can be a local server or a cloud server, and the embodiments of the present application are not limited. For example Figure 1As shown, the remote sensing image cloud removal method based on the attention mechanism and diffusion model can include the following operations:

[0076] 101. Obtain a remote sensing image, and perform gray scale conversion on the remote sensing image to obtain a gray scale remote sensing image.

[0077] In the embodiment of the application, the optical remote sensing image I is first converted into a gray scale remote sensing image. For an RGB color image, it contains three channels: red (R), green (G), and blue (B). When converting the color image into a gray scale image, a weighted average formula is used, so that different color channels contribute differently to the gray scale value.

[0078] The calculation formula of the gray scale value is as follows:

[0079] Gray = 0.2989 * R + 0.5870 * G + 0.1140 * B

[0080] Where: R represents the value of the red channel; G represents the value of the green channel; and B represents the value of the blue channel.

[0081] These weights are derived from the perception model of human vision. The human eye has different sensitivities to different colors, with the greatest sensitivity to green, followed by red, and finally blue. Therefore, the weight of the green channel is the largest, followed by the red channel, and the blue channel is the smallest.

[0082] 102. Perform weighted processing on the remote sensing image through a high gray scale value attention mechanism and the gray scale remote sensing image to obtain a weighted remote sensing image.

[0083] 103. Input the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks.

[0084] 104. Merge all the denoised image blocks to obtain a cloud-removed remote sensing image.

[0085] In an optional embodiment, the obtaining of the weighted remote sensing image by performing weighted processing on the remote sensing image through a high gray scale value attention mechanism and the gray scale remote sensing image in step 102 includes:

[0086] extracting a high gray scale value region from the gray scale remote sensing image using a learnable gray scale threshold to obtain a high gray scale value region mask;

[0087] generating an attention weight matrix using the high gray scale value region mask and the remote sensing image;

[0088] performing weighted processing on the remote sensing image using the attention weight matrix to obtain a weighted remote sensing image.

[0089] In an optional implementation, the step of generating an attention weight matrix by using the high gray value region mask and the remote sensing image comprises:

[0090] Multiplying the high gray value region mask and the remote sensing image to obtain a high gray value feature;

[0091] Element-wise merging the high gray value feature and the remote sensing image after passing through a convolution layer and a batch normalization layer respectively, and performing nonlinear function mapping by using a first activation function to obtain a merged feature after nonlinear mapping;

[0092] Performing convolution normalization operation on the merged feature after nonlinear mapping, and mapping data by using a second activation function to obtain an attention weight matrix.

[0093] In an optional implementation, the step of performing weighted processing on the remote sensing image by using the attention weight matrix to obtain a weighted remote sensing image comprises:

[0094] Multiplying the attention weight matrix and the remote sensing image to obtain a weighted remote sensing image.

[0095] In the embodiment of the application, a learnable gray threshold is used to extract a high gray value region from the gray remote sensing image to generate a high gray value region mask M1, which is mainly used for identifying a cloud coverage region. The gray threshold is a learnable parameter, and the initial value is set to 0.5, allowing the model to dynamically adjust the threshold according to different features of the input remote sensing image. This adaptive mechanism enables the attention module to more accurately focus on the cloud region, thereby effectively improving the feature expression capability in the cloud removal process. The high gray value region mask M1 is multiplied with the input remote sensing image I to generate a high gray value feature F t =M1×I; then the input remote sensing image I and the high gray value feature are element-wise merged after passing through a convolution layer (W1) and a batch normalization layer (W2) respectively, and then nonlinear function mapping is performed by using a first activation function to obtain a merged feature after nonlinear mapping. It should be noted that ReLU activation function is used as the first activation function here; convolution normalization operation (W3) is performed on the merged feature after nonlinear mapping, and data is mapped to the interval (0, 1) through a second activation function, thereby obtaining an attention weight matrix HGA(I). It should be noted that Sigmoid activation function is used as the second activation function here, and the expression of HGA(I) is as follows:

[0096] HGA(I)=σ Sigmoid (W2(σ Relu (W0(I)+W1(F t ))))

[0097] Finally, the HGA(I) is multiplied by the input remote sensing image to obtain a weighted remote sensing image F', and the expression of F' is as follows:

[0098] F' = I x HGA(I)

[0099] It can be seen that by using the learnable gray threshold to extract the high gray value region from the gray remote sensing image, a high gray value region mask is generated. This mask can accurately reflect the position and range of the cloud layer in the image, providing key information for subsequent generation of attention weight matrix. Then, the attention weight matrix is generated using the high gray value region mask and the remote sensing image, and the remote sensing image is weighted. This weighting process can enhance the feature representation of the high gray value region in the image, so that the model can more accurately identify and process the cloud layer in the subsequent denoising process, improving the accuracy and efficiency of cloud removal. At the same time, since a convolutional neural network is used for feature extraction, this method also has certain adaptive ability and robustness, and can cope with remote sensing image cloud removal tasks under different scenes and conditions. Therefore, this step improves the accuracy and efficiency of cloud removal, laying a foundation for generating high-quality cloud-free remote sensing images.

[0100] In an optional implementation, the step of inputting the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks in step 103 comprises:

[0101] Randomly sampling an initial noise image from a standard normal distribution, the size of the initial noise image being the same as the size of the weighted remote sensing image;

[0102] According to a preset position dictionary, the initial noise image and the weighted remote sensing image are cropped to obtain a plurality of noise image blocks and a plurality of degraded image blocks, wherein there is an overlapping area between two adjacent noise image blocks;

[0103] Using a conditional diffusion model to perform noise estimation on the noise image blocks and the degraded image blocks to obtain estimated noise;

[0104] According to the estimated noise and the cumulative weight, noise mean value calculation is performed to obtain average estimated noise;

[0105] According to the average estimated noise, a plurality of noise image blocks are updated to obtain a plurality of denoised image blocks.

[0106] In an optional implementation, the step of performing noise mean value calculation according to the estimated noise and the cumulative weight to obtain average estimated noise in the above step comprises:

[0107] Element-level division is performed on the estimated noise and the cumulative weight to obtain average estimated noise.

[0108] In the embodiment of the present application, initial noise sampling is first performed to generate an initial noise image X0from a standard normal distribution t , and to initialize the accumulated estimated noise , and to initialize the accumulated weight M=0, according to a preset dictionary containing D overlapping image block positions, using the block position Pd d , the current noise image Xd t , and the weighted remote sensing image Yd are respectively cropped to obtain the noise image block Xd and the degraded image block Yd , where Pd d is a binary mask matrix of the same dimension as X0and Y0, representing the p x p block position in the image, and d e D.

[0109] A time step t is randomly sampled from a uniform distribution set {1, 2,..., T} to determine a specific stage of the model in the diffusion process, and step-by-step sampling is performed according to a preset number of implicit sampling steps S, and the following operations are performed for each sampling step i until i = 1 to obtain a plurality of denoised image blocks:

[0110] The time step t is calculated according to the current step i, and the calculation expression is: t = (i-1)·T / S+1, if i > 1, the next time step t next is calculated, and the calculation expression is: t next = (i-2)·T / S+1, if i = 1, t next = 0;

[0111] The estimated noise is updated and the accumulated weight is estimated for each image block position d using the conditional diffusion model, and the update expression of the estimated noise is:

[0112] The estimated noise and the accumulated weight are element-wise divided to obtain the average estimated noise of the block, and the calculation expression is: where, represents element-wise division;

[0113] The noise image block is updated by the following formula:

[0114] wherein the training process of the diffusion model includes the following steps:

[0115] In the training process, an unknown cloud-free image of any size is defined as X0e R C×H×W , and a cloudy observation image is defined as Y0 is element-wise, Pd i is the same as X0and Y0 ​​A binary mask matrix of the same dimension as the image, indicating the position of the i-th p x p block in the image.

[0116] Training step S1, input a pair of images, including a cloud-occluded image and a clear image without cloud, where X0 is the clear image. The image is affected by cloud cover.

[0117] Training step S2, randomly sample a binary mask P i :

[0118] Randomly generate a binary mask P i for selecting part of the image area.

[0119] Training step S3, randomly crop the image area:

[0120] Use the binary mask P i to select part of the image area for cropping, which includes the following steps: for the clear image X0, use the mask to crop to get the cropped image block That is For the weather-affected image Similarly, use the mask to crop to get the cropped degraded image block

[0121] Training step S4, time step sampling t:

[0122] Randomly sample a time step t from the uniform distribution set {1, 2,..., T}, which is used to determine a specific stage of the diffusion process.

[0123] Training step S5, noise sampling t :

[0124] Sample a noise t from a normal distribution , which is used to add randomness to the model input so that the model can process and remove noise.

[0125] Training step S6, gradient descent optimization:

[0126] Perform a gradient descent step to optimize the model parameters θ to minimize the following objective function:

[0127]

[0128] The objective function is used to measure the error between the model output and the image with added noise, and the goal is to make the model learn to restore clear visual effects in different levels of degraded images.

[0129] The training step S7, the training steps S2 to S6 are repeated until the loss function of the model reaches the preset convergence threshold value. Finally, the trained model parameters θ are output, and the model training is completed.

[0130] In an optional embodiment, the merging of all the denoised image blocks in step 104 to obtain a cloud-removed remote sensing image comprises:

[0131] The denoised image blocks are merged, wherein the estimated noise of the overlapping region is averaged by using a weighted average method to ensure smooth transition between the merged denoised image blocks.

[0132] In the embodiment of the application, for the overlapping region, the average update formula of the estimated noise is:

[0133]

[0134] wherein, represents the average estimated noise of the overlapping pixel region. d represents different sampling directions or blocks of the overlapping region (for example, 1 to 4 represent four directions). represents the noise estimation of the model in the dth overlapping region at the time step t, wherein: represents the current noise image block of the dth direction or sampling block; represents the corresponding degraded image block; t represents the current time step, which controls the stage of the model in the diffusion process.

[0135] It can be seen that, by introducing the high gray value attention mechanism, the attention to the high gray value region in the image can be effectively enhanced. By suppressing or filtering out the low gray value region, the model is allowed to automatically identify the thick cloud region, better learn the feature information of the cloud, allocate more attention weight to the thick cloud region in the reconstruction process, and prioritize the reconstruction of the thick cloud region; secondly, on the basis of the above, combined with the block-based denoising diffusion model, by using the guided denoising process of smooth noise estimation of overlapping image blocks in the inference process, the size-agnostic image restoration is realized, the image reconstruction process is more stable compared with other generation models, and the detail texture and color restoration process is more accurate, effectively solving the problems of weak adaptability, long inference time and poor recovery effect of thick cloud region in the prior art.

[0136] Embodiment two

[0137] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of a remote sensing image cloud removal device based on an attention mechanism and a diffusion model according to an embodiment of the application. Wherein, Figure 3The described apparatus can be applied to a corresponding cloud removal terminal, cloud removal device or server, and the server can be a local server or a cloud server, and the embodiments of the present application are not limited. As shown in Figure 3 The apparatus can include:

[0138] The gray image acquisition module 201 is configured to acquire a remote sensing image and perform gray conversion on the remote sensing image to obtain a gray remote sensing image.

[0139] In the embodiments of the present application, the optical remote sensing image I is first converted into a gray remote sensing image. For an RGB color image, it contains three channels: red (R), green (G) and blue (B). When converting the color image into a gray image, a weighted average formula is used, so that different color channels contribute differently to the gray value.

[0140] The calculation formula of the gray value is as follows:

[0141] Gray = 0.2989 * R + 0.5870 * G + 0.1140 * B

[0142] Wherein: R represents the value of the red channel; G represents the value of the green channel; and B represents the value of the blue channel.

[0143] These weights are derived from the perception model of human vision. The human eye has different sensitivities to different colors, with the greatest sensitivity to green, followed by red, and finally blue. Therefore, the weight of the green channel is the largest, followed by the red channel, and the blue channel is the smallest.

[0144] The weighted image acquisition module 202 is configured to perform weighted processing on the remote sensing image by a high gray value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image.

[0145] The denoising module 203 is configured to input the weighted remote sensing image into a diffusion model for denoising processing to obtain a plurality of denoised image blocks.

[0146] The merging module 204 is configured to merge all the denoised image blocks to obtain a cloud-removed remote sensing image.

[0147] In an optional embodiment, the weighted image acquisition module 202 performs weighted processing on the remote sensing image by a high gray value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image, including:

[0148] A learnable gray threshold is used to extract a high gray value region from the gray remote sensing image to obtain a high gray value region mask;

[0149] An attention weight matrix is generated using the high gray value region mask and the remote sensing image.

[0150] The remote sensing image is weighted using the attention weight matrix to obtain a weighted remote sensing image.

[0151] In an optional implementation, the weighted image acquisition module 202 generates an attention weight matrix using the high gray value region mask and the remote sensing image, including:

[0152] The high gray value region mask is multiplied by the remote sensing image to obtain a high gray value feature.

[0153] The high gray value feature and the remote sensing image are respectively subjected to convolution layers and batch normalization layers, then element-wise combined, and subjected to non-linear function mapping using a first activation function to obtain a combined feature after non-linear mapping.

[0154] The combined feature after non-linear mapping is subjected to convolution normalization operation, and data is mapped using a second activation function to obtain an attention weight matrix.

[0155] In an optional implementation, the weighted image acquisition module 202 uses the attention weight matrix to perform weighted processing on the remote sensing image to obtain a weighted remote sensing image, including:

[0156] The attention weight matrix is multiplied by the remote sensing image to obtain a weighted remote sensing image.

[0157] In the embodiments of the present application, a learnable gray threshold is used to extract a high gray value region from the gray remote sensing image to generate a high gray value region mask M1, which is mainly used to identify a cloud coverage region. The gray threshold is a learnable parameter, and the initial value is set to 0.5, allowing the model to dynamically adjust the threshold according to different features of the input remote sensing image. This adaptive mechanism enables the attention module to more accurately focus on the cloud region, thereby effectively improving the feature expression capability in the cloud removal process. The high gray value region mask M1 is multiplied by the input remote sensing image I to generate a high gray value feature F tM1xI; then the input remote sensing image I and the high gray value feature are respectively subjected to convolution layer (W1) and batch normalization layer (W2), and then are subjected to element-wise merging, and then a first activation function is used for non-linear function mapping to obtain the merged feature after non-linear mapping, and it should be noted that ReLU activation function is used as the first activation function; the merged feature after non-linear mapping is subjected to convolution normalization operation (W3), and a second activation function is used to map the data to the interval (0, 1), so as to obtain the attention weight matrix HGA(I), and it should be noted that Sigmoid activation function is used as the second activation function, and the expression of HGA(I) is as follows:

[0158] HGA(I) = σ Sigmoid (W2(σ Relu (W0(I)+W1(F t ))))

[0159] Finally, HGA(I) is multiplied by the input remote sensing image to obtain the weighted remote sensing image F', and the expression of F' is as follows:

[0160] F' = I x HGA(I)

[0161] It can be seen that by using the learnable gray threshold to extract the high gray value region from the gray remote sensing image, a high gray value region mask is generated. This mask can accurately reflect the position and range of the cloud layer in the image, providing key information for generating the attention weight matrix. Then, the high gray value region mask and the remote sensing image are used to generate the attention weight matrix, and the remote sensing image is weighted. This weighting processing can enhance the feature representation of the high gray value region in the image, so that the model can more accurately identify and process the cloud layer in the subsequent denoising process, improving the accuracy and efficiency of cloud removal. At the same time, since convolutional neural network is used for feature extraction, this method also has certain adaptive ability and robustness, and can cope with remote sensing image cloud removal tasks under different scenes and conditions. Therefore, this step improves the accuracy and efficiency of cloud removal, and lays a foundation for generating high-quality cloud-free remote sensing images.

[0162] In an optional embodiment, the denoising module 203 inputs the weighted remote sensing image into a diffusion model for denoising to obtain a plurality of denoised image blocks, including:

[0163] An initial noise image is randomly sampled from a standard normal distribution, and the size of the initial noise image is the same as that of the weighted remote sensing image;

[0164] The initial noisy image and the weighted remote sensing image are cropped according to a preset location dictionary to obtain multiple noisy image blocks and multiple degraded image blocks, wherein there is an overlapping area between two adjacent noisy image blocks;

[0165] The noise is estimated by using a conditional diffusion model to evaluate the noise in the noisy image patch and the degraded image patch.

[0166] The average estimated noise is obtained by calculating the noise mean based on the estimated noise and the accumulated weights.

[0167] The multiple noisy image blocks are updated based on the average estimated noise to obtain multiple denoised image blocks.

[0168] In an optional implementation, the denoising module 203 calculates the noise mean based on the estimated noise and the accumulated weights to obtain the average estimated noise, including:

[0169] The estimated noise is divided element-wise by the accumulated weights to obtain the average estimated noise.

[0170] In this embodiment of the invention, initial noise sampling is first performed, starting from a standard normal distribution. The initial noisy image X is generated by random sampling. t and initialize the accumulated estimated noise. And initialize the accumulated weight M=0, and use the block position P according to the preset dictionary containing the positions of D overlapping image blocks. d For the current noisy image X respectively t and weighted remote sensing images Crop the image to obtain noisy image patches. and degraded image patches Where P d For with X0 and A binary mask matrix of the same dimension represents the position of the d-th p×p block in the image, where d∈D.

[0171] A time step t is randomly sampled from a uniformly distributed set {1,2,...,T} to determine a specific stage in the diffusion process of the model. Sampling is then performed progressively according to a preset number of implicit sampling steps S. For each sampling step i, the following operation is performed until i=1, to obtain multiple denoised image patches:

[0172] Calculate the time step t based on the current step i. The calculation expression is: t = (i-1)·T / S + 1. If i > 1, then calculate the next time step t. next The calculation expression is: t next = (i-2)·T / S+1, if i=1, then t next =0;

[0173] The estimated noise is updated and accumulated weight for each image block position d using the conditional diffusion model, and the update expression of the estimated noise is:

[0174] The estimated noise is divided by the accumulated weight at the element level to obtain the average estimated noise of the block, and the calculation expression is: wherein, represents element division;

[0175] The noise image block is updated by the following formula:

[0176] In an optional embodiment, the merging module 204 merges all the denoised image blocks to obtain a cloud-removed remote sensing image, including:

[0177] All the denoised image blocks are merged, wherein the estimated noise of the overlapping region is averaged by the weighted average method to ensure the smooth transition between the merged denoised image blocks.

[0178] In the embodiment of the application, for the overlapping region, the average update formula of the estimated noise is:

[0179]

[0180] wherein, represents the average estimated noise of the overlapping pixel region. d represents different sampling directions or blocks of the overlapping region (for example, 1 to 4 represent four directions). represents the noise estimation of the d-th overlapping region by the model at time step t, wherein: represents the current noise image block of the d-th direction or sampling block; represents the corresponding degraded image block; t represents the current time step, which controls the stage of the model in the diffusion process.

[0181] It can be seen that the introduction of the high gray value attention mechanism can effectively enhance the attention to the high gray value region in the image. By suppressing or filtering out the low gray value region, the model can automatically identify the thick cloud region, better learn the feature information of the cloud, allocate more attention weight to the thick cloud region in the reconstruction process, and prioritize the reconstruction thereof; secondly, on the basis of the above, combined with the block-based denoising diffusion model, by using the guided denoising process of smooth noise estimation of overlapping image blocks in the inference process, the size-agnostic image restoration is realized, the image reconstruction process is more stable compared to other generation models, and the detail texture and color restoration process is more accurate, effectively solving the problems of weak adaptability, long inference time and poor recovery effect of thick cloud region in the prior art.

[0182] Embodiment three

[0183] Please refer to Figure 3 , Figure 3 is another structure diagram of a remote sensing image cloud removal device based on an attention mechanism and a diffusion model according to an embodiment of the present application. As shown in the figure, the device can include: Figure 3

[0184] a memory 301 storing executable program codes;

[0185] a processor 302 coupled with the memory 301;

[0186] The processor 302 invokes the executable program codes stored in the memory 301 to execute part or all steps of the remote sensing image cloud removal method based on the attention mechanism and the diffusion model according to the embodiment one of the present application.

[0187] Embodiment four

[0188] The embodiment of the present application discloses a computer storage medium, which stores computer instructions. When the computer instructions are invoked, part or all steps of the remote sensing image cloud removal method based on the attention mechanism and the diffusion model according to the embodiment one of the present application are executed.

[0189] The device embodiments described above are only schematic, wherein the modules illustrated as separate components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, that is, they can be located in one place, or distributed on multiple network modules. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0190] ​Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform through the specific description of the above embodiments, and of course, the various embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other computer readable medium that can be used to carry or store data.

[0191] Finally, it should be noted that: the remote sensing image cloud removal method and system based on the attention mechanism and diffusion model disclosed in the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that; the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for cloud removal of remote sensing images based on an attention mechanism and a diffusion model, characterized in that, The method comprises: acquiring a remote sensing image, and performing grayscale conversion on the remote sensing image to obtain a grayscale remote sensing image; performing weighted processing on the remote sensing image through a high-gray-value attention mechanism and the grayscale remote sensing image to obtain a weighted remote sensing image; inputting the weighted remote sensing image into a diffusion model to perform denoising processing to obtain a plurality of denoised image blocks; performing merging processing on all the denoised image blocks to obtain a cloud-removed remote sensing image; the performing weighted processing on the remote sensing image through the high-gray-value attention mechanism and the grayscale remote sensing image to obtain the weighted remote sensing image comprises: extracting a high-gray-value region from the grayscale remote sensing image using a learnable grayscale threshold to obtain a high-gray-value region mask; generating an attention weight matrix using the high-gray-value region mask and the remote sensing image; performing weighted processing on the remote sensing image using the attention weight matrix to obtain a weighted remote sensing image; the generating the attention weight matrix using the high-gray-value region mask and the remote sensing image comprises: multiplying the high-gray-value region mask and the remote sensing image to obtain a high-gray-value feature; performing element-wise merging on the high-gray-value feature and the remote sensing image after passing through a convolution layer and a batch normalization layer respectively, and performing nonlinear function mapping using a first activation function to obtain a merged feature after nonlinear mapping; performing convolution normalization operation on the merged feature after nonlinear mapping, and mapping data using a second activation function to obtain an attention weight matrix.

2. The method of claim 1, wherein, the performing weighted processing on the remote sensing image using the attention weight matrix to obtain the weighted remote sensing image comprises: multiplying the attention weight matrix and the remote sensing image to obtain a weighted remote sensing image.

3. The method of claim 1, wherein, the inputting the weighted remote sensing image into the diffusion model to perform denoising processing to obtain a plurality of denoised image blocks comprises: randomly sampling an initial noise image from a standard normal distribution, the initial noise image having the same size as the weighted remote sensing image; cropping the initial noise image and the weighted remote sensing image according to a preset position dictionary to obtain a plurality of noise image blocks and a plurality of degraded image blocks, wherein there is an overlapping area between two adjacent noise image blocks; performing noise estimation on the noise image blocks and the degraded image blocks using a conditional diffusion model to obtain estimated noise; performing noise mean value calculation according to the estimated noise and an accumulated weight to obtain average estimated noise; updating a plurality of the noise image blocks according to the average estimated noise to obtain a plurality of denoised image blocks.

4. The method of claim 3, wherein, the performing noise mean value calculation according to the estimated noise and the accumulated weight to obtain the average estimated noise comprises: performing element-level division on the estimated noise and the accumulated weight to obtain the average estimated noise.

5. The method of claim 3, wherein the method further comprises: the performing merging processing on all the denoised image blocks to obtain the cloud-removed remote sensing image comprises: merging all the denoised image blocks, wherein a weighted average method is used for the estimated noise of the overlapping area to ensure smooth transition between the merged denoised image blocks.

6. A remote sensing image cloud removal system based on an attention mechanism and a diffusion model, characterized in that, The system comprises: The gray image acquisition module is configured to acquire a remote sensing image and perform gray conversion on the remote sensing image to obtain a gray remote sensing image. The weighted image acquisition module is configured to perform weighted processing on the remote sensing image by using a high-gray-value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image. The denoising module is configured to input the weighted remote sensing image into a diffusion model to perform denoising processing to obtain a plurality of denoised image blocks. The merging module is configured to perform merging processing on all the denoised image blocks to obtain a cloud-removed remote sensing image. The weighted image acquisition module is configured to perform weighted processing on the remote sensing image by using a high-gray-value attention mechanism and the gray remote sensing image to obtain a weighted remote sensing image. The high-gray-value region mask is obtained by extracting a high-gray-value region from the gray remote sensing image using a learnable gray threshold. The attention weight matrix is generated by using the high-gray-value region mask and the remote sensing image. The remote sensing image is weighted processed using the attention weight matrix to obtain a weighted remote sensing image. The attention weight matrix is generated by using the high-gray-value region mask and the remote sensing image. The high-gray-value region mask and the remote sensing image are multiplied to obtain a high-gray-value feature. The high-gray-value feature and the remote sensing image are respectively subjected to convolution layer and batch normalization layer, then element-wise merged, and subjected to non-linear function mapping using a first activation function to obtain a merged feature after non-linear mapping. The merged feature after non-linear mapping is subjected to convolution normalization operation and data mapping using a second activation function to obtain an attention weight matrix.

7. A remote sensing image cloud removal device based on an attention mechanism and a diffusion model, characterized in that, The device comprises: a memory storing executable program codes; a processor coupled with the memory; the processor invokes the executable program codes stored in the memory to execute the remote sensing image cloud removal method based on the attention mechanism and diffusion model according to any one of claims 1-5.

8. A computer storage medium, characterized in that The computer storage medium stores computer instructions which, when invoked, are used to execute the remote sensing image cloud removal method based on the attention mechanism and diffusion model according to any one of claims 1-5.

Images