Remote sensing image cloud removal method fusing multi-dimensional attention and physical perception

Abstract

The application discloses a remote sensing image cloud removing method fusing multi-dimensional attention and physical perception, and relates to the technical field of image processing. An initial feature map is subjected to gradually reduced resolution feature learning and channel depth expansion by a feature extraction structure with a multi-dimensional query module and a down-sampling layer staggered arrangement, so that an intermediate feature map is obtained, partial feature learning and adaptive redundancy suppression are realized, the multi-dimensional query mechanism not only reduces the calculation cost, but also enhances the sensitivity to the nonlinear distribution of clouds and fog, and more accurately captures fine-grained details. By using a feature recovery structure with an up-sampling layer, an attention-based cross-scale Transformer module, a physical guiding fusion module and a multi-dimensional query module staggered arrangement, cross-scale attention calculation and physical prior fusion are performed on the intermediate feature map to gradually restore the resolution, the cross-scale detail capturing capability is improved, the physical consistency is ensured, and efficient remote sensing image cloud and fog removal is realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method for cloud removal from remote sensing images that integrates multi-dimensional attention and physical perception. Background Technology

[0002] Cloud removal from remote sensing images is a core task in the field of remote sensing data processing. Its goal is to recover clear, cloud-free original surface information from remote sensing images covered by clouds and fog. Traditional cloud removal methods are mainly based on atmospheric scattering models or statistical models, estimating atmospheric light, transmittance, or cloud composition to achieve cloud removal. For example, the Dark Channel Prior (DCP) method assumes that most local areas in the image contain low-intensity pixels, thus deriving transmittance. With the rapid development of deep learning technology, cloud removal methods based on convolutional neural networks (CNNs) and attention mechanisms have gradually become the mainstream research direction. These methods can automatically learn cloud and fog features and perform end-to-end image restoration, improving the generalization ability of cloud removal and its performance in handling complex scenes.

[0003] Existing methods for cloud removal from remote sensing images suffer from several drawbacks in practical applications. First, traditional physical models and statistical methods are computationally expensive, sensitive to parameter selection, and have limited effectiveness with thick clouds or data lacking specific bands. Second, multimodal fusion and temporal processing methods are computationally complex, dependent on multi-temporal or SAR data availability, and their performance degrades when surface changes are drastic or data is missing. Finally, methods based on deep learning networks and attention mechanisms have high model complexity, demand significant computational resources, and depend on the quality of simulated data. Their generalization ability is limited in real-world thick clouds, complex scenes, non-thin clouds, or out-of-data scenarios, and sampling accuracy may be affected. These drawbacks lead to problems such as halos and artifacts, loss of detail, color distortion, inconsistent restoration results, and weak robustness when processing non-uniform clouds and fog. Summary of the Invention

[0004] The purpose of this application is to provide a remote sensing image cloud removal method that integrates multi-dimensional attention and physical perception, which solves the problems of high computational cost, limited generalization ability, loss of detail and artifact generation in the existing cloud removal process.

[0005] This application is achieved through the following technical solution:

[0006] A cloud removal method for remote sensing images that integrates multi-dimensional attention and physical perception includes:

[0007] Acquire a target remote sensing image and preprocess the target remote sensing image to obtain a preprocessed target remote sensing image; wherein, the target remote sensing image refers to the remote sensing image of the cloud to be removed;

[0008] The preprocessed target remote sensing image is then subjected to patch embedding processing to obtain an initial feature map of fixed dimensions;

[0009] A feature extraction structure with a multi-dimensional query module and a downsampling layer is used to perform feature learning with gradually reduced resolution and channel depth expansion on the initial feature map to obtain an intermediate feature map.

[0010] A feature recovery structure, which alternates between an upsampling layer, an attention-based cross-scale Transformer module, a physically guided fusion module, and a multi-dimensional query module, is used to perform cross-scale attention calculations to gradually restore the resolution of the intermediate feature map and physical prior fusion to obtain the final feature map.

[0011] The final feature map is then subjected to patching and de-embedding operations and soft residual connection operations to obtain the target remote sensing image after cloud removal.

[0012] In one possible implementation, the target remote sensing image is preprocessed to obtain a preprocessed target remote sensing image, including:

[0013] The target remote sensing image is sequentially subjected to channel adjustment, normalization, and resolution unification to obtain a preprocessed target remote sensing image.

[0014] In one possible implementation, the preprocessed target remote sensing image is subjected to patch embedding processing to obtain an initial feature map of fixed dimensions, including:

[0015] Starting with patch embedding, the preprocessed target remote sensing image is divided into 4×4×3 image blocks. These image blocks can be flattened into 48-dimensional vectors, and matrix multiplication is performed using a 48×64 weight matrix to map the 48-dimensional input space to a 64-dimensional space, resulting in an initial feature map of fixed dimensions. The preprocessed target remote sensing image has a dimension of 512×512.

[0016] In one possible implementation, a feature extraction structure with alternating multi-dimensional query modules and downsampling layers is used to perform feature learning with gradually decreasing resolution and channel depth expansion on the initial feature map to obtain an intermediate feature map, including:

[0017] The initial feature map is subjected to initial feature learning using a first multi-dimensional query module, a second multi-dimensional query module, and a third multi-dimensional query module to obtain an initial feature map after initial feature learning.

[0018] The initial feature map after the initial feature learning is downsampled using a first downsampling layer to obtain the initial feature map after the initial downsampling.

[0019] The initial feature map after the initial downsampling is subjected to secondary feature learning using the fourth multi-dimensional query module, the fifth multi-dimensional query module and the sixth multi-dimensional query module to obtain the initial feature map after secondary feature learning.

[0020] A second downsampling layer is used to perform a second downsampling on the initial feature map after the second feature learning to obtain the initial feature map after the second downsampling;

[0021] The initial feature map after the second downsampling is subjected to three feature learning operations using the seventh, eighth, and ninth multi-dimensional query modules to obtain the intermediate feature map.

[0022] In one possible implementation, a feature recovery structure, consisting of an upsampling layer, an attention-based cross-scale Transformer module, a physically guided fusion module, and a multi-dimensional query module, is used to perform cross-scale attention computation and physical prior fusion on the intermediate feature map to gradually restore its resolution, resulting in a final feature map, including:

[0023] The intermediate feature map is initially upsampled using a first upsampling layer to obtain an intermediate feature map after the initial upsampling.

[0024] The intermediate feature map after the initial upsampling is processed by a first cross-scale Transformer module based on attention, a first physical guided fusion module, and a tenth multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the fused intermediate feature map; wherein, the inputs of the first physical guided fusion module are the outputs of the first cross-scale Transformer module and the sixth multi-dimensional query module, respectively.

[0025] A second upsampling layer is used to perform a second upsampling on the fused intermediate feature map to obtain a second upsampled intermediate feature map;

[0026] The intermediate feature map after the second upsampling is processed by a second attention-based cross-scale Transformer module, a second physical guided fusion module, and an eleventh multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the final feature map. The inputs of the second physical guided fusion module are the outputs of the second cross-scale Transformer module and the third multi-dimensional query module, respectively.

[0027] In one possible implementation, the working principle of the first multi-dimensional query module to the eleventh multi-dimensional query module is the same, and all include:

[0028] The first input feature map is grouped by channel into a first sub-feature map, a second sub-feature map, a third sub-feature map, and a fourth sub-feature map; the first input feature map refers to the input feature map of the multi-dimensional query module.

[0029] The first sub-feature map, the second sub-feature map, and the third sub-feature map are respectively processed by query generation to obtain the first query-generated sub-feature map, the second query-generated sub-feature map, and the third query-generated sub-feature map, respectively.

[0030] Multiply the first sub-feature map and the first query-generated sub-feature map using the Hadamard product to obtain the first product feature map; multiply the second sub-feature map and the second query-generated sub-feature map using the Hadamard product to obtain the second product feature map; multiply the third sub-feature map and the third query-generated sub-feature map using the Hadamard product to obtain the third product feature map.

[0031] The first product feature map, the second product feature map, the third product feature map, and the fourth sub-feature map are sequentially concatenated and convolved to obtain the first output feature map.

[0032] In one possible implementation, the first cross-scale Transformer module and the second cross-scale Transformer module operate on the same principle and both include:

[0033] The second input feature map is normalized and convolved to obtain the first cross-scale feature map; wherein, the second input feature map refers to the input feature map of the cross-scale Transformer module;

[0034] The first cross-scale feature is convolved to obtain the second cross-scale feature map;

[0035] The second input feature map is sequentially subjected to pixel inverse reconstruction, convolution processing, normalization processing, and convolution processing to obtain the first reconstructed feature and the second reconstructed feature.

[0036] Based on the first cross-scale feature map, self-attention calculation is performed on the first reconstructed feature and the second reconstructed feature to obtain an attention feature map;

[0037] The attention feature map is added to the second cross-scale feature map element by element to obtain the first added feature map. The first added feature map is then subjected to convolution, GeLU activation and convolution processing to obtain the second output feature map.

[0038] In one possible implementation, the first physical guidance fusion module and the second physical guidance fusion module operate on the same principle and both include:

[0039] The third input feature map is added to the fourth input feature map to obtain the second added feature map; wherein, the third input feature map refers to the input feature map from the cross-scale Transformer module; and the fourth input feature map refers to the input feature map from the multi-dimensional query module.

[0040] The second summed feature map is subjected to global average pooling, multilayer perceptron processing and softmax activation processing to obtain the first activation feature map and the second activation feature map.

[0041] The second additive feature map is processed by depth convolution, multilayer perceptron processing and sigmoid activation to obtain the third and fourth activation feature maps.

[0042] Based on the first, second, third, and fourth excitation feature maps, physical prior fusion is performed to obtain the first and second fusion feature maps.

[0043] The first fused feature map is added to the second fused feature map to obtain the third output feature map.

[0044] In one possible implementation, the final feature map is sequentially subjected to patch de-embedding and soft residual connection operations to obtain a cloud-free target remote sensing image, including:

[0045] Patch inversion uses a learnable linear transformation matrix to map the final feature map back from the high-dimensional feature space to the original image space, resulting in a mapped image.

[0046] The input target remote sensing image is weighted by soft residual connection, and the weighted target remote sensing image is added pixel by pixel to the mapped image to obtain the cloud-free target remote sensing image.

[0047] In one possible implementation, before removing clouds from the preprocessed target remote sensing image, the method further includes: training and evaluating the cloud removal network composed of the feature extraction structure and the feature recovery structure.

[0048] Compared with the prior art, this application has the following advantages and beneficial effects:

[0049] This application provides a remote sensing image cloud removal method that integrates multi-dimensional attention and physical perception. A feature extraction structure with alternating multi-dimensional query modules and downsampling layers is used to gradually reduce the resolution of the initial feature map through feature learning and channel depth expansion, resulting in an intermediate feature map. This achieves partial feature learning and adaptive redundancy suppression. The multi-dimensional query mechanism not only reduces computational overhead but also enhances the model's sensitivity to non-linear cloud and fog distributions, enabling more accurate capture of fine-grained details and avoiding the edge blurring problem common in existing methods. Furthermore, a feature recovery structure with alternating upsampling layers, attention-based cross-scale Transformer modules, physically guided fusion modules, and multi-dimensional query modules is used to gradually restore the resolution of the intermediate feature map through cross-scale attention computation and physical prior fusion. This improves the cross-scale detail capture capability while ensuring physical consistency, achieving efficient cloud and fog removal from remote sensing images. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0051] Figure 1 A flowchart illustrating a remote sensing image cloud removal method that integrates multi-dimensional attention and physical perception, provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the overall structure of the cloud-free network provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the structure of the multi-dimensional query module provided in the embodiments of this application;

[0054] Figure 4 A schematic diagram of the structure of the cross-scale Transformer module provided in the embodiments of this application;

[0055] Figure 5 This is a schematic diagram of the physical guidance fusion module provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0057] like Figure 1As shown in the embodiments of this application, a method for cloud removal from remote sensing images that integrates multi-dimensional attention and physical perception is provided, including:

[0058] S101. Acquire the target remote sensing image and preprocess the target remote sensing image to obtain the preprocessed target remote sensing image; wherein, the target remote sensing image refers to the remote sensing image of the cloud to be removed;

[0059] S102. Perform patch embedding processing on the preprocessed target remote sensing image to obtain an initial feature map with fixed dimensions.

[0060] S103. Using a feature extraction structure that alternates between a multi-dimensional query module and a downsampling layer, the initial feature map is subjected to feature learning with gradually reduced resolution and channel depth expansion to obtain an intermediate feature map.

[0061] S104. A feature recovery structure with an alternating upsampling layer, attention-based cross-scale Transformer module, physical guided fusion module and multi-dimensional query module is used to perform cross-scale attention calculation and physical prior fusion on the intermediate feature map to gradually restore the resolution, so as to obtain the final feature map.

[0062] S105. Perform patching and de-embedding and soft residual connection operations on the final feature map in sequence to obtain the target remote sensing image after cloud removal.

[0063] like Figure 2 As shown, based on the above technical solution, this application provides a cloud-free network based on feature extraction and feature recovery structures. In this cloud-free network, the multi-dimensional query module performs query enhancement and Hadamard product operations on multiple dimensions (width-height, channel-height, and channel-width) through channel partitioning and a dynamic query generator, reducing redundant information; the cross-scale Transformer module uses inverse pixel rearrangement to create RGB proxies as queries and performs attention calculations on the original features using key-value pairs, improving cross-scale detail capture; the physics-guided fusion module fuses encoded and decoded features based on an atmospheric scattering model, ensuring physical consistency and color fidelity; combining downsampling, upsampling, and residual connections ensures lightweight computation while preserving edge details and global context, thus addressing the shortcomings of existing methods. It is worth noting that... Figure 2 In the middle, Patch Embedding is the patch embedding processing module; MDQB is the multi-dimensional query module; CSTB is the cross-scale Transformer module; and PGF is the physical guidance fusion module.

[0064] In one possible implementation, the target remote sensing image is preprocessed to obtain a preprocessed target remote sensing image, including:

[0065] The target remote sensing image is sequentially subjected to channel adjustment, normalization, and resolution unification to obtain a preprocessed target remote sensing image.

[0066] For example, RGB three-band remote sensing images of a specific region can be acquired, with varying degrees of cloud cover. Simultaneously, corresponding cloudless images are collected as ground truth labels for supervising model learning. To adapt to network input requirements, the original images undergo preprocessing, including channel adjustment, normalization, and resolution unification. Cropping operations can also be performed, cropping clouded images and their cloudless labels into blocks of the same pixel size. Finally, the dataset is divided into training, validation, and test sets. The training set is used for parameter optimization, the validation set is used to monitor overfitting, and the test set is used for final performance evaluation.

[0067] In one possible implementation, the preprocessed target remote sensing image is subjected to patch embedding processing to obtain an initial feature map of fixed dimensions, including:

[0068] Starting with patch embedding, the preprocessed target remote sensing image is divided into 4×4×3 image blocks. These image blocks can be flattened into 48-dimensional vectors, and matrix multiplication is performed using a 48×64 weight matrix to map the 48-dimensional input space to a 64-dimensional space, resulting in an initial feature map of fixed dimensions. The preprocessed target remote sensing image has a dimension of 512×512.

[0069] In one possible implementation, a feature extraction structure with alternating multi-dimensional query modules and downsampling layers is used to perform feature learning with gradually decreasing resolution and channel depth expansion on the initial feature map to obtain an intermediate feature map, including:

[0070] The initial feature map is subjected to initial feature learning using a first multi-dimensional query module, a second multi-dimensional query module, and a third multi-dimensional query module to obtain an initial feature map after initial feature learning.

[0071] The initial feature map after the initial feature learning is downsampled using a first downsampling layer to obtain the initial feature map after the initial downsampling.

[0072] The initial feature map after the initial downsampling is subjected to secondary feature learning using the fourth multi-dimensional query module, the fifth multi-dimensional query module and the sixth multi-dimensional query module to obtain the initial feature map after secondary feature learning.

[0073] A second downsampling layer is used to perform a second downsampling on the initial feature map after the second feature learning to obtain the initial feature map after the second downsampling;

[0074] The initial feature map after the second downsampling is subjected to three feature learning operations using the seventh, eighth, and ninth multi-dimensional query modules to obtain the intermediate feature map.

[0075] In this embodiment, the encoder is composed of stacked multidimensional query blocks, divided into three layers (L1, L2, L3) with channel dimensions of 64, 128, and 256, respectively. Each multidimensional query block partitions the input features into four groups along the channel dimension (each group has 1 / 4 of the total channels), and applies a dynamic query generator to generate small-sized query masks in the spatial dimension, channel-height dimension, and channel-width dimension. Subsequently, these query masks are upsampled using bilinear interpolation, and multiplied with the corresponding feature groups by Hadamard products to achieve partial feature learning and redundancy control. Convolutional downsampling connections are used between encoder layers to progressively reduce spatial resolution and expand channel depth. In the bottleneck layer (L3), multidimensional query blocks are further applied to extract high semantic features.

[0076] In one possible implementation, a feature recovery structure, consisting of an upsampling layer, an attention-based cross-scale Transformer module, a physically guided fusion module, and a multi-dimensional query module, is used to perform cross-scale attention computation and physical prior fusion on the intermediate feature map to gradually restore its resolution, resulting in a final feature map, including:

[0077] The intermediate feature map is initially upsampled using a first upsampling layer to obtain an intermediate feature map after the initial upsampling.

[0078] The intermediate feature map after the initial upsampling is processed by a first cross-scale Transformer module based on attention, a first physical guided fusion module, and a tenth multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the fused intermediate feature map; wherein, the inputs of the first physical guided fusion module are the outputs of the first cross-scale Transformer module and the sixth multi-dimensional query module, respectively.

[0079] A second upsampling layer is used to perform a second upsampling on the fused intermediate feature map to obtain a second upsampled intermediate feature map;

[0080] The intermediate feature map after the second upsampling is processed by a second attention-based cross-scale Transformer module, a second physical guided fusion module, and an eleventh multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the final feature map. The inputs of the second physical guided fusion module are the outputs of the second cross-scale Transformer module and the third multi-dimensional query module, respectively.

[0081] In this embodiment, the decoder part uses upsampling to restore resolution and introduces a cross-scale Transformer block and a physically guided fusion module at each level. The cross-scale Transformer block performs attention calculations on the original features by creating RGB proxies as queries and keys, thereby improving the ability to capture cross-scale details. The physically guided fusion module integrates encoder and decoder features based on an atmospheric scattering model, introduces physical constraints into the model, and ensures physical consistency.

[0082] In one possible implementation, the working principle of the first multi-dimensional query module to the eleventh multi-dimensional query module is the same, and all include:

[0083] The first input feature map is grouped by channel into a first sub-feature map, a second sub-feature map, a third sub-feature map, and a fourth sub-feature map; the first input feature map refers to the input feature map of the multi-dimensional query module.

[0084] For example, the query generation process extracts the global context information of the input sub-feature map through global pooling, and maps it to an attention weight map corresponding to the dimension of the sub-feature map through nonlinear transformation. Essentially, it is a parameterized feature-to-weight mapping function. A spatial dimension weight map is generated for the first sub-feature map to focus on important regions, a weight vector along the height direction is generated for the second sub-feature map to capture vertical structural features, and a weight vector along the width direction is generated for the third sub-feature map to capture horizontal structural features. This enables the network to adaptively assign importance scores to features at different locations and channels, thereby achieving data-driven feature selective enhancement. After the query generation process, the first query-generated sub-feature map, the second query-generated sub-feature map, and the third query-generated sub-feature map are obtained respectively.

[0085] The first sub-feature map, the second sub-feature map, and the third sub-feature map are respectively processed by query generation to obtain the first query-generated sub-feature map, the second query-generated sub-feature map, and the third query-generated sub-feature map, respectively.

[0086] Multiply the first sub-feature map and the first query-generated sub-feature map using the Hadamard product to obtain the first product feature map; multiply the second sub-feature map and the second query-generated sub-feature map using the Hadamard product to obtain the second product feature map; multiply the third sub-feature map and the third query-generated sub-feature map using the Hadamard product to obtain the third product feature map.

[0087] The first product feature map, the second product feature map, the third product feature map, and the fourth sub-feature map are sequentially concatenated and convolved to obtain the first output feature map.

[0088] like Figure 3As shown, the multi-dimensional query module achieves partial learning of features and suppression of redundant information by uniformly partitioning the input features along the channel dimension and dynamically generating query masks for each partition in terms of spatial, height, and width channel dimensions. This design not only reduces computational overhead but also enhances the model's ability to capture cloud edges and texture details, avoiding the inefficiency of global computation in traditional attention mechanisms. It is worth noting that... Figure 3 In the diagram, C, H, and W represent the number of channels, feature map height, and feature map width, respectively; X1, X2, X3, and X4 represent the feature subsets obtained after grouping the input feature map X by channels; X1', X2', and X3' represent the output features after weighting each dimension by query; and concat represents the concatenation operation.

[0089] In one possible implementation, the first cross-scale Transformer module and the second cross-scale Transformer module operate on the same principle and both include:

[0090] The second input feature map is normalized and convolved to obtain the first cross-scale feature map; wherein, the second input feature map refers to the input feature map of the cross-scale Transformer module;

[0091] The first cross-scale feature is convolved to obtain the second cross-scale feature map;

[0092] The second input feature map is sequentially subjected to pixel inverse reconstruction, convolution processing, normalization processing, and convolution processing to obtain the first reconstructed feature and the second reconstructed feature. The convolution processing maps the normalized features to the first reconstructed feature Q and the second reconstructed feature K through two independent weight matrices. The parameters of the two weight matrices are independently learned and optimized during training.

[0093] Based on the first cross-scale feature map, self-attention calculation is performed on the first reconstructed feature and the second reconstructed feature to obtain an attention feature map;

[0094] For example, the similarity matrix between channels is obtained by calculating the inner product of the transposes of the first reconstructed feature Q and the second reconstructed feature K through matrix multiplication. This similarity matrix is ​​then adjusted by a learnable scaling parameter and normalized using softmax to obtain the attention weight matrix. Simultaneously, the first cross-scale feature map is linearly transformed and normalized to obtain the value features, which are then flattened into a two-dimensional matrix. The attention weight matrix is ​​then multiplied with the value features to obtain the weighted feature representation, which is then reconstructed back to the original spatial dimension to obtain the attention feature map.

[0095] The attention feature map is added to the second cross-scale feature map element by element to obtain the first added feature map. The first added feature map is then subjected to convolution, GeLU activation and convolution processing to obtain the second output feature map.

[0096] like Figure 4 As shown, the cross-scale Transformer module utilizes pixel downsampling to create proxy images as queries and keys, and integrates multi-scale features through attention computation, improving its adaptability to non-uniform cloud and fog distributions. This module emphasizes cross-level information interaction, effectively capturing the global and local characteristics of clouds and fog, and reducing artifacts and detail loss in the recovered images. It is worth noting that... Figure 4 In the diagram, X represents the input feature map; Pixel Unshuffle is the pixel unshuffle operation; Q and K are the query tensor and key tensor obtained by multiplying the feature vector by two independent weight matrices, respectively; and GeLU is the GeLU activation function.

[0097] In one possible implementation, the first physical guidance fusion module and the second physical guidance fusion module operate on the same principle and both include:

[0098] The third input feature map is added to the fourth input feature map to obtain the second added feature map; wherein, the third input feature map refers to the input feature map from the cross-scale Transformer module; and the fourth input feature map refers to the input feature map from the multi-dimensional query module.

[0099] The second summed feature map is subjected to global average pooling, multilayer perceptron processing and softmax activation processing to obtain the first activation feature map and the second activation feature map.

[0100] The second summed feature map is a single tensor, but it requires input feature maps from two different sources (the third input feature map). and the fourth input feature map Adaptive atmospheric illumination weights are generated separately. By doubling the number of channels in the fully connected layer output, it is equivalent to simultaneously learning two independent sets of weight parameters using a shared set of global context information, and the first half of the output tensor is interpreted as the first activation feature map. (correspond (Atmospheric illumination weights), the latter half of the channel is interpreted as the second excitation feature map. (correspond Atmospheric illumination weight).

[0101] The second summed feature map is processed by depthwise convolution, multilayer perceptron processing, and sigmoid activation to obtain the third and fourth activation feature maps. Similarly, the second summed feature map is processed by depthwise convolution, multilayer perceptron processing, and sigmoid activation, and the first half of the output tensor is interpreted as the third activation feature map. The latter half of the channel is interpreted as the fourth activation feature map. .

[0102] Based on the first, second, third, and fourth excitation feature maps, physical prior fusion is performed to obtain the first and second fusion feature maps.

[0103] The first fused feature map is added to the second fused feature map to obtain the third output feature map.

[0104] like Figure 5 As shown, an adaptive weight adjustment mechanism based on an atmospheric scattering physics model is used to fuse features from the encoder and decoder within a single layer. Through operations such as global average pooling and depthwise separable convolution, the module dynamically balances the feature contributions at different scales, making the fused features more like a "dehazed" representation, thus generating a more robust output. This ensures that the cloud removal process conforms to physical priors, thereby improving color fidelity and structural consistency. It is worth noting that... Figure 4 middle This is an approximation of atmospheric light features for encoder characteristics; This is an approximate representation of atmospheric light features for the decoder. and These are feature approximations of the transmission map; MLP is a fully connected layer consisting of two 1×1 convolutions and a ReLU activation function.

[0105] For example, the fusion process can be represented as: as well as ; It can be used as the first fused feature map. It can be used as the second fusion feature map, which is the Hadamard product.

[0106] During the fusion of physical priors, physical constraints are introduced to ensure physical consistency. These physical constraints are:

[0107]

[0108] in, A clear image indicating no clouds. For foggy images, This is a transmission image. Indicates atmospheric illumination.

[0109] In one possible implementation, the final feature map is sequentially subjected to patch de-embedding and soft residual connection operations to obtain a cloud-free target remote sensing image, including:

[0110] Patch inversion uses a learnable linear transformation matrix to map the final feature map back from the high-dimensional feature space to the original image space, resulting in a mapped image.

[0111] The input target remote sensing image is weighted by soft residual connection, and the weighted target remote sensing image is added pixel by pixel to the mapped image to obtain the cloud-free target remote sensing image.

[0112] For example, the patch de-embedding uses a learnable linear transformation matrix to map the final feature map from the high-dimensional feature space back to the original image space. Specifically, it transforms the feature map of dimension [B, C, H / 4, W / 4] into an RGB image of dimension [B, 3, H, W]. The parameters of the transformation matrix are learned and optimized during training, so that the reconstructed image can approximate the real cloud-free image. The soft residual connection uses a learnable scaling parameter α to weight the input cloud-containing image. The weighted input image is then added pixel by pixel to the cloud-free image output by the patch de-embedding to obtain the final target remote sensing image. α is automatically adjusted during training.

[0113] In one possible implementation, before removing clouds from the preprocessed target remote sensing image, the method further includes: training and evaluating the cloud removal network composed of the feature extraction structure and the feature recovery structure.

[0114] If two images are similar in human visual perception, their representations in the VGG feature space should also be similar. Therefore, to further improve feature-level consistency, a pre-trained VGG16 model is introduced to extract multi-layer features, minimizing the perceptual similarity between the de-clouded image and the real image. The feature loss is calculated as follows:

[0115]

[0116] Where VGGθ represents the feature extractor, X represents the reconstructed image, and Y represents the cloudless label. Furthermore, to address the common color deviation problem in thin cloud removal, a color loss is introduced to enhance the network's reconstruction capability. The specific formula is shown in formula (4):

[0117]

[0118] in, This represents the average value of the red channel. , , This represents the numerical difference between the red, green, and blue channels.

[0119] The final portfolio loss is:

[0120]

[0121] λ1 and λ2 are weight hyperparameters, which are adjusted experimentally to balance the contributions of various losses.

[0122] Model Training: End-to-end training was performed using the training set with the Adam optimizer, an initial learning rate of 0.001, a batch size of 8, and a total of 150 training epochs. During training, performance was evaluated on the validation set every 10 epochs to monitor loss convergence and overfitting risk. If the validation loss increased, the learning rate was dynamically adjusted (with a decay factor of 0.5). Furthermore, data augmentation strategies such as random flipping, rotation, and brightness adjustment were introduced to improve generalization ability. The training platform utilized GPU (Graphics Processing Unit) acceleration to ensure efficient iteration.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0128] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for cloud removal from remote sensing images that integrates multi-dimensional attention and physical perception, characterized in that, include: Acquire a target remote sensing image and preprocess the target remote sensing image to obtain a preprocessed target remote sensing image; wherein, the target remote sensing image refers to the remote sensing image of the cloud to be removed; The preprocessed target remote sensing image is then subjected to patch embedding processing to obtain an initial feature map of fixed dimensions; An intermediate feature map is obtained by using a feature extraction structure that alternates between a multi-dimensional query module and a downsampling layer to perform feature learning with gradually reduced resolution and channel depth expansion on the initial feature map. A feature recovery structure, which alternates between an upsampling layer, an attention-based cross-scale Transformer module, a physically guided fusion module, and a multi-dimensional query module, is used to perform cross-scale attention calculations to gradually restore the resolution of the intermediate feature map and physical prior fusion to obtain the final feature map. The final feature map is then subjected to patching and de-embedding operations and soft residual connection operations in sequence to obtain the target remote sensing image after cloud removal.

2. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 1, characterized in that, The target remote sensing image is preprocessed to obtain a preprocessed target remote sensing image, including: The target remote sensing image is sequentially subjected to channel adjustment, normalization, and resolution unification to obtain a preprocessed target remote sensing image.

3. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 1, characterized in that, The preprocessed target remote sensing image is subjected to patch embedding processing to obtain an initial feature map of fixed dimensions, including: Starting with patch embedding, the preprocessed target remote sensing image is divided into 4×4×3 image blocks. These image blocks can be flattened into 48-dimensional vectors, and matrix multiplication is performed using a 48×64 weight matrix to map the 48-dimensional input space to a 64-dimensional space, resulting in an initial feature map of fixed dimensions. The preprocessed target remote sensing image has a dimension of 512×512.

4. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 1, characterized in that, A feature extraction structure employing a multi-dimensional query module and a downsampling layer is used to progressively reduce the resolution of the initial feature map through feature learning and channel depth expansion, resulting in an intermediate feature map, including: The initial feature map is subjected to initial feature learning using a first multi-dimensional query module, a second multi-dimensional query module, and a third multi-dimensional query module to obtain an initial feature map after initial feature learning. The initial feature map after the initial feature learning is downsampled using a first downsampling layer to obtain the initial feature map after the initial downsampling. The initial feature map after the initial downsampling is subjected to secondary feature learning using the fourth multi-dimensional query module, the fifth multi-dimensional query module and the sixth multi-dimensional query module to obtain the initial feature map after secondary feature learning. A second downsampling layer is used to perform a second downsampling on the initial feature map after the second feature learning to obtain the initial feature map after the second downsampling; The initial feature map after the second downsampling is subjected to three feature learning operations using the seventh, eighth, and ninth multi-dimensional query modules to obtain the intermediate feature map.

5. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 4, characterized in that, A feature recovery structure, employing an alternating configuration of an upsampling layer, an attention-based cross-scale Transformer module, a physically guided fusion module, and a multi-dimensional query module, is used to perform cross-scale attention computation to gradually restore resolution on the intermediate feature map, along with physical prior fusion, to obtain the final feature map, including: The intermediate feature map is initially upsampled using a first upsampling layer to obtain an intermediate feature map after the initial upsampling. The intermediate feature map after the initial upsampling is processed by a first cross-scale Transformer module based on attention, a first physical guided fusion module, and a tenth multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the fused intermediate feature map; wherein, the inputs of the first physical guided fusion module are the outputs of the first cross-scale Transformer module and the sixth multi-dimensional query module, respectively. A second upsampling layer is used to perform a second upsampling on the fused intermediate feature map to obtain a second upsampled intermediate feature map; The intermediate feature map after the second upsampling is processed by a second attention-based cross-scale Transformer module, a second physical guided fusion module, and an eleventh multi-dimensional query module to perform cross-scale attention calculation and physical prior fusion to obtain the final feature map. The inputs of the second physical guided fusion module are the outputs of the second cross-scale Transformer module and the third multi-dimensional query module, respectively.

6. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 5, characterized in that, The working principle of the first multi-dimensional query module to the eleventh multi-dimensional query module is the same, and all include: The first input feature map is grouped by channel into a first sub-feature map, a second sub-feature map, a third sub-feature map, and a fourth sub-feature map; the first input feature map refers to the input feature map of the multi-dimensional query module. The first sub-feature map, the second sub-feature map, and the third sub-feature map are respectively processed by query generation to obtain the first query-generated sub-feature map, the second query-generated sub-feature map, and the third query-generated sub-feature map, respectively. Multiply the first sub-feature map and the first query-generated sub-feature map using the Hadamard product to obtain the first product feature map; multiply the second sub-feature map and the second query-generated sub-feature map using the Hadamard product to obtain the second product feature map; multiply the third sub-feature map and the third query-generated sub-feature map using the Hadamard product to obtain the third product feature map. The first product feature map, the second product feature map, the third product feature map, and the fourth sub-feature map are sequentially concatenated and convolved to obtain the first output feature map.

7. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 5, characterized in that, The first cross-scale Transformer module and the second cross-scale Transformer module operate on the same principle and both include: The second input feature map is normalized and convolved to obtain the first cross-scale feature map; wherein, the second input feature map refers to the input feature map of the cross-scale Transformer module; The first cross-scale feature is convolved to obtain the second cross-scale feature map; The second input feature map is sequentially subjected to pixel inverse reconstruction, convolution processing, normalization processing, and convolution processing to obtain the first reconstructed feature and the second reconstructed feature. Based on the first cross-scale feature map, self-attention calculation is performed on the first reconstructed feature and the second reconstructed feature to obtain an attention feature map; The attention feature map is added to the second cross-scale feature map element by element to obtain the first added feature map. The first added feature map is then subjected to convolution, GeLU activation and convolution processing to obtain the second output feature map.

8. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 5, characterized in that, The first physical guidance fusion module and the second physical guidance fusion module operate on the same principle and both include: The third input feature map is added to the fourth input feature map to obtain the second added feature map; wherein, the third input feature map refers to the input feature map from the cross-scale Transformer module; and the fourth input feature map refers to the input feature map from the multi-dimensional query module. The second summed feature map is subjected to global average pooling, multilayer perceptron processing and softmax activation processing to obtain the first activation feature map and the second activation feature map. The second additive feature map is processed by depth convolution, multilayer perceptron processing and sigmoid activation to obtain the third and fourth activation feature maps. Based on the first, second, third, and fourth excitation feature maps, physical prior fusion is performed to obtain the first and second fusion feature maps. The first fused feature map is added to the second fused feature map to obtain the third output feature map.

9. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 1, characterized in that, The final feature map is sequentially subjected to patch de-embedding and soft residual connection operations to obtain the cloud-removed target remote sensing image, including: Patch inversion uses a learnable linear transformation matrix to map the final feature map back from the high-dimensional feature space to the original image space, resulting in a mapped image. The input target remote sensing image is weighted by soft residual connection, and the weighted target remote sensing image is added pixel by pixel to the mapped image to obtain the cloud-free target remote sensing image.

10. The remote sensing image cloud removal method integrating multi-dimensional attention and physical perception according to claim 1, characterized in that, Before removing clouds from the preprocessed target remote sensing image, the process further includes training and evaluating the cloud removal network composed of the feature extraction structure and the feature recovery structure.

Images