A remote sensing image defogging method based on improved DehazeFormer

Abstract

The application provides a remote sensing image defogging method based on an improved DehazeFormer, and solves the technical problems of incomplete structure and poor fineness of an image after defogging by an existing defogging model. The method comprises the following steps: obtaining an original foggy image, constructing an improved DehazeFormer model, defogging the original foggy image based on the improved DehazeFormer model, and obtaining a final defogged image. The application can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and more specifically, relates to a remote sensing image dehazing method based on an improved DehazeFormer. Background Technology

[0002] As the main body of Earth's life system and a key resource treasure trove, the dynamic monitoring and accurate understanding of the ocean are crucial for the sustainable development of human society. Satellite and airborne remote sensing, with their advantages of wide coverage and short revisit cycles, have become core technologies in fields such as marine environmental monitoring. However, the complex and ever-changing atmospheric environment over the ocean, especially sea fog, poses a significant obstacle to obtaining high-quality marine remote sensing images. Sea fog severely degrades the visual quality of remote sensing images, significantly weakening key information about ground targets and making the overall image appear blurry.

[0003] The existing Chinese invention patent, CN117934326A, proposes a Transformer-based single-image dehazing method and an aerial fixed-wing UAV. It utilizes the SCU DehazeFormer network model to avoid negative impacts that are unimportant to high-level vision tasks but crucial for low-level vision tasks, thus achieving dehazing of remote sensing images. However, the SCU DehazeFormer network model in this patent cannot distinguish the importance of channels and often loses subtle edge textures during segmentation. This results in large numerical errors between the dehazed image and the actual fog-free scene, severe loss of detail in the restored image, and poor clarity. Summary of the Invention

[0004] The purpose of this application is to provide a remote sensing image dehazing method based on an improved DehazeFormer, so as to solve the technical problems of incomplete image structure and poor precision after dehazing in the prior art.

[0005] To achieve the above objectives, this application provides a remote sensing image dehazing method based on an improved DehazeFormer, comprising the following steps:

[0006] Obtain the original foggy image, construct an improved DehazeFormer model, and dehaze the original foggy image based on the improved DehazeFormer model to obtain the final dehazed image.

[0007] The improved DehazeFormer model introduces the CASaE and BiFormer modules;

[0008] The CASaE module includes a coordinate attention mechanism, residual connection layers, a convolutional layer sequence module, and a SaE module, which are used to restore local details of the original hazy image and reduce chromatic aberration blocks and false edges.

[0009] Preferably, the BiFormer module includes depthwise separable convolutional layers, residual connection layers, layer normalization layers, a two-layer routing attention mechanism, and a multilayer perceptron, which are used to balance the global perception and computational efficiency of the improved DehazeFormer model.

[0010] Preferably, the convolutional layer sequence module includes a two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer, used to extract local features of fog, normalize the distribution of local features, suppress negative features in local features, retain positive features in local features, and introduce nonlinearity.

[0011] Preferably, the improved DehazeFormer model sequentially includes a convolutional layer, a DehazeFormer module, a first downsampling layer, a DehazeFormer module, a second downsampling layer, a DehazeFormer module, a BiFormer module, a first upsampling layer, an SK fusion layer, a CASaE module, a DehazeFormer module, a second upsampling layer, a CASaE module, an SK fusion layer, a DehazeFormer module, a convolutional layer, and a soft reconstruction layer.

[0012] Preferably, the original foggy image is processed through a convolutional layer, a DehazeFormer module, a first downsampling layer, a DehazeFormer module, a second downsampling layer, and another DehazeFormer module to obtain a processed second feature map. This processed second feature map is then input into a BiFormer module for feature extraction and attention processing to obtain an integrated second feature map. This integrated second feature map is then sequentially input into a first upsampling layer and an SK fusion layer for upsampling and feature fusion to obtain a fused third feature map. This fused third feature map is then input into a CASaE module to obtain a processed third feature map. Finally, the processed third feature map is input into a DehazeFormer module, a second upsampling layer, a CASaE module, an SK fusion layer, a DehazeFormer module, a convolutional layer, and a soft reconstruction layer to obtain the final dehazed image.

[0013] Preferably, the process of obtaining the processed third feature map includes:

[0014] The fused third feature map is input into the coordinate attention mechanism to obtain the enhanced third feature map. It is then connected with the fused third feature map via residual connection to obtain the third feature map after secondary connection. After convolution operation, the activated third feature map is obtained by inputting it into the convolutional layer sequence. After residual connection, the third feature map after tertiary connection is obtained. This third feature map is then input into the SaE module and processed through a multi-branch fully connected layer to obtain the processed third feature map.

[0015] Preferably, the process of obtaining the enhanced third feature map includes:

[0016] The fused third feature map is residually connected to obtain a residually connected third feature map. Average pooling is then performed along the horizontal and vertical directions to obtain horizontal and vertical feature maps. The dependencies in the horizontal and vertical directions are concatenated along the spatial dimension and then fed into a two-dimensional convolutional layer for convolution. The convolution is then normalized using the sigmoid activation function to obtain attention maps in the horizontal and vertical directions. These attention maps are then multiplied element-wise with the residually connected third feature map to obtain the enhanced third feature map.

[0017] Preferably, the process of obtaining the activated third feature map includes:

[0018] The third feature map after residual connection is input into a two-dimensional convolutional layer for convolution operation to obtain the convolutional third feature map. Normalization operation is then performed to obtain the normalized third feature map. The ReLU activation function is used to suppress the negative features of the normalized third feature map and improve the feature representation ability to obtain the activated third feature map.

[0019] Preferably, the process of inputting the third feature map after three connections into the SaE module to obtain the processed third feature map includes:

[0020] The third feature map after three connections is reduced in dimensionality to obtain the third feature map after dimensionality reduction. The key features for effective dehazing are selected by using the ReLU activation function. The results of the multi-branch fully connected layer are concatenated to restore the number of channels and generate channel weights. The inter-channel dependencies are captured to obtain the processed third feature map.

[0021] Preferably, the process of obtaining the integrated second feature map includes: inputting the processed second feature map into a depthwise separable convolutional layer for depthwise convolution and pointwise convolution to obtain a depthwise convolutional second feature map; performing residual connection with the processed second feature map to obtain a residually connected second feature map; performing normalization processing in the input layer normalization layer to obtain a normalized second feature map; calculating the attention weights of the normalized second feature map through a two-layer routing attention mechanism; performing residual connection with the residually connected second feature map to obtain a second-connected second feature map; performing normalization again in the input layer normalization layer to obtain a second-normalized second feature map; and processing using a multilayer perceptron to obtain the integrated second feature map.

[0022] The beneficial effects of this application are as follows: This application provides a remote sensing image dehazing method based on an improved DehazeFormer. By introducing the CASaE module and the BiFormer module into the DehazeFormer model, the integrity and refinement of the image structure after dehazing are improved. Specifically, the introduced CASaE module can effectively repair local details of the original hazy image and reduce color difference blocks and false edges; the introduced BiFormer module can balance the global perception and computational efficiency of the improved DehazeFormer model, making the numerical error between the final dehazed image and the real fog-free scene smaller, the restoration result more accurate, and the structure of the final dehazed image more complete. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram of the overall process of a remote sensing image dehazing method based on an improved DehazeFormer provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of CASaE provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of a convolutional layer sequence module provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the coordinate attention mechanism provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a SaE module provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of the BiFormer module provided in one embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of the DehazeFormer module provided in one embodiment of this application;

[0031] Figure 8 A comparison image before and after dehazing using the conventional DehazeFormer model, provided as an embodiment of this application;

[0032] Figure 9This is a magnified view of a conventional DehazeFormer model after dehazing, provided in an embodiment of this application.

[0033] Figure 10 A graph showing the PSNR and SSIM of a conventional DehazeFormer model provided in an embodiment of this application;

[0034] Figure 11 A comparison image before and after dehazing using the improved DehazeFormer model, provided as an embodiment of this application;

[0035] Figure 12 This is a magnified view of a portion of the improved DehazeFormer model after dehazing, provided in an embodiment of this application.

[0036] Figure 13 A graph showing the PSNR and SSIM of the improved DehazeFormer model provided in an embodiment of this application. Detailed Implementation

[0037] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0038] Please see Figure 1 An embodiment of this application provides a remote sensing image dehazing method based on an improved DehazeFormer, comprising:

[0039] S1: Obtain the original foggy image.

[0040] The area to be observed is selected, and remote sensing images of the area on a foggy day are acquired using a remote sensing device equipped with a multispectral imaging module, resulting in the original foggy image I.

[0041] S2: Construct an improved DehazeFormer model.

[0042] Please see Figure 1 This is an improved DehazeFormer model, which is an improvement on the traditional DehazeFormer model.

[0043] Specifically, the improved DehazeFormer model introduces a BiFormer (visual transformer with dual horizontal routing attention) module on the basis of the traditional DehazeFormer model to balance global perception and computational efficiency; and sets up a CASaE (coordinate enhancement compression aggregation module) module to optimize local detail repair and reduce color difference blocks and false edges.

[0044] The improved DehazeFormer model consists of a top-down 3×3 convolutional layer, a DehazeFormer module, a first downsampling layer, a DehazeFormer module, a second downsampling layer, a DehazeFormer module, a BiFormer module, a first upsampling layer, an SK fusion layer (selective kernel fusion layer), a CASaE module, a DehazeFormer module, a second upsampling layer, a CASaE module, an SK fusion layer, a DehazeFormer module, a 3×3 convolutional layer, and a soft reconstruction layer.

[0045] Please see Figure 2 The CASaE module comprises, from top to bottom, a coordinate attention mechanism, residual connection layers, a sequence of convolutional layers, and the SaE module. The SaE module refers to the Squeeze Aggregated Excitation module.

[0046] Specifically, please refer to Figure 3 The convolutional layer sequence, from top to bottom, consists of a 3×3 two-dimensional convolutional layer (Conv2d), a batch normalization layer (BatchNorm), and a ReLU activation function layer, repeated twice. The two-dimensional convolutional layer extracts local features of the fog (edges, textures), refines spatial details (local morphology of the fog), and enhances the ability to model subtle structures (such as the blurred texture of fog edges). The batch normalization layer normalizes the feature distribution, accelerates training convergence (stabilizes data alignment), and avoids training oscillations caused by uneven feature distribution. The ReLU activation function layer suppresses negative features (filters out invalid noise), preserves positive information (such as changes in fog brightness), introduces non-linearity, and improves feature representation.

[0047] Please see Figure 4 This is a schematic diagram of the coordinate attention mechanism. In this application, the coordinate attention mechanism includes a residual connection layer, an average pooling layer along the X-axis, an average pooling layer along the Y-axis, a concatenation layer, a two-dimensional convolutional layer, a batch normalization layer, a nonlinear transformation layer, a two-dimensional convolutional layer, and a sigmoid activation function layer. In this application, the coordinate attention mechanism uses two one-dimensional global pooling layers to aggregate features along the X-axis and Y-axis, respectively. This means it can perform global pooling along one direction to capture the global context of that direction (such as the horizontal distribution of objects), or it can retain the spatial dimension of the other direction (such as the vertical pixel position), avoiding the loss of positional information. In this way, the coordinate attention mechanism generates direction-aware and position-sensitive attention maps, achieving joint modeling of spatial and channel information.

[0048] Please see Figure 5 The diagram shows the structure of the SaE module, which includes convolutional layers, global average pooling layers, multi-branch fully connected layers, sigmoid activation function layers, and residual connected layers. The SaE module introduces multi-branch fully connected layers to divide the squeezed features into multiple branches for parallel processing, and then aggregates them to generate the final weights. Different branches are used to capture the dependencies between different channels, thereby enhancing the feature interaction between channels and the global representation capability.

[0049] Please see Figure 6 The diagram illustrates the structure of the BiFormer module, which, from top to bottom, includes a 3×3 depthwise separable convolutional layer, a residual connection layer, a layer normalization layer, a two-layer routing attention mechanism, another residual connection layer, a layer normalization layer, a multilayer perceptron, and another residual connection layer. A dynamic two-layer routing strategy filters key regions at both coarse-grained and fine-grained levels, significantly reducing computational complexity while maintaining the ability to model long-range dependencies.

[0050] S3: The original foggy image is dehazed based on the improved DehazeFormer network model to obtain the final dehazed image.

[0051] The original foggy image I is input into an improved DehazeFormer network model. The model's multi-layer feature extraction and multi-scale feature fusion are used to defog the foggy image I, resulting in the final defog image J.

[0052] Specifically, the original foggy image I is input into a 3×3 two-dimensional convolutional layer to extract initial features, and the output size is... The original feature map is then input into the DehazeFormer module for feature extraction and enhancement to obtain the processed original feature map.

[0053] Specifically, please refer to Figure 7 The DehazeFormer module is used to extract and enhance features from the original feature map to obtain a processed original feature map. First, the input original feature map is preprocessed to obtain a preprocessed original feature map.

[0054] The preprocessing process includes: inputting the original feature map into a scaling and normalization layer for scaling and normalization to obtain a scaled and normalized original feature map; and inputting the normalized original feature map into a reflection filling layer for reflection filling to obtain a preprocessed original feature map. This preprocessing method effectively avoids the boundary effect problem caused by missing boundary pixels when the convolution kernel slides to the boundary of the original feature map. Reflection filling expands the boundary features and reduces convolution boundary effects.

[0055] Next, the preprocessed original feature map is input into a window-based multi-head self-attention (W-MHSA) module to extract local and global features. Specifically, the preprocessed original feature map is input into multiple linear layers for linear transformation to obtain the query vector. Key vector Sum value vector Each query vector and key vector Each performs matrix multiplication independently. → Matrix multiplication Softmax normalization → Matrix multiplication ·V. Wherein the query vector and key vector Perform matrix multiplication The feature matrix is ​​then obtained, and the value of the feature matrix reflects the degree of correlation between features at different locations.

[0056] Furthermore, the feature matrix is ​​subjected to Softmax normalization to transform it into attention weights in the form of a probability distribution. , The importance of each location when calculating attention. Attention weights. Then with value vector Perform matrix multiplication and apply attention weights to the value vector. The features in the vector were weighted and aggregated to highlight those relevant to the query vector. The relevant features are then identified. Next, Merge merges the outputs of each attention head in Multi-Head Attention (MHSA) to obtain the merged features. Conv(V) enhances the local contextual information through convolution operations to obtain the enhanced original feature map. The enhanced original feature map and the merged features are then added element-wise (Residual Connection) to obtain the final fusion result.

[0057] The final fusion result is input into a linear layer for linear transformation. Multiplying this result with the weight matrix of the linear layer yields the original feature map after linear transformation, adjusting the numerical distribution and representation of the features. Before calculating window attention, to maintain a consistent window size, the original feature map after linear transformation is cropped and subjected to affine transformation. Cropping removes edge padding pixels added during the affine transformation stage to restore the original spatial dimensions. Finally, residual connections are performed on the feature map after restoring the original spatial dimensions, and a multilayer perceptron (MLP) is used for linear transformation and activation to obtain the processed original feature map. Specifically, the feature map after restoring the original spatial dimensions undergoes residual connections, is added element-wise to the original feature map, and then fed into the MLP. The MLP in this application consists of two linear layers and a ReLU activation function. The feature map after element-wise addition undergoes linear transformation and activation in the MLP, and is then output as the processed original feature map through a final residual connection. It is worth noting that the affine transformation in this application can perform operations such as translation, rotation, and scaling on the features. In the process of image dehazing, affine transformation can further adjust the spatial location and scale of features, making features from different sources or processed in different ways more consistent and coordinated in spatial structure, which helps to better integrate and utilize these feature information in the future.

[0058] This application uses multiple DehazeFormer modules to extract features from the original feature maps layer by layer. Each DehazeFormer module processes the extracted features through an attention mechanism and a multilayer perceptron (MLP) to refine the feature representation.

[0059] Please see Figure 1 After the DehazeFormer module extracts the original feature map, it obtains a processed original feature map. Then, the processed original feature map is input into the first downsampling layer for downsampling to obtain a size of... The number of channels is 2 The first feature map is then input into the DehazeFormer module for feature extraction, resulting in a processed first feature map. This processed feature map is then input into a second downsampling layer for downsampling to obtain a size of [size missing]. The number of channels is 4 The second feature map is then input into the DehazeFormer module for feature extraction, resulting in a processed second feature map. This processed second feature map is then input into the BiFormer module for feature extraction and attention mechanism processing to obtain an integrated second feature map. The purpose of the first and second downsampling layers in this application is to extract global features at low resolution.

[0060] Specifically, please refer to Figure 6The process of inputting the extracted second feature map into the BiFormer module for feature extraction and attention mechanism processing to obtain the integrated third feature map includes:

[0061] First, the processed second feature map is input into a depthwise separable convolutional layer for convolution, resulting in a convolutional second feature map. The depthwise separable convolutional layer includes depthwise convolution and pointwise convolution. Depthwise convolution performs a convolution operation on each input channel of the second feature map, resulting in a depthwise convolutional second feature map. Pointwise convolution is used to fuse the channel information in the depthwise convolutional second feature map, resulting in a convolutional second feature map. The 3x3 in the diagram indicates that the convolution kernel size is 3x3. Through depthwise separable convolution, not only can the computational cost be reduced, but it can also effectively extract local features from the second feature map, capture the relationships between adjacent pixels in the image, and extract basic image feature information such as edges and textures.

[0062] Then, the second feature map after depthwise convolution is input into the residual connection layer and residually connected with the processed second feature map to obtain the residually connected second feature map. Residual connections solve the gradient vanishing problem in deep networks while preserving the original input information, avoiding the loss of important information during multi-layer network computation, and enhancing the model's stability and performance.

[0063] Next, the second feature map after residual connections is input into the normalization layer for normalization, resulting in a normalized second feature map. The normalization layer calculates the mean and variance of each feature map after residual connections, and then standardizes the features to ensure they have zero mean and unit variance. This accelerates the model training process, alleviates the vanishing and exploding gradient problems, and makes the network converge more easily. Simultaneously, it makes the feature distribution of the second feature maps after different residual connections more stable, enhancing the model's generalization ability and helping subsequent modules better process features.

[0064] Furthermore, the attention weights of the normalized second feature map are calculated using a two-layer routing attention mechanism. This mechanism calculates the attention weights through a two-layer routing process, filtering and weighting features at different levels based on the input features, focusing on more important feature parts and suppressing less important ones. The specific steps are as follows:

[0065] First, normalize the second feature map Divide into S×S non-overlapping regions, such that each region contains Each feature vector (token) is used to analyze the normalized second feature map. The reshaping operation yields the region feature representation. This completes the process of creating the second feature map after normalization. To region feature representation The conversion.

[0066] Then, the regional feature representation Performing linear transformations yields the query, key, and value tensors, respectively, as shown in the following formulas:

[0067] ;

[0068] ;

[0069] ;

[0070] In the formula, , , These are the projection weight matrices for query, key, and value, each with a dimension of 1. superscript This indicates that it is related to a region. Features after region division and reshaping operations, and To obtain the representative vector after calculating the mean at the region level, the input normalized second feature map is converted into a form suitable for attention calculation.

[0071] Secondly, based on region-to-region routing of the directed graph, the resulting query tensor... Bond tensor Perform an averaging operation on each region to obtain a representative vector query. s and keys (All dimensions are) This approach captures the relationships between features from a regional perspective. The affinity matrix between regions is calculated using matrix multiplication. In the formula, For the region and The semantic relevance is such that a larger value indicates a closer semantic connection between the two regions.

[0072] Next, the affinity matrix Filtering is performed to reduce computational load. Specific steps include filtering each region (i.e....) For each row, perform a top-k operation, retaining the rows with the highest semantic relevance to the current region. From each region, the routing index matrix is ​​obtained. .matrix The The line contains the first The most relevant region The indexes of each region are used. By filtering out the most relevant regions and removing those that contribute little to the current region, the complexity of subsequent attention calculations is reduced.

[0073] Furthermore, based on the routing index matrix (Record the first k regions associated with each region), from the original key tensor Sum tensor In the process, all tokens (a token refers to a basic unit after discretization or characterization of the original data) within the associated region are extracted to form a contiguous block of memory. and The formula is as follows:

[0074] ;

[0075] ;

[0076] In the formula, K and V are region-organized key and value tensors (of shape ). , (Number of tokens for each region). The scattered tokens are consolidated into a contiguous memory block, completing the aggregation of the key tensor K and value tensor V.

[0077] Finally, attention is calculated among tokens. For each query token, attention weights are calculated only on the aggregated key-value pairs. The formula is as follows:

[0078] ;

[0079] In the formula, This is a local context enhancement module (depth convolution, kernel size=5) used to supplement local details.

[0080] Subsequently, the attention weights are residually connected to the second feature map after the residual connection to obtain the second feature map after the second connection. The input normalization layer of the second feature map after the second connection is normalized again. The mean and variance of the feature dimensions of each second feature map after the second connection are standardized again to further stabilize the feature distribution and ensure that the features processed by the attention mechanism still have a suitable numerical range and distribution shape, which is convenient for subsequent processing by the multilayer perceptron module. At the same time, it helps to improve the stability of training and the performance of the model.

[0081] Finally, a multilayer perceptron is used to further process the second feature map after secondary normalization, thereby extracting and enhancing the abstract representation of the features and improving the model's ability to understand and express data. In this application, the multilayer perceptron consists of multiple fully connected layers, connected by activation functions. By performing multiple linear and nonlinear transformations on the input features, more complex relationships between features are uncovered. Finally, residual connections are used to integrate the connected second feature map and the deeply processed second feature map, resulting in an integrated second feature map that improves the overall model performance.

[0082] After the BiFormer module outputs the integrated second feature map, it inputs the integrated second feature map into the first upsampling layer for upsampling to obtain a size of... The number of channels is 2 The third feature map is used to restore a higher resolution. The third feature map is then input into the SK fusion layer for feature fusion to obtain the fused third feature map; the fused third feature map is then input into the CASaE module to obtain the processed third feature map.

[0083] Specifically, please refer to Figure 2 and Figure 4 The fused third feature map is first input into the coordinate attention mechanism to obtain the enhanced third feature map.

[0084] First, the input fused third feature map (C is the number of channels, H×W is the spatial size) Perform residual connections to obtain the third feature map after residual connections, so as to ensure the stability of feature transfer.

[0085] Then, average pooling operations are performed along the horizontal X and vertical Y directions, respectively. The horizontal X-direction average pooling operation includes: performing global average pooling along the vertical direction (H dimension) to obtain the horizontal feature map. The formula is as follows:

[0086] ;

[0087] In the formula, For the first The global aggregate value of each channel at the horizontal position. This is the height position index of the feature map, representing the pixel row position in the vertical direction (Y-axis) of the feature map, corresponding to the height dimension H of the feature map. The width of the feature map represents the total number of pixel columns in the horizontal direction (X-axis) of the feature map. Let H be the 2D feature matrix of the c-th channel, with dimensions H×W. To represent the pixel column position in the horizontal direction (X-axis) of the feature map, we use the width dimension W of the feature map.

[0088] The steps of the vertical Y-axis average pooling operation include: performing global average pooling along the horizontal direction (W dimension) to obtain the vertical feature map. The formula is as follows:

[0089] ;

[0090] In the formula, For the first Each channel is in a vertical position The global aggregate value at that location, This represents the height of the feature map, which is the number of pixel rows in the vertical direction. The pixel value located in the j-th row vertically and the w-th column horizontally in the feature map representing the c-th channel.

[0091] Horizontal feature map Captures global dependencies in the horizontal direction (such as whether an object is on the left or right side of the image) while preserving positional information in the vertical direction. Coordinates). Vertical feature map. Capture global dependencies in the vertical direction (such as whether the object is in the upper or lower half) while preserving positional information in the horizontal direction. Coordinates). and By stitching and fusing information from horizontal and vertical directions along the spatial dimension, cross-directional dependencies are established to obtain... Next, through 1×1 2D convolution (Conv2d), batch normalization (BN), and nonlinear transformation, batch normalization is first used to normalize the features after 1×1 convolution, stabilizing the feature distribution, accelerating model training, and improving generalization ability. Then, nonlinear transformation is used to activate and enhance the feature representation ability, compressing the concatenated features to C / r channels (r is the reduction rate) to reduce computational complexity, resulting in an enhanced feature map. The formula is as follows:

[0092] ;

[0093] In the formula, For the enhanced feature map, For convolution operations, It is a nonlinear function.

[0094] Subsequently, the enhanced feature map f is split into two branches along the spatial dimension, corresponding to the feature maps in the horizontal and vertical directions, respectively. and Then through 1×1 two-dimensional convolution Horizontal feature map Restore to The channels, after being activated by the Sigmoid function, generate a horizontal attention map. vertical direction Similarly, the vertical attention map is obtained. ,in, , , This is the Sigmoid function. To emphasize the importance of each position in the horizontal direction, To assess the importance of each position in the vertical direction, the Sigmoid activation function normalizes the attention values ​​to 0 to 1 for subsequent feature weighting.

[0095] Finally, the third feature map is obtained by concatenating the horizontal and vertical attention maps with the residuals. Element-wise multiplication yields the enhanced third feature map. The formula is as follows:

[0096] ;

[0097] In the formula, These are spatial coordinates.

[0098] Strengthen target-related areas (such as horizontal positions) by focusing attention weights. And vertical position The feature enhancement method suppresses irrelevant backgrounds and combines inter-channel dependencies (through convolutional transformation) and spatial location information (coordinates preserved through 1D pooling) to achieve more accurate feature enhancement.

[0099] Further, its reference Figure 3 The enhanced third feature map is then concatenated with the residual of the fused third feature map to obtain the second concatenated third feature map. This second concatenation is then used to input the convolutional layer sequence to obtain the activated third feature map.

[0100] First, the third feature map after the second concatenation is input into a 3×3 two-dimensional convolution layer to obtain the convolutional third feature map. The 3×3 convolutional layer is used to extract local features of fog, enhancing the ability to model fine structures. Next, a batch normalization operation is performed to normalize the feature distribution of the convolutional third feature map, resulting in a normalized third feature map. This accelerates training convergence and avoids training oscillations caused by uneven feature distribution. Finally, the ReLU activation function is used to suppress the negative features of the normalized third feature map, improving the feature representation ability and obtaining the activated third feature map.

[0101] Repeat this process and fuse the third feature map after the second connection through residual connection to obtain the third feature map after the third connection, so as to ensure that information is effectively transmitted in the deep network, avoid information dilution after multi-layer calculation, and enhance detailed features.

[0102] Finally, the third feature map after three connections is input into the SaE module, and the channel features are processed through a multi-branch fully connected layer to obtain the processed third feature map.

[0103] Specifically, firstly, the number of channels is reduced from C / G to C / r to reduce computational cost; then, the key features for effective dehazing are screened using the ReLU activation function, and the results of multi-branch fully connected layers are concatenated to restore the number of channels and generate robust channel weights to capture more complex inter-channel dependencies (distinguishing channel differences between foggy and clear regions).

[0104] In the CASaE module of this application, a coordinate attention mechanism is used to compensate for the deficiencies in spatial perception. The designed convolutional layer sequence module can enhance local details, the SaE module can improve channel discrimination, and the residual connections throughout the CASaE module can ensure information integrity. Through a pipeline mode of "localization-refinement-optimization", features are enhanced in all three dimensions of space, locality, and channel to balance accuracy and efficiency.

[0105] Furthermore, the processed third feature map output from the CASaE module is input into the DehazeFormer module for feature extraction, resulting in the extracted third feature map. This extracted third feature map is then input into the second upsampling layer for upsampling, yielding a size of [size missing]. The number of channels is The fourth feature map is then input into the CASaE module to obtain the processed fourth feature map. This processed fourth feature map is then input into the SK fusion layer to fuse multi-scale features.

[0106] Specifically, the core idea of ​​SK Fusion is to enable the model to automatically select an appropriate receptive field size based on the input content, thereby better handling targets of different scales. Traditional convolutional neural networks typically use a single-size convolutional kernel, making it difficult to adapt to complex scenes with multi-scale variations. This application uses SK Fusion to use multiple convolutional kernels of different sizes in parallel, and then dynamically adjusts their weights based on the input to achieve adaptive fusion of features at different scales, resulting in a fused feature map.

[0107] Specifically, the SK fusion layer mainly includes three key steps: Split, Fuse, and Select.

[0108] The splitting process includes processing the processed fourth feature map of the input through multiple convolutional kernels of different scales to generate multiple sets of fourth feature maps with different receptive fields. This application uses 3x3 and 5x5 convolutional kernels to apply them to the input features in parallel. The specific convolutional process formula is as follows:

[0109] ;

[0110] ;

[0111] In the formula, X is the input feature, and U1 and U2 are the fourth feature maps output after 3x3 convolution and 5x5 convolution, respectively.

[0112] The fusion process includes fusing the fourth feature maps U1 and U2 at different scales, enabling subsequent networks to dynamically adjust the importance of the fourth feature maps U1 and U2 at different scales based on the input features.

[0113] Specifically, the steps for summing elements include:

[0114] The fourth feature maps U1 and U2, which have different branches and scales, are added element-wise as follows:

[0115] ;

[0116] In the formula, U is the fourth feature map after addition, U1 is the fourth feature map output by 3x3 convolution, and U2 is the fourth feature map output by 5x5 convolution.

[0117] Next, global information aggregation is performed on the summed fourth feature map. Global Average Pooling (GAP) is used to obtain global context information, as shown in the following formula:

[0118] ;

[0119] In the formula, s is the fourth feature map after pooling, GAP() is global average pooling, H is the height, W is the width, and U is the fourth feature map after summation.

[0120] Next, the pooled fourth feature map s undergoes dimensionality reduction and nonlinear transformation. Specifically, a fully connected (FC) layer and a ReLU activation function layer are used to reduce the dimensionality of the pooled fourth feature map s, thereby reducing computational cost. The formula is as follows:

[0121] ;

[0122] In the formula, z is the fourth feature map after dimensionality reduction, and ReLU() is the activation function. This is a dimension reduction matrix with dimension 1. ,in, It is the dimensionality reduction ratio (usually taken as 16). This represents the number of channels.

[0123] Furthermore, attention weights are generated based on global information, and adaptive selection is performed on features at different scales. The specific steps are as follows:

[0124] Attention weights corresponding to different convolutional kernel branches are generated through multiple fully connected layers, as shown in the following formula:

[0125] ;

[0126] ;

[0127] In the formula, and These are the attention weights corresponding to feature maps U1 and U2, respectively. and All are weight matrices of the fully connected layer, and z is the third feature map after dimensionality reduction.

[0128] Next, Softmax normalization is used to normalize the attention weights, ensuring that the sum of a and b is 1, as shown in the following formula:

[0129] ;

[0130] ;

[0131] In the formula, and , respectively, are the normalized attention weights corresponding to the fourth feature maps U1 and U2, and a and b are the attention weights corresponding to the fourth feature maps U1 and U2, respectively.

[0132] Furthermore, the normalized attention weights corresponding to the fourth feature maps U1 and U2 are respectively... and The features are then applied to the corresponding branches, and a weighted fusion is performed to obtain the fourth feature map after fusion, as shown in the following formula:

[0133] ;

[0134] In the formula, This is the fourth feature map after fusion. and These are the attention weights for the normalized fourth feature maps U1 and U2, respectively. U1 is the fourth feature map output by a 3x3 convolution, and U2 is the fourth feature map output by a 5x5 convolution.

[0135] The fused fourth feature map is input into the DehazeFormer module for feature extraction, and the extracted fourth feature map is then input into a convolutional layer with a kernel size of 3×3 for convolution, resulting in the convolutional fourth feature map.

[0136] Furthermore, the fourth feature map after convolution is input into the soft reconstruction layer to obtain the final dehazed image J.

[0137] The soft reconstruction layer is designed based on an atmospheric scattering model, and improves the defogging effect by introducing weak prior constraints. Its physical model reconstruction and mathematical expression are derived from the formulas of the traditional atmospheric scattering model as follows:

[0138] ;

[0139] In the formula, This is the fourth feature map after convolution. For the final dehazed image J, Transmittance, This refers to global atmospheric light.

[0140] Example 1: Dehazing comparison experiment between traditional DehazeFormer model and improved DehazeFormer model.

[0141] The foggy image is input into both the traditional DehazeFormer model and the improved DehazeFormer model for dehazing, resulting in a dehazed image. Figure 8 The image shows a comparison of the traditional DehazeFormer model before (left) and after (right) dehazing. Figure 9 This is a magnified view of a portion of the image after dehazing. Figure 10 The graphs show the PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) metrics, respectively. The PSNR of the traditional DehazeFormer model is 26.0868, and the SSIM is 0.8771.

[0142] Figure 11 To improve the DehazeFormer model, a comparison image is shown before (left) and after (right) dehazing. Figure 12 This is a magnified view of a portion of the image after dehazing. Figure 13 The graphs show the PSNR and SSIM for the improved DehazeFormer model, with a PSNR of 28.1174 and an SSIM of 0.9037.

[0143] The comparison shows that the improved DehazeFormer model has higher PSNR and SSIM than the traditional DehazeFormer model. A local magnification comparison of the dehazing results of the traditional and improved DehazeFormer models is also provided. Figure 9 and Figure 12 It can be seen that the improved DehazeFormer model has a clearer image and a more complete structure.

[0144] In summary, the improved DehazeFormer model of this application produces dehazed images with smaller numerical errors compared to real fog-free scenes, resulting in more accurate restoration results and more complete image structures.

[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A remote sensing image dehazing method based on an improved DehazeFormer, characterized in that, include: Obtain the original foggy image, construct an improved DehazeFormer model, and dehaze the original foggy image based on the improved DehazeFormer model to obtain the final dehazed image; The improved DehazeFormer model introduces the CASaE module and the BiFormer module; The CASaE module includes a coordinate attention mechanism, a residual connection layer, a convolutional layer sequence module, and a SaE module, which are used to repair local details of the original hazy image and reduce color difference blocks and false edges. The BiFormer module includes depthwise separable convolutional layers, residual connection layers, layer normalization layers, a two-layer routing attention mechanism, and a multilayer perceptron, which are used to balance the global perception and computational efficiency of the improved DehazeFormer model. The improved DehazeFormer model sequentially includes a convolutional layer, a DehazeFormer module, a first downsampling layer, a DehazeFormer module, a second downsampling layer, a DehazeFormer module, a BiFormer module, a first upsampling layer, an SK fusion layer, a CASaE module, a DehazeFormer module, a second upsampling layer, a CASaE module, an SK fusion layer, a DehazeFormer module, a convolutional layer, and a soft reconstruction layer. The original foggy image is processed through the convolutional layer, DehazeFormer module, first downsampling layer, DehazeFormer module, second downsampling layer, and DehazeFormer module to obtain a processed second feature map. This second feature map is then input into the BiFormer module for feature extraction and attention mechanism processing to obtain an integrated second feature map. This integrated second feature map is then sequentially input into the first upsampling layer and SK fusion layer for upsampling and feature fusion to obtain a fused third feature map. This fused third feature map is then input into the CASaE module to obtain a processed third feature map. Finally, the image is input into the DehazeFormer module, second upsampling layer, CASaE module, SK fusion layer, DehazeFormer module, convolutional layer, and soft reconstruction layer to obtain the final dehazed image. The process of obtaining the processed third feature map includes: The fused third feature map is input into the coordinate attention mechanism to obtain an enhanced third feature map. The enhanced third feature map is then residually connected to the fused third feature map to obtain a second-connected third feature map. This second-connected third feature map is then input into a convolutional layer sequence for convolution operations to obtain an activated third feature map. This activated third feature map is then residually connected to the second-connected third feature map to obtain a third-connected third feature map. This third-connected third feature map is then input into the SaE module, where channel features are processed through a multi-branch fully connected layer to obtain the processed third feature map.

2. The remote sensing image dehazing method based on the improved DehazeFormer as described in claim 1, characterized in that, The convolutional layer sequence module includes a two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer, which are used to extract local features of fog, normalize the distribution of the local features, suppress negative features in the local features, retain positive features in the local features, and introduce nonlinearity.

3. The remote sensing image dehazing method based on the improved DehazeFormer as described in claim 1, characterized in that, The process of obtaining the enhanced third feature map includes: The fused third feature map is residually concatenated to obtain a residually concatenated third feature map. Average pooling is then performed along the horizontal and vertical directions to obtain horizontal and vertical feature maps. The dependencies in the horizontal and vertical directions are concatenated along the spatial dimension and then fed into a two-dimensional convolutional layer for convolution. After normalization using the Sigmoid activation function, attention maps in the horizontal and vertical directions are obtained. These attention maps are then multiplied element-wise with the residually concatenated third feature map to obtain the enhanced third feature map.

4. The remote sensing image dehazing method based on the improved DehazeFormer as described in claim 3, characterized in that, The process of obtaining the activated third feature map includes: The third feature map after residual concatenation is input into a two-dimensional convolutional layer for convolution to obtain the convolutional third feature map. Normalization is then performed to obtain the normalized third feature map. The ReLU activation function is used to suppress the negative features of the normalized third feature map and improve the feature representation ability to obtain the activated third feature map.

5. The remote sensing image dehazing method based on the improved DehazeFormer as described in claim 1, characterized in that, The process of inputting the third feature map after the three connections into the SaE module to obtain the processed third feature map includes: The third feature map after the three connections is reduced in dimensionality to obtain the third feature map after dimensionality reduction. The key features for effective dehazing are selected by using the ReLU activation function. The results of the multi-branch fully connected layer are concatenated to restore the number of channels and generate channel weights. The inter-channel dependencies are captured to obtain the processed third feature map.

6. The remote sensing image dehazing method based on the improved DehazeFormer as described in claim 1, characterized in that, The process of obtaining the integrated second feature map includes: inputting the processed second feature map into a depthwise separable convolutional layer for depthwise convolution and pointwise convolution to obtain a depthwise convolutional second feature map; performing a residual connection with the processed second feature map to obtain a residually connected second feature map; performing normalization processing in the input layer normalization layer to obtain a normalized second feature map; calculating the attention weights of the normalized second feature map through a two-layer routing attention mechanism; performing a residual connection with the residually connected second feature map to obtain a second-connected second feature map; performing normalization again in the input layer normalization layer to obtain a second-normalized second feature map; and processing the integrated second feature map using the multilayer perceptron.

Images