Fog removal method based on inter-layer multi-scale sequence interaction and fourier domain frequency-space enhancement

Abstract

The method for removing fog based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement comprises the following steps: step S1: collect a public fog data set containing different concentrations of fog and haze, and pre-process the images therein to improve data quality and model training effect; step S2: construct a network framework for removing fog based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement, which comprises a U-shaped fog removal network, an inter-layer multi-scale sequence interaction module IMSIM and a Fourier domain frequency-space enhancement module FDFSEM; step S3: train the network using the labeled image data set, define a suitable loss function and use an optimizer to train; step S4: evaluate the fog removal effect using two indicators of peak signal-to-noise ratio PSNR and structural similarity index SSIM, optimize the trained model, and deploy it to the actual application for online inference and real-time fog removal; through the above steps, the input feature map can be removed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a dehazing method based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement. BACKGROUND

[0002] With the acceleration of urbanization and industrialization, haze weather has become a common phenomenon worldwide, which has a profound impact on social operation and economic activities. In the haze environment, the image quality collected by the visible light imaging system decreases significantly, which is manifested as reduced contrast, lost details and color deviation. These image degradation phenomena not only weaken the perception ability of the human visual system, but also pose a major challenge to computer vision tasks such as automatic driving, target detection and remote sensing image analysis, which rely on high-quality visual data. Therefore, it is of great theoretical research value and practical application significance to study the dehazing technology for effectively restoring the clarity and details of images in the haze environment.

[0003] Existing image dehazing methods can be roughly divided into two categories: parameter prior-based dehazing methods and deep learning-based dehazing methods. The parameter prior-based dehazing method usually relies on the classic atmospheric scattering model, estimates the intermediate parameters such as global atmospheric light and transmittance, and processes the haze image to restore the clear image. Although this method has certain dehazing ability under certain conditions, it has inherent limitations. For example, in scenes with uneven illumination or high haze concentration, the estimation error of the intermediate parameters may cause problems such as artifacts, color distortion and loss of details. In addition, since the parameter prior hypothesis is usually for specific scene conditions, the generalization ability of this method in complex or unknown environments is poor.

[0004] In recent years, with the development of deep learning technology, end-to-end deep learning dehazing methods have gradually become mainstream. These methods learn the mapping relationship between foggy images and clear images directly, avoiding the dependence on intermediate parameters in the atmospheric scattering model, and significantly improving the dehazing performance and generalization ability. For example, Dong et al. proposed MSBDN, which enhanced the dehazing effect and detail restoration ability of the network in complex foggy scenes through multi-scale feature fusion and dense connection, especially under different weather conditions. Liu et al. designed GridDehazeNet, which successfully realized efficient dehazing of complex foggy images by introducing grid structure, multi-scale feature extraction and residual module. Song et al. developed DehazeFormer, which improved the image dehazing effect by introducing Transformer architecture and self-attention mechanism, especially in global information modeling and long-distance dependency capture. Jiang et al. proposed KFA-Net, which combined asymmetric size feature cascade, k-means pixel attention network and channel attention network FCA, and improved feature extraction, thick fog area focusing and frequency domain information attention for non-uniform foggy remote sensing images, significantly improving the image dehazing performance.

[0005] Although existing dehazing methods have improved image quality to some extent, it is still difficult to completely eliminate artifacts and detail distortion when dealing with complex foggy scenes. The specific problems are as follows:

[0006] (1) Most dehazing networks adopt U-shaped structure, usually ignoring the inter-layer feature interaction between encoding layers, resulting in limited multi-scale feature fusion efficiency and effect. In addition, the limitations of convolution operation make the network receptive field mainly concentrated in local area, making it difficult to effectively model long-distance dependency, limiting the capture and expression ability of global features. This is particularly evident when dealing with complex foggy scenes, making it difficult to restore image details and maintain the consistency of global structure at the same time.

[0007] (2) Many existing methods mainly focus on time domain or spatial domain features, ignoring the mining of frequency domain information. Although frequency domain information such as global structure and high-frequency details is crucial to improving image clarity and contrast, the combination of time and frequency domain information is still insufficient. The restoration of image details and texture is limited, further affecting the visual effect of the image and the accuracy of subsequent processing. SUMMARY

[0008] The present application aims to solve several key problems in existing image defogging methods, specifically: most U-shaped structure-based defogging networks fail to fully realize the depth interaction of inter-layer features, leading to insufficient multi-scale feature fusion, and in complex haze scenes, it is difficult to effectively combine global information and local details, thereby causing the texture of the defogged image to be blurred and details to be lost; in addition, existing technologies are insufficient in frequency domain feature mining, especially high-frequency information processing, high-frequency information is crucial for detail recovery and edge enhancement, but many methods fail to fully utilize the high-frequency components and global structural features in the frequency domain, resulting in unclear image details and poor structural consistency. More importantly, the combination of time-frequency domain information is still insufficient, further limiting the improvement of image restoration effect.

[0009] To solve the above technical problems, the technical scheme adopted by the present application is:

[0010] The defogging method based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement comprises the following steps:

[0011] Step S1: collect a public fog dataset containing different concentrations of fog, and preprocess the images therein to improve data quality and model training effect;

[0012] Step S2: construct a defogging network framework based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement, which includes a U-shaped defogging network, an inter-layer multi-scale sequence interaction module IMSIM, and a Fourier domain frequency-space enhancement module FDFSEM;

[0013] Step S3: train the network using the labeled image dataset, define a suitable loss function, and use an optimizer to train it;

[0014] Step S4: evaluate the defogging effect using the peak signal-to-noise ratio PSNR and structural similarity index SSIM, and compare it with other methods to verify the model performance. Optimize the trained model and deploy it to actual applications for online inference and real-time defogging;

[0015] Through the above steps, the defogging of the input feature map can be realized.

[0016] In step S1, a public fog dataset with different concentrations of fog is collected, and then the images are screened and sorted; then, the images in the dataset are preprocessed, including image normalization and data enhancement, thereby improving the generalization ability of the model.

[0017] In step S2, when constructing the U-shaped defogging network, specifically comprising: an encoding layer feature extraction module, a residual module, a decoding layer image recovery module third decoding layer; the encoding layer feature extraction module comprises a first encoding layer, a second encoding layer, a third encoding layer, the residual module, the decoding layer image recovery module comprises a third decoding layer, a second decoding layer, a first decoding layer;

[0018] When constructing the inter-layer multi-scale sequence interaction module IMSIM, the local and global context information in the different encoding layer features is extracted through multi-scale convolution. At the same time, the long-distance dependence relationship between the cross layers is captured by using the sequence modeling mechanism mLSTM, so as to strengthen the interaction and expression ability of the multi-scale features. The features output by the module are further optimized through the residual module to generate aggregated features that fuse edge details and semantic information, and finally are transmitted to the third decoding layer to significantly improve the defogging effect of the network;

[0019] When constructing the Fourier domain frequency-space enhancement module FDFSEM, in the decoding stage, for the corresponding encoding layer, the frequency domain features are extracted through Fourier transform to strengthen the high frequency information, and the spatial domain features are extracted through convolution operation to maintain the texture integrity. After the frequency-space collaborative processing, the optimized features are transmitted to the corresponding decoding layer to remove the artifacts and enhance the structural consistency.

[0020] Specifically, in the encoding stage, an input foggy image is extracted through the first encoding layer, the second encoding layer and the third encoding layer. At the same time, the different encoding layer features are modeled and interacted by combining the inter-layer multi-scale sequence interaction module IMSIM, so as to improve the fusion ability of the cross-layer information. In this process, the encoding layer not only captures local details, but also extracts global context information to provide high-quality features for the decoding stage. In the encoding stage, for the encoding features corresponding to each decoding layer, the Fourier domain frequency-space enhancement module FDFSEM is introduced to strengthen the high frequency detail expression through the collaborative processing of the frequency domain and the spatial domain. The optimized features are directly transmitted to the decoding stage through the jump connection to ensure the efficient transmission of multi-scale information. In the decoding stage, the spatial resolution of the image is restored through stepwise upsampling. The third decoding layer, the second decoding layer and the first decoding layer combine the encoding features and the output of the frequency-space enhancement module to realize feature fusion through the residual connection, effectively improving the image detail expression ability. Finally, the network outputs the clear image after defogging. In order to optimize the network performance, a series of loss functions are used to constrain the pixel consistency, detail restoration ability and global feature expression of the defogged image, so as to learn the mapping relationship between the input foggy image and the target clear image, and significantly improve the defogging effect and image quality.

[0021] In constructing the U-shaped network, the features of the input image are extracted layer by layer through four times of downsampling operations. The first encoding layer completes preliminary feature extraction, the second encoding layer further extracts features, and the third encoding layer captures deep semantic features and context information. Each encoding layer first performs feature downsampling through convolution operation to reduce spatial dimensions and extract higher-level features, then performs normalization processing on the feature map to ensure consistency and stability, and enhances the non-linear expression ability of the network through the ReLU activation function. The down-sampled feature map is further extracted by multiple residual modules to capture more rich semantic information and global context.

[0022] In the decoding stage, the features extracted by the encoder are gradually restored to the spatial resolution of the image through upsampling operations. Among them, the third decoding layer restores the global features, the second decoding layer further fuses the feature information, and the first decoding layer gradually restores the detailed information of the image and restores it to the spatial size of the input image. Through layer-by-layer upsampling and feature fusion, the final output of the defogging image is obtained, and the defogging task is completed.

[0023] In constructing the inter-layer multi-scale sequence interaction module IMSIM, different encoding layers of the U-shaped defogging network carry important information, such as shallow features focusing on capturing detailed texture information, and deep features containing more global semantic information such as color and brightness. Therefore, in order to fully utilize the information of different levels of features in the encoding stage, reduce the loss of feature information in the transmission process, enhance the reconstruction effect of the defogging image in the subsequent decoding stage, optimize the feature interaction and fusion between layers, and further improve the overall defogging performance of the network. For the input feature F k ∈R C×H×W (k∈{i,j}) of the encoding stage, first, Layer Normalization operation is performed on it to standardize the feature distribution and enhance the training stability. Subsequently, the feature F k is sequentially passed through three different receptive field convolution operations, which are 3*3 convolution, 5*5 convolution, and 7*7 convolution, respectively, to gradually extract local and global context information. The overall feature extraction process can be described as:

[0024] F ms,k =Conv 7×7 (Conv 5x5 Conv 3×3 (LN(F k ))), (k∈{i,j})

[0025] where F ms,k ∈R C×H×W represents the feature map obtained after multi-scale convolution operation. The step-by-step operation of the three convolution kernels can capture feature information under different receptive fields, thereby preserving edge details while enhancing global semantic representation. The extracted multi-scale features Fms,k is converted into a two-dimensional sequence form F seq,k ∈R (H·W)×C Modeling requirements with adaptive sequences:

[0026] F seq,k = Flatten(F ms,k ), (k∈{i,j})

[0027] The feature sequence is input to the mLSTM module, which further optimizes the feature representation by capturing long-distance dependencies between sequences, generating updated sequence features F' seq,k ∈R (H·W)×C :

[0028] F' seq,k = mLSTM(F seq,k ), k∈{i,j}

[0029] Through this operation, the module can effectively extract long-distance dependency information across layers, making the relationship between shallow and deep features more closely, laying the foundation for subsequent feature fusion. Then, cross-layer feature relationship calculation and fusion are performed. Calculate the relationship matrix between F' seq,i ∈R (H·W)×C , the transpose of the shallow feature F' seq,i T ∈R C×(H·W) and F' seq,j ∈R (H·W)×C deep sequence features:

[0030] W = Softmax(F' seq,i T · F' seq,j )

[0031] The weight matrix W ∈R C×C is used to measure the interaction between the two features, capturing complementary information from different encoding layer features. Using the weight matrix W, the features F' seq,i of the shallow path are weighted and summed to generate the fused features:

[0032] F weighted = F' seq,i · W

[0033] The weighted feature F weighted ∈R (H·W)×C is reshaped to restore the original feature map shape F res ∈R C×H×W , and the fused features are further compressed through 1*1 convolution. Channel information to generate the final module output feature F final ∈R C×H×WThis operation can integrate multi-scale information across layers, optimize the spatial and semantic representation of features, and provide more accurate and efficient feature support for subsequent dehazing decoding.

[0034] The inter-layer multi-scale sequence interaction module IMSIM extracts feature representations under different receptive fields through multi-scale convolution, captures long-distance dependencies across layers using sequence modeling, and further optimizes inter-layer interaction through feature fusion. This module strengthens the integration of edge details and global semantic information, improves the collaborative ability of shallow and deep features, alleviates the feature dilution problem of traditional U-shaped networks, enhances context modeling, provides comprehensive feature support for the dehazing task, and effectively improves network performance.

[0035] When constructing the Fourier domain frequency-space enhancement module FDFSEM, the module effectively extracts and strengthens high-frequency information through collaborative processing of frequency domain and spatial domain features, while retaining rich texture details, providing high-quality multi-dimensional feature support for the decoding stage, and significantly improving dehazing performance.

[0036] In the feature processing stage, the input encoded feature T n ∈R C×H×W is first evenly divided into two parts, each part containing C / 2 channels, for frequency domain processing and spatial domain processing, respectively. Among them, T p ∈R C / 2×H×W represents the feature allocated to the frequency domain, and T q ∈R C / 2×H×W represents the feature allocated to the spatial domain. In this way, the feature is divided into two independent components in the channel dimension, laying the foundation for subsequent frequency domain and spatial domain feature enhancement. The specific representation is:

[0037] T p =T n [C / 2,:,:],T q =T n [C / 2,:,:]

[0038] In the Fourier domain processing, the input encoded feature T p is first converted to the frequency domain through two-dimensional fast Fourier transform FFT, obtaining the frequency domain feature T freq ∈R C×H×W :

[0039] T freq =FFT(T p )

[0040] The frequency domain feature T freq contains the real part Re(T freq )∈R C / 2×H×W and the imaginary part Im(T freq )∈RC / 2×H×W The real and imaginary parts of the frequency domain features are respectively subjected to 3*3 convolution operations to extract high frequency information, and Batch Normalization and ReLU activation functions are combined to optimize feature expression:

[0041] T freq-enhanced = ReLU(BN(Conv 3×3 (Re(T freq )))) + ReLU(BN(Conv 3×3 (Im(T freq )))

[0042] Subsequently, the enhanced frequency domain features T freq-enhanced ∈R C×H×W are mapped back to the spatial domain through two-dimensional inverse Fourier transform IFFT, thereby generating enhanced features T reconstructed ∈R C / 2×H×W containing global structure information:

[0043] T reconstructed = IFFT(T freq-enhanced )

[0044] where T reconstucted is the output feature in the frequency domain. In this way, the high frequency components and global features are fully exploited, and the edges, textures and global consistency in the image are optimized. The spatial domain focuses on extracting local information and edge details of the image. The input encoded features T q are sequentially subjected to two 3*3 convolution operations to extract local information, and are further optimized through BN and ReLU activation functions. The spatial domain retains the edge and texture information in the input feature map, thereby providing a supplement for the details not covered by the frequency domain processing. The specific formula is as follows:

[0045] T spatial = ReLU(BN(Conv 3×3 (ReLU(BN(Conv 3×3 (T q ))))))

[0046] where T spatial ∈R C / 2×H×W is the output feature in the spatial domain. In this way, the spatial domain provides rich local information for subsequent fusion operations. The enhanced features T reconsturcted from the frequency domain and the detail features T spatial from the spatial domain are fused through channel dimension concatenation operation Concat to generate fusion features T fused ∈R C×H×W . The formula for the fusion operation is as follows:

[0047] T fused=Concat(T reconstructed ,T spatial )

[0048] Here, represents the fused feature map. This fusion method combines global features in the frequency domain and local details in the spatial domain to generate richer multi-scale feature representations. The fused feature T fused A 1x1 convolution is used to compress the channel dimension, improving feature compactness and representational power. Following this, a channel attention module (CA) further optimizes the importance distribution among channels. The channel attention mechanism learns the importance weights of different channels, enhancing the contribution of key channels to the image dehazing task. The optimized features are then further compressed using a 1x1 convolution to generate the final output feature T. output ∈R C×H×W The calculation formula is as follows:

[0049] T output =Conv 1×1 (CA(Conv 1×1 (T fused )))

[0050] Among them, T outpout This represents the output characteristics of the Fourier domain frequency-space enhancement module FDFSEM.

[0051] In step S3, the network is trained using the labeled image dataset. First, a suitable loss function is defined, including L1 loss, perceptual loss, multi-scale structural similarity loss, and adversarial loss, to measure the difference between the model's prediction results and the true labels. Then, the Adam optimizer is selected to adjust the network parameters, and the model is continuously optimized through the backpropagation algorithm to improve the dehazing performance and accuracy.

[0052] In step S4, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the dehazing effect. PSNR measures the image reconstruction quality; a higher value indicates a higher similarity between the dehazed image and the real image. SSIM evaluates the structural similarity of the image; a value closer to 1 indicates a closer similarity between the generated image and the real image in terms of structure and visual perception. These metrics can quantitatively analyze the model's dehazing effect and compare it with other methods to verify the model's advantages in detail recovery, sharpness, and visual quality. The optimized model will be deployed in a real-world application environment to achieve real-time dehazing processing.

[0053] The constructed dehazing network is based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement. The specific structure of the network is as follows:

[0054] like Figure 1As shown, the foggy image with a size of 3xHxW is padded by reflection padding ReflectionPad2d to preserve the image edge information. Then the image is input into the first encoding layer, using a 7*7 convolution kernel with a step size of 1, and passing through normalization and ReLU activation function to generate an initial feature map with a size of 64xHxW;

[0055] The output of the first encoding layer is connected to the input of the second encoding layer, and the initial feature map enters the second encoding layer, which is down-sampled by a 3*3 convolution kernel with a step size of 2, reducing the feature map size from 64xHxW to 128xH / 2xW / 2. The output of the first encoding layer and the output of the second encoding layer are respectively connected to the input of the first multi-scale sequence interaction module to capture the multi-scale interaction information between texture and semantic features.

[0056] The output of the second encoding layer is connected to the input of the third encoding layer, which is further down-sampled by a 3*3 convolution kernel with a step size of 2, reducing the feature map size from 128xH / 2xW / 2 to 256xH / 4xW / 4. The output of the third encoding layer and the output of the first multi-scale sequence interaction module are respectively connected to the input of the second multi-scale sequence interaction module to interact the obtained feature maps and further integrate the inter-layer multi-scale information.

[0057] The output of the second multi-scale sequence interaction module is connected to the input of a plurality of consecutive residual block modules. The residual block module performs deep semantic modeling on the features to extract high-level features while maintaining the stability of information flow. The output of the residual block module is connected to the input of the third decoding layer after residual fusion with the output of the third Fourier domain space-time enhancement module.

[0058] The third decoding layer completes the up-sampling by deconvolution operation, restoring the feature map size from 256xH / 4xW / 4 to 128xH / 2xW / 2. Meanwhile, the output of the third decoding layer is connected to the input of the second decoding layer after residual fusion with the output of the second Fourier domain space-time enhancement module.

[0059] The feature is up-sampled again to restore the feature map size from 128xH / 2xW / 2 to 64xHxW, and the output of the second decoding layer is connected to the input of the first decoding layer after residual fusion with the output of the first Fourier domain space-time enhancement module.

[0060] The first decoding layer performs a final mapping operation, first uses reflection padding ReflectionPad2d to preserve edge information, then convolves the feature map with a 7*7 convolution kernel to map the channel number to the target channel number, and finally outputs a clear image after dehazing with a resolution of 3xHxW through a Tanh activation function. This process not only restores the resolution, but also ensures the preservation of edge details and the naturalness of the output image.

[0061] The structure of the constructed inter-layer multi-scale sequence interaction module IMSIM is specifically as follows:

[0062] As shown in the formula (1), first, the first feature map and the second feature map output are respectively connected to the input of the first normalization Layer Normalization block and the second normalization Layer Normalization block to standardize the feature distribution, enhance the training stability, and reduce the influence of the feature scale difference on the subsequent calculation. Figure 2

[0063] Subsequently, the first normalization Layer Normalization block outputs the feature map to the first 3*3 convolution block to obtain the first 3*3 convolution block feature map, the first 3*3 convolution block feature map is input to the first 5*5 convolution block to obtain the first 5*5 convolution block feature map, and the first 5*5 convolution block feature map is input to the first 7*7 convolution block to obtain the first 7*7 convolution block feature map. The second normalization Layer Normalization output feature map is input to the second 3*3 convolution block to obtain the second 3*3 convolution block feature map, the second 3*3 convolution block feature map is input to the second 5*5 convolution block to obtain the second 5*5 convolution block feature map, and the second 5*5 convolution block feature map is input to the second 7*7 convolution block to obtain the second 7*7 convolution block feature map. This multi-scale convolution design can capture different scale context semantic information, making the feature expression more rich and comprehensive.

[0064] For the first 7*7 convolution block feature map and the second 7*7 convolution block feature map, they are respectively input to the first Flatten block and the second Flatten block to convert them from the spatial dimension CxHxW to the sequence form to obtain the first feature sequence and the second feature sequence, which are in the shape of (H·W)xC. Then, the first feature sequence and the second feature sequence are respectively input to the first mLSTM block and the second mLSTM block to obtain the enhanced first sequence feature and the enhanced second sequence feature. This process can model the context relationship in the spatial and channel dimensions, thereby strengthening the inter-layer information interaction.

[0065] To capture the interaction relationship between the enhanced first sequence feature and the enhanced second sequence feature, first, the enhanced first sequence feature is input to the Transpose block for transposition operation to obtain the transposed sequence, which is converted from the shape of (H·W)xC to Cx(H·W), so as to be able to perform matrix multiplication operation with the enhanced second sequence feature in the shape of (H·W)xC to calculate the interaction relationship between them, and input the calculation result to the Softmax block to obtain the inter-layer attention weight matrix in the shape of CxC. Subsequently, the weighted sum of the inter-layer attention weight matrix and the enhanced first sequence feature is calculated to obtain the fusion sequence feature, which fully captures the interaction information between the shallow features and the deep features.​

[0066] The fusion sequence feature input Reshape block is restored to the spatial dimension of the original feature map to obtain a fusion feature map with a shape of CxHxW. To further integrate the information of the channel dimension, the fusion feature map is input into a 1*1 convolution block operation to obtain a final output feature with a shape of CxHxW. This operation further optimizes the semantic representation of the feature and the channel expression capability, and provides high-quality feature support for subsequent modules.

[0067] The structure of the Fourier domain frequency-space enhancement module FDFSEM is specifically as follows:

[0068] As shown in Figure 3 The encoding layer input feature map with a size of CxHxW is first input into a Split block to obtain a frequency domain feature map and a spatial domain feature map, and the number of channels of each part is C / 2. The frequency domain feature map is processed in the frequency domain to capture global structure information and high-frequency details; the spatial domain feature map is processed in the spatial domain to strengthen local details and texture information.

[0069] The frequency domain feature map after channel division is input into a two-dimensional fast Fourier transform FFT-2D block to obtain a real-imaginary combined frequency domain feature map. The real-imaginary combined frequency domain feature map is connected to the input of a frequency domain 3*3 convolution block, the output of the frequency domain 3*3 convolution block is connected to the input of a frequency domain batch normalization block BN and a frequency domain ReLU block, the output of the frequency domain batch normalization block BN and the frequency domain ReLU block is connected to the input of an inverse fast Fourier transform IFFT-2D block, and after the processing of the inverse fast Fourier transform IFFT-2D block, an enhanced frequency domain feature map is obtained, and the size remains C / 2xHxW.

[0070] The spatial domain feature map after channel division is connected to the input of a first spatial domain 3*3 convolution block, the output of the first spatial domain 3*3 convolution block is connected to the input of a first spatial domain batch normalization block BN and a first spatial domain ReLU block, the output of the first spatial domain batch normalization block BN and the first spatial domain ReLU block is connected to the input of a second spatial domain 3*3 convolution block, the output of the second spatial domain 3*3 convolution block is connected to the input of a second spatial domain batch normalization block BN and a second spatial domain ReLU block, and an enhanced spatial domain feature map is obtained, and the size is also C / 2xHxW.

[0071] The frequency domain feature map and the spatial domain feature map are spliced in a channel dimension to form a frequency-space fusion feature map with a size of C*H*W. This fusion strategy can effectively combine the global modeling capability of the frequency domain and the local optimization capability of the spatial domain, thereby significantly improving the expression effect of high-frequency details. Subsequently, the frequency-space fusion feature map is input into a first 1*1 convolution block to obtain a first 1*1 convolution block feature map, so as to improve the compactness and representation capability of the feature. Then, the first 1*1 convolution block feature map is input into a channel attention module CA to obtain a weighted feature map. Finally, the weighted feature map is input into a second 1*1 convolution block to obtain a final output feature map with a size of C*H*W.

[0072] Compared with the prior art, the present application has the following technical effects:

[0073] 1) In order to overcome the problem of feature information dilution in the traditional defogging network and enhance the global and local feature expression capability of the network, the present application proposes an inter-layer multi-scale sequence interaction module IMSIM. The module connects the features of adjacent layers in the encoder, and uses multi-scale convolution and sequence modeling mechanism to capture the cross-layer feature dependency, while fusing multi-scale context information, so as to effectively alleviate the feature dilution problem and improve the image reconstruction quality;

[0074] 2) The present application designs a Fourier domain frequency-space enhancement module FDFSEM based on frequency domain and spatial domain collaborative enhancement. The module combines the frequency domain features extracted by Fourier transform and the local features extracted by spatial convolution, and through collaborative processing of high-frequency details and global structure information, enhances the network's ability to retain edge information and remove artifacts, effectively improving the accuracy and visual effect of image defogging;

[0075] 3) The present application proposes a defogging network framework based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement, which is used for foggy images in complex scenes. By combining cross-layer feature dependency modeling and frequency-space information collaborative enhancement, the present application significantly improves the image defogging performance and generalization ability of the network in multiple scenes. BRIEF DESCRIPTION OF DRAWINGS

[0076] The present application will be further described below in conjunction with the drawings and examples:

[0077] Figure 1 The figure is a structure diagram of the overall network framework of the present application;

[0078] Figure 2 The figure is a structure diagram of the overall network framework of the present application; Figure 1 The figure is a structure diagram of the inter-layer multi-scale sequence interaction module IMSIM;

[0079] Figure 3 The figure is a structure diagram of the overall network framework of the present application; Figure 1A structural diagram of a Fourier domain frequency-space enhancement module FDFSEM. DETAILED DESCRIPTION

[0080] The method for removing fog based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement includes the following steps:

[0081] Step S1: Collect a public fog dataset containing different concentrations of fog, and pre-process the images therein to improve data quality and model training effect.

[0082] Step S2: Construct a network framework for removing fog based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement, which includes a U-shaped fog removal network, an inter-layer multi-scale sequence interaction module IMSIM, and a Fourier domain frequency-space enhancement module FDFSEM.

[0083] Step S3: Train the network using the labeled image dataset, define a suitable loss function, and use an optimizer to train.

[0084] Step S4: Evaluate the fog removal effect using the peak signal-to-noise ratio PSNR and the structural similarity index SSIM, compare with other methods to verify the model performance, optimize the trained model, and deploy it to the actual application for online inference and real-time fog removal.

[0085] Through the above steps, the input feature map can be de-fogged.

[0086] Step S1 is specifically as follows:

[0087] A public fog dataset with different concentrations of fog is collected, and then the images are screened and sorted. Then, the images in the dataset are pre-processed, including image normalization and data enhancement, so as to improve the generalization ability of the model.

[0088] Step S2 is specifically as follows:

[0089] As Figure 1As shown, the dehazing network framework based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement includes a U-shaped dehazing network, an inter-layer multi-scale sequence interaction module IMSIM, and a Fourier domain frequency-space enhancement module FDFSEM. Specifically, in the encoding stage, an input foggy image is gradually extracted multi-scale features through the first encoding layer 100, the second encoding layer 101, and the third encoding layer 102. At the same time, in combination with the inter-layer multi-scale sequence interaction module IMSIM, the features of different encoding layers are modeled and interacted to improve the fusion capability of cross-layer information. In this process, the encoding layer not only captures local details but also extracts global context information to provide high-quality features for the decoding stage. In the encoding stage, for the encoding features corresponding to each decoding layer, the Fourier domain frequency-space enhancement module FDFSEM is introduced to strengthen the expression of high-frequency details through the collaborative processing of the frequency domain and the spatial domain. The optimized features are directly transmitted to the decoding stage through the skip connection to ensure efficient transmission of multi-scale information. In the decoding stage, the spatial resolution of the image is restored through stepwise upsampling. The third decoding layer 103, the second decoding layer 104, and the first decoding layer 105 combine the encoding features and the output of the frequency-space enhancement module to realize feature fusion through residual connection, effectively improving the image detail expression capability. Finally, the network outputs a clear image after dehazing. To optimize the network performance, the network uses a series of loss functions to constrain the pixel consistency, detail restoration capability, and global feature expression of the dehazed image, thereby learning the mapping relationship between the input foggy image and the target clear image, and significantly improving the dehazing effect and image quality.

[0090] In constructing the U-shaped network, the features of the input image are extracted layer by layer through four times downsampling operation, the first encoding layer 100 completes preliminary feature extraction, the second encoding layer 101 further extracts features, and the third encoding layer 102 captures deep semantic features and context information. Each encoding layer first performs feature downsampling through convolution operation to reduce spatial dimensions and extract higher-level features, then normalizes the feature map to ensure consistency and stability, and enhances the non-linear expression capability of the network through the ReLU activation function. The downsampled feature map is further extracted through multiple residual modules 6 to capture more rich semantic information and global context. In the decoding stage, the features extracted by the encoder are gradually restored to the spatial resolution of the image through upsampling operation, among which the third decoding layer 103 restores the global features, the second decoding layer 104 further fuses feature information, and the first decoding layer 105 gradually restores the image details and restores to the spatial size of the input image. Through layer-by-layer upsampling and feature fusion, the dehazed image is finally output, completing the dehazing task.

[0091] As Figure 2As shown, when constructing the inter-layer multi-scale sequence interaction module IMSIM, different encoding layers of the U-shaped dehazing network carry important information, such as shallow features focusing on capturing detailed texture information, and deep features more containing global semantic information such as color and brightness. Therefore, in order to fully utilize the information of different levels of features in the encoding stage, reduce the loss of feature information in the transmission process, enhance the reconstruction effect of the subsequent decoding stage on the dehazed image, optimize the feature interaction and fusion between layers, and further improve the overall dehazing performance of the network, the input feature F k ∈R C×H×W (k∈{i,j}) is first subjected to Layer Normalization operation to standardize the feature distribution and enhance the training stability. Subsequently, the feature F k is sequentially subjected to convolution operations of three different receptive fields, namely 3*3 convolution, 5*5 convolution and 7*7 convolution, to gradually extract local and global context information. The overall feature extraction process can be described as:

[0092] F ms,k =Conv 7×7 (Conv 5×5 Conv 3×3 (LN(F k ))),(k∈{i,j})

[0093] wherein F ms,k ∈R C×H×W represents the feature map obtained after multi-scale convolution operation. The step-by-step operation of the three convolution kernels can capture feature information under different receptive fields, thereby preserving edge details while enhancing global semantic representation. The extracted multi-scale feature F ms,k is converted into a two-dimensional sequence form F seq,k ∈R (H·W)×C to meet the sequence modeling requirements:

[0094] F seq,k =Flatten(F ms,k ),(k∈{i,j})

[0095] The feature sequence is input into the mLSTM module to further optimize the feature representation by capturing long-distance dependencies between sequences, generating updated sequence features F' seq,k ∈R (H·W)×C :

[0096] F' seq,k =mLSTM(F seq,k ),k∈{i,j}

[0097] Through this operation, the module can effectively extract long-distance dependency information across layers, making the relationship between shallow and deep features more closely, laying the foundation for subsequent feature fusion. Then, cross-layer feature relationship calculation and fusion are performed. The relationship matrix between the shallow features F seq,i ∈R (H·W)×C The transpose of the shallow feature F' seq,i T ∈R C×(H·W) and the deep sequence feature F' seq,j ∈R (H·W)×C The relationship matrix between the deep sequence features:

[0098] W = Softmax (F' seq,i T · F' seq,j )

[0099] The weight matrix W ∈R C×C is used to measure the interaction relationship between the two layers of features, capturing the complementary information from different encoding layer features. Using the weight matrix W, the features F' seq,i of the shallow path are weighted and summed to generate the fused features:

[0100] F weighted = F' seq,i · W

[0101] The weighted features F weighted ∈R (H·W)×C are reshaped to restore the original feature map shape F res ∈R C×H×W , and the fused features are further compressed through 1*1 convolution to generate the final module output features F final ∈R C×H×W This operation can integrate multi-scale information across layers while optimizing the spatial and semantic representation of features, providing more accurate and efficient feature support for subsequent defogging decoding.

[0102] The inter-layer multi-scale sequence interaction module IMSIM extracts feature representations under different receptive fields through multi-scale convolution, captures long-distance dependencies across layers using sequence modeling, and further optimizes the inter-layer interaction relationship through feature fusion. This module strengthens the integration of edge details and global semantic information, improves the collaborative ability of shallow and deep features, alleviates the feature dilution problem of traditional U-shaped networks, enhances context modeling, provides comprehensive feature support for defogging tasks, and effectively improves network performance.

[0103] As Figure 3As shown, in the construction of the Fourier domain frequency-space enhancement module FDFSEM, through the collaborative processing of frequency domain and spatial domain features, the module effectively extracts and enhances high-frequency information while retaining rich texture details, providing high-quality multi-dimensional feature support for the decoding stage and significantly improving the dehazing performance.

[0104] In the feature processing stage, the input encoded feature T n ∈R C×H×W is first divided into two parts in the channel dimension, with each part containing C / 2 channels, for frequency domain processing and spatial domain processing respectively. Among them, T p ∈R C / 2×H×W represents the feature allocated to the frequency domain, and T q ∈R C / 2×H×W represents the feature allocated to the spatial domain. In this way, the feature is divided into two independent components in the channel dimension, laying the foundation for subsequent frequency domain and spatial domain feature enhancement. Specifically, it is represented as:

[0105] T p =T n [C / 2,:,:],T q =T n [C / 2,:,:]

[0106] In the Fourier domain processing, the input encoded feature T p is first converted to the frequency domain through two-dimensional fast Fourier transform FFT to obtain the frequency domain feature T freq ∈R C×H×W :

[0107] T freq =FFT(T p )

[0108] The frequency domain feature T freq contains real part Re(T freq )∈R C / 2×H×W and imaginary part Im(T freq )∈R C / 2×H×W . The real part and imaginary part of the frequency domain feature are respectively subjected to 3*3 convolution operation to extract high-frequency information, and combined with Batch Normalization and ReLU activation function to optimize feature expression:

[0109] T freq-enhanced =ReLU(BN(Conv 3×3 (Re(T freq ))))+ReLU(BN(Conv 3×3 (Im(T freq ))))

[0110] Subsequently, the enhanced frequency domain feature Tfreq-enhanced ∈R C×H×W Back to the spatial domain by two-dimensional inverse Fourier transform IFFT, thus generating enhanced features T containing global structure information reconstructed ∈R C / 2×H×W :

[0111] T reconstructed = IFFT(T freq-enhanced )

[0112] Where T reconstucted is the output feature in the frequency domain. In this way, the high-frequency components and global features are fully exploited, and the edges, textures and global consistency in the image are optimized. The spatial domain focuses on extracting local information and edge details of the image. The input encoded feature T q is sequentially passed through two 3*3 convolution operations to extract local information, and is further optimized by BN and ReLU activation functions. The spatial domain retains the edge and texture information in the input feature map, thereby providing a supplement to the details not covered by the frequency domain processing. The specific formula is as follows:

[0113] T spatial = ReLU(BN(Conv 3×3 (ReLU(BN(Conv 3×3 (T q ))))))

[0114] Where T spatial ∈R C / 2×H×W is the output feature of the spatial domain. In this way, the spatial domain provides rich local information for subsequent fusion operations. The enhanced feature T reconsturcted from the frequency domain and the detail feature T spatial from the spatial domain are fused by channel dimension concatenation operation Concat to generate fusion feature T fused ∈R C×H×W . The formula of the fusion operation is as follows:

[0115] T fused = Concat(T reconstructed ,T spatial )

[0116] Where T fusedThe channel dimension is compressed by a 1*1 convolution to improve the compactness and representation ability of the features. output ∈R C×H×W The calculation formula is as follows:

[0117] T output =Conv 1×1 (CA(Conv 1×1 (T fused )))

[0118] Wherein, T outpout represents the output feature of the Fourier domain frequency-space enhancement module FDFSEM.

[0119] Step S3 is specifically as follows:

[0120] The network is trained using the labeled image dataset. First, a suitable loss function is defined, including L1 loss, perception loss, multi-scale structural similarity loss and adversarial loss, to measure the difference between the model prediction result and the true label. Then, the Adam optimizer is selected to adjust the parameters of the network, and the model is continuously optimized through the back propagation algorithm, so as to improve the dehazing performance and accuracy.

[0121] Step S4 is specifically as follows:

[0122] The peak signal-to-noise ratio PSNR and the structural similarity index SSIM are used to evaluate the dehazing effect. PSNR measures the image reconstruction quality, and the higher the value, the higher the similarity between the dehazed image and the real image. SSIM evaluates the structural similarity of the image, and the closer the value is to 1, the closer the structure and visual perception of the generated image to the real image. These indicators can quantitatively analyze the dehazing effect of the model and compare with other methods to verify the advantages of the model in detail recovery, clarity and visual quality. The optimized model will be deployed to the actual application environment to realize real-time dehazing processing.

[0123] Regarding the network constructed in the application, the following is specific:

[0124] In step S2, the dehazing network based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement is constructed, and the structure of the network is specifically as follows:

[0125] As Figure 1As shown, the foggy image with a size of 3xHxW is filled by reflection to preserve the image edge information. Then the image is input into the first encoding layer 100, using a 7*7 convolution kernel with a step size of 1, and passing through normalization and ReLU activation function to generate an initial feature map with a size of 64xHxW;

[0126] The output of the first encoding layer 100 is connected to the input of the second encoding layer 101, and the initial feature map enters the second encoding layer 101, which is completed by a 3*3 convolution kernel with a step size of 2, and the feature map size is reduced from 64xHxW to 128xH / 2xW / 2. The output of the first encoding layer 100 and the output of the second encoding layer 101 are respectively connected to the input of the first multi-scale sequence interaction module 1, which captures the multi-scale interaction information between texture and semantic features.

[0127] The output of the second encoding layer 101 is connected to the input of the third encoding layer 102, which is completed by a 3*3 convolution kernel with a step size of 2, and the feature map size is reduced from 128xH / 2xW / 2 to 256xH / 4xW / 4. The output of the third encoding layer 102 and the output of the first multi-scale sequence interaction module 1 are respectively connected to the input of the second multi-scale sequence interaction module 2, which interacts the obtained feature maps to further integrate the inter-layer multi-scale information.

[0128] The output of the second multi-scale sequence interaction module 2 is connected to the input of a plurality of consecutive residual block modules 6. The residual block module 6 performs deep semantic modeling on the features to extract high-level features while maintaining the stability of information flow. The output of the residual block module 6 is connected to the input of the third decoding layer 103 after the residual fusion of the output of the third Fourier domain space-time enhancement module 5.

[0129] The third decoding layer 103 completes the upsampling by deconvolution operation, and the feature map size is restored from 256xH / 4xW / 4 to 128xH / 2xW / 2. At the same time, the output of the third decoding layer 103 is connected to the input of the second decoding layer 104 after the residual fusion of the output of the second Fourier domain space-time enhancement module 4.

[0130] The feature is upsampled again, and the feature map size is restored from 128xH / 2xW / 2 to 64xHxW, and the output of the second decoding layer 104 is connected to the input of the first decoding layer 105 after the residual fusion of the output of the first Fourier domain space-time enhancement module 3.

[0131] The first decoding layer 105 first adopts a reflection padding ReflectionPad2d to reserve edge information through a final mapping operation, then performs convolution on the feature map through a 7*7 convolution kernel to map the channel number to the target channel number, and finally outputs a clear image after defogging with a resolution of 3*H*W through a Tanh activation function. This process not only restores the resolution, but also ensures the preservation of edge details and the naturalness of the output image.

[0132] The structure of the constructed inter-layer multi-scale sequence interaction module IMSIM is specifically as follows:

[0133] As shown in Figure 2 , first, the first feature map 200 and the second feature map 210 output are connected to the inputs of the first normalization Layer Normalization block and the second normalization Layer Normalization block, respectively, to standardize the feature distribution, enhance the training stability, and reduce the influence of feature scale difference on subsequent calculation.

[0134] Subsequently, the first normalization Layer Normalization block output feature map is input to the first 3*3 convolution block to obtain the first 3*3 convolution block feature map 201, the first 3*3 convolution block feature map 201 is input to the first 5*5 convolution block to obtain the first 5*5 convolution block feature map 202, and the first 5*5 convolution block feature map 202 is input to the first 7*7 convolution block to obtain the first 7*7 convolution block feature map 203. The second normalization Layer Normalization output feature map is input to the second 3*3 convolution block to obtain the second 3*3 convolution block feature map 211, the second 3*3 convolution block feature map 211 is input to the second 5*5 convolution block to obtain the second 5*5 convolution block feature map 212, and the second 5*5 convolution block feature map 212 is input to the second 7*7 convolution block to obtain the second 7*7 convolution block feature map 213. This multi-scale convolution design can capture context semantic information of different scales, making the feature expression more rich and comprehensive.

[0135] For the first 7*7 convolution block feature map 203 and the second 7*7 convolution block feature map 213, they are respectively input to the first Flatten block and the second Flatten block to convert them from the spatial dimension C*H*W to the sequence form, to obtain the first feature sequence 204 and the second feature sequence 114, with the shape of (H*W)*C. Then, the first feature sequence 204 and the second feature sequence 114 are respectively input to the first mLSTM block and the second mLSTM block to obtain the enhanced first sequence feature 205 and the enhanced second sequence feature 215. This process can model the context relationship in the spatial and channel dimensions, thereby strengthening the inter-layer information interaction.

[0136] To capture the interaction between enhanced first sequence feature 205 and enhanced second sequence feature 215, the enhanced first sequence feature 205 is first transposed into the Transpose block to obtain a transposed sequence 220, transforming it from a shape (H·W)×C to C×(H·W). This allows for matrix multiplication with the enhanced second sequence feature 215, which also has a shape of (H·W)×C, to calculate the interaction between them. The result is then input into the Softmaax block to obtain an inter-layer attention weight matrix 221, with a shape of C×C. Subsequently, a weighted sum of the inter-layer attention weight matrix 221 and the enhanced first sequence feature 205 is calculated to obtain a fused sequence feature 222, fully capturing the interaction information between shallow and deep features.

[0137] The fused sequence feature 222 is input to the Reshape block to restore the spatial dimensions of the original feature map, resulting in a fused feature map 223 with a shape of C×H×W. To further integrate channel-dimensional information, the fused feature map 223 is input to a 1*1 convolutional block to obtain the final output feature 224 with a shape of C×H×W. This operation further optimizes the semantic representation and channel expression capabilities of the features, providing high-quality feature support for subsequent modules.

[0138] The specific structure of the constructed Fourier domain frequency-space enhancement module FDFSEM is as follows:

[0139] like Figure 3 As shown, the input feature map 300 of the coding layer has a size of C×H×W. First, the input feature map 300 of the coding layer is input into a Split block to obtain a frequency domain feature map 310 and a spatial domain feature map 320, with each part having C / 2 channels. Among them, the frequency domain feature map 310 is processed in the frequency domain to capture global structural information and high-frequency details; the spatial domain feature map 320 is processed in the spatial domain to enhance local details and texture information.

[0140] The frequency domain feature map 310 after channel equalization is input into a two-dimensional Fast Fourier Transform (FFT-2D) block to obtain a real-virtual combined frequency domain feature map 311. The real-virtual combined frequency domain feature map 311 is connected to the input of a frequency domain 3*3 convolutional block. The output of the frequency domain 3*3 convolutional block is connected to the input of a frequency domain batch normalization block (BN) and a frequency domain ReLU block. The outputs of the frequency domain batch normalization block (BN) and the frequency domain ReLU block are connected to the input of an inverse Fast Fourier Transform (IFFT-2D) block. After processing by the inverse Fast Fourier Transform (IFFT-2D) block, an enhanced frequency domain feature map 312 is obtained, whose size remains C / 2×H×W.

[0141] The channel-equalized spatial domain feature map 320 is connected to the input of a first spatial domain 3*3 convolution block, the output of the first spatial domain 3*3 convolution block is connected to the input of a first spatial domain batch normalization block BN and a first spatial domain ReLU block, the output of the first spatial domain batch normalization block BN and the first spatial domain ReLU block is connected to the input of a second spatial domain 3*3 convolution block, the output of the second spatial domain 3*3 convolution block is connected to the input of a second spatial domain batch normalization block BN and a second spatial domain ReLU block, and an enhanced spatial domain feature map 321 is obtained, which has the same size of C / 2*H*W.

[0142] The frequency domain feature map 312 and the spatial domain feature map 321 are spliced in the channel dimension to form a frequency-space fusion feature map 330, which has a size of C*H*W. This fusion strategy can effectively combine the global modeling ability of the frequency domain and the local optimization ability of the spatial domain, thereby significantly improving the expression effect of high-frequency details. Subsequently, the frequency-space fusion feature map 330 is input into a first 1*1 convolution block to obtain a first 1*1 convolution block feature map 331, so as to improve the compactness and representation ability of the feature. Then, the first 1*1 convolution block feature map 331 is input into a channel attention module CA to obtain a weighted feature map 332. Finally, the weighted feature map 332 is input into a second 1*1 convolution block to obtain a final output feature map 333, which maintains a size of C*H*W.

[0143] In order to better understand the present application for those skilled in the art, further description is as follows:

[0144] 1) Parameter setting

[0145] The experiment uses an NVIDIA RTX 4090 graphics card with 24 GB of video memory, and the code is implemented based on the Pytorch framework with CUDA version 11.8. The optimization strategy uses the Adam optimizer with momentum decay index set to β1=0.9 and β2=0.999. The initial learning rate is 0.001, which is dynamically adjusted using the MultiStepLR strategy, and the batch size is 4. The experiment uses two public datasets, SateHaze1k and HRSD, to evaluate the performance of the model. The SateHaze1k dataset consists of three subsets: SateHaze1k-Thin, SateHaze1k-Moderate, and SateHaze1k-Thick, each containing 320 pairs of training images, 35 pairs of validation images, and 45 pairs of test images, covering different scenes of light, moderate, and heavy haze, aiming to evaluate the performance of the dehazing model under various haze conditions. The HRSD dataset includes two subsets, LHID and DHID. The LHID subset consists of 30,517 images from Google Earth, which are simulated under light haze conditions by adding light haze. The DHID subset contains 14,990 images from the DLR 3k Munich Vehicle Aerial Dataset, which are generated by adding dense haze to simulate heavy haze conditions, suitable for evaluating the performance of dehazing algorithms under strong haze conditions. To evaluate the performance of the model, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as indicators, which are commonly used standards for measuring the quality of dehazed images.

[0146] 2) Experimental results

[0147] To verify the effectiveness of the algorithm, the present invention is compared with the current excellent dehazing algorithms, including DCP, DCRD, FCTF, SGID, MAXIM, DCI-Net, and Trinity-Net, for quantitative comparison. The datasets include SateHaze1k and HRSD.

[0148] Table 1 shows the PSNR and SSIM index results of each dehazing algorithm on the above datasets, and the specific analysis is as follows:

[0149] On the SateHaze1k-Thin dataset, the PSNR of the algorithm reaches 26.74dB, and the SSIM is 0.927, which is significantly better than other algorithms. Compared with the second place DCI-Net, the PSNR and SSIM are improved by 2.27dB and 0.045 respectively, showing excellent ability for light fog removal. On the SateHaze1k-Moderate dataset, the PSNR and SSIM of the algorithm reach 27.65dB and 0.950 respectively, which is also better than other algorithms. Compared with the closest MAXIM, the algorithm shows obvious advantages in detail restoration and global consistency. On the SateHaze1k-Thick dataset, the PSNR of the algorithm is 24.68dB, and the SSIM is 0.882, which is still better than DCI-Ne and MAXIM, proving its robustness in complex fog scenes. On the HRSD-LHID dataset, the PSNR and SSIM of the algorithm reach 28.68dB and 0.887 respectively, which is significantly better than other comparison algorithms. This performance improvement is mainly due to the modeling of cross-layer long-distance dependence by the inter-layer multi-scale sequence interaction module IMSIM, which can effectively capture special hierarchical features in complex scenes. On the HRSD-DHID dataset, the PSNR and SSIM of the algorithm reach 28.23dB and 0.893 respectively, exceeding all comparison methods. This shows that the dehazing method proposed in the application has more excellent detail restoration ability in high dynamic range and complex fog scenes. The algorithm performs well in each test scene, especially in complex thick fog and high dynamic range scenes. With the multi-scale feature interaction capability of the inter-layer multi-scale sequence interaction module IMSIM and the high-frequency information capture capability of the Fourier domain frequency-space enhancement module FDFSEM, the PSNR and SSIM are significantly improved. Compared with SOTA methods, the algorithm shows excellent performance in detail restoration and context information modeling, reaching the SOTA level.

[0150]

[0151] Table 1 Comparison with SOTA methods on SateHaze1k and HRSD datasets

[0152] 3) Ablation experiment

[0153] To evaluate the effectiveness of each module of the application, we designed an ablation experiment according to the framework and the innovations proposed, which includes four experiments: (1) Base represents the U-shaped basic framework, mainly including the encoding layer, six residual blocks and the decoding layer, where the encoding and decoding layers are directly connected through the jump connection. (2) Base+IMSIM module. (3) Base+FDFSEM module. (4) Base+IMSIM module+FDFSEM module.

[0154]

[0155] Table 2 PSNR and SSIM results on SateHaze1k-Moderate dataset

[0156] As can be seen from Table 2, the Base framework has a PSNR of 23.97 dB and an SSIM of 0.844, which reflects its basic performance as a benchmark model. By introducing the inter-layer multi-scale sequence interaction module IMSIM, the PSNR of the model is improved to 26.16 dB, and the SSIM is improved to 0.891, which is improved by 2.19 dB and 0.047, respectively, compared with the benchmark model. This shows that the IMSIM module can effectively model the long-distance dependence relationship of cross-level features, and enhance the comprehensive expression ability of the network to global semantic information and detailed texture. Further, by adding the Fourier domain frequency-space enhancement module FDFSEM to the Base framework, the PSNR and SSIM of the model are improved to 25.39 dB and 0.882, respectively, which is improved by 1.42 dB and 0.038, respectively, compared with the benchmark model. The results verify the effectiveness of the FDFSEM module in capturing high-frequency information and enhancing the expression of spatial features, indicating that it can significantly improve the adaptability of the model to complex scenes. When the IMSIM and FDFSEM modules are introduced at the same time, the model performance reaches the best level, with a PSNR of 27.65 dB and an SSIM of 0.950, which is improved by 3.68 dB and 0.106, respectively, compared with the benchmark model. Compared with the case of introducing IMSIM or FDFSEM alone, the combined use of the two further enhances the ability of feature interaction and multi-scale feature capture, verifying the synergistic advantages and effectiveness of the modules in improving the overall performance of the dehazing network.

Claims

1. A dehazing method based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement, characterized in that, Includes the following steps: Step S1: Collect publicly available hazy datasets containing different concentrations of haze, and preprocess the images to improve data quality and model training performance; Step S2: Construct a dehazing network framework based on inter-layer multi-scale sequence interaction and Fourier domain frequency-space enhancement. This framework includes a U-shaped dehazing network, an inter-layer multi-scale sequence interaction module IMSIM, and a Fourier domain frequency-space enhancement module FDFSEM. Step S3: Train the network using the labeled image dataset, define the loss function, and train using an optimizer; Step S4: Use two metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), to evaluate the dehazing effect, optimize the trained model, and deploy it to real-world applications for online inference and real-time dehazing. The above steps can be used to dehaze the input feature map; In step S2, the construction of the U-shaped dehazing network specifically includes: a coding layer feature extraction module, a residual module (6), and a decoding layer image restoration module with a third decoding layer; the coding layer feature extraction module includes a first coding layer (100), a second coding layer (101), a third coding layer (102), a residual module (6), and a decoding layer image restoration module including a third decoding layer (103), a second decoding layer (104), and a first decoding layer (105); When constructing the inter-layer multi-scale sequence interaction module IMSIM, local and global context information in the features of different coding layers is extracted by multi-scale convolution; at the same time, the sequence modeling mechanism mLSTM is used to capture long-distance dependencies between layers; the features output by the inter-layer multi-scale sequence interaction module IMSIM are further optimized by the residual module (6) to generate aggregated features that integrate edge details and semantic information, and finally passed to the third decoding layer (103). When constructing the Fourier domain frequency-space enhancement module FDFSEM, in the decoding stage, for the corresponding coding layer, frequency domain features are extracted through Fourier transform to enhance high-frequency information, and spatial domain features are extracted by convolution operation to maintain texture integrity; after frequency-space co-processing, the optimized features are passed to the corresponding decoding layer.

2. The method according to claim 1, characterized in that, In step S1, a public dataset of foggy images with different concentrations of haze is collected, and then the images are filtered and organized. Next, the images in the dataset are preprocessed, including image normalization and data augmentation.

3. The method according to claim 1, characterized in that, In the encoding stage, a foggy image is input, and multi-scale features are extracted step by step through the first encoding layer (100), the second encoding layer (101), and the third encoding layer (102). At the same time, the inter-layer multi-scale sequence interaction module IMSIM is used to model and interact the features of different encoding layers. In this process, the encoding layers not only capture local details but also extract global context information, providing high-quality features for the decoding stage. In the encoding stage, for the encoding features corresponding to each decoding layer, the Fourier domain frequency-space enhancement module FDFSEM is introduced to enhance high-frequency details through the collaborative processing of the frequency domain and the spatial domain. The optimized features are directly passed to the decoding stage through skip connections. In the decoding stage, the spatial resolution of the image is restored by progressive upsampling. The third decoding layer (103), the second decoding layer (104), and the first decoding layer (105) combine the encoded features with the output of the frequency-space enhancement module and achieve feature fusion through residual connections. Finally, the network outputs a clear image after dehazing. In order to optimize the network performance, the network adopts a loss function to constrain the pixel consistency, detail restoration ability and global feature expression of the dehazed image, so as to learn the mapping relationship between the input hazy image and the target clear image.

4. The method according to claim 3, characterized in that, in, When constructing the U-shaped network, the features of the input image are extracted layer by layer through a four-fold downsampling operation. The first coding layer (100) completes the initial feature extraction, the second coding layer (101) further extracts features, and the third coding layer (102) captures deep semantic features and contextual information. Each coding layer first performs feature downsampling through convolution operation, then normalizes the feature map, and enhances the nonlinear expression ability of the network through the ReLU activation function. The downsampled feature map is further extracted for deep features through multiple residual modules (6).

5. The method according to claim 3, characterized in that, In the decoding stage, the features extracted by the encoder are gradually restored to the spatial resolution of the image through upsampling operations. The third decoding layer (103) restores the global features, the second decoding layer (104) further fuses the feature information, and the first decoding layer (105) gradually restores the detailed information of the image and restores it to the spatial size of the input image. Through layer-by-layer upsampling and feature fusion, the dehazed image is finally output, and the dehazing task is completed.

6. The method according to any one of claims 3 to 5, characterized in that, When constructing the interlayer multi-scale sequence interaction module IMSIM, for the input features in the encoding stage First, a LayerNormalization operation is performed on the features to standardize the feature distribution and enhance training stability; subsequently, the features... The feature extraction process sequentially extracts local and global contextual information through three convolutional operations with different receptive fields: 3x3, 5x5, and 7x7 convolutions. The overall feature extraction process can be described as follows: ； in, This represents the feature map obtained after multi-scale convolution operations; the hierarchical operation of the three convolution kernels can capture feature information under different receptive fields, thereby enhancing the global semantic representation while preserving edge details; the extracted multi-scale features Transformed into a two-dimensional sequence form To adapt to sequence modeling requirements: ； The feature sequences are input into the mLSTM module, which further optimizes the feature representation by capturing long-range dependencies between sequences, generating updated sequence features. : ； Through this operation, the module can effectively extract long-distance dependency information across layers, making the relationship between shallow and deep features closer; then, it performs cross-layer feature relationship calculation and fusion; the calculation comes from... Transpose of shallow features and Relationship matrix between deep sequence features: ； weight matrix This is used to measure the interaction between two layers of features, capturing complementary information from features from different coding layers; using the weight matrix W, features of the shallow path are... Perform a weighted summation to generate the fused features: ； Weighted features After reshaping, the original feature map shape is restored. The fused features are further compressed using 1x1 convolutions to generate the final module output features. This operation can integrate multi-scale information across layers, while optimizing the spatial and semantic representation of features; The interlayer multi-scale sequence interaction module IMSIM extracts feature representations under different receptive fields through multi-scale convolution, captures long-distance dependencies across layers using sequence modeling, and further optimizes interlayer interaction relationships through feature fusion.

7. The method according to any one of claims 1 to 3, characterized in that, When constructing the Fourier domain frequency-space enhancement module FDFSEM, during the feature processing stage, the input encoded features... First, the features are divided into two equal parts along the channel dimension, with each part containing C / 2 channels, used for frequency domain processing and spatial domain processing respectively; where, This represents the features assigned to the frequency domain. This represents the feature assigned to the spatial domain; in this way, the feature is divided into two independent components along the channel dimension, specifically represented as follows: ； In Fourier domain processing, the encoded features of the input First, the frequency domain features are transformed using a two-dimensional Fast Fourier Transform (FFT) to obtain the frequency domain characteristics. : ； Frequency domain characteristics , including the real part and the virtual part ; Perform 3x3 convolution operations on the real and imaginary parts of the frequency domain features respectively to extract high-frequency information, and combine Batch Normalization and ReLU activation functions to optimize the feature representation: ； Subsequently, enhanced frequency domain features By mapping back to the spatial domain using the two-dimensional inverse Fourier transform (IFFT), enhanced features containing global structural information are generated. : ； in, The frequency domain represents the output features; the spatial domain focuses on extracting local information and edge details of the image; the input represents the encoded features. The local information is extracted through two 3x3 convolution operations, and further optimized using BN and ReLU activation functions. The spatial domain preserves the edge and texture information in the input feature map, thus supplementing the details not covered in the frequency domain. The specific formula is as follows: ； in, These are the output features from the spatial domain; in this way, the spatial domain provides rich local information for subsequent fusion operations; and the enhanced features from the frequency domain... and detailed features of the spatial domain By performing a concat operation on the channel dimension, fused features are generated. The formula for the fusion operation is as follows: ； in, This represents the fused feature map; this fusion method combines global features in the frequency domain and local details in the spatial domain to generate a multi-scale feature representation; the fused features The channel dimension is compressed using a 1x1 convolution, followed by a channel attention module (CA) to further optimize the importance distribution between channels. The optimized features are then further compressed using a 1x1 convolution to generate the final output features. The calculation formula is as follows: ； in, This represents the output characteristics of the Fourier domain frequency-space enhancement module FDFSEM.

8. The method according to claim 1, characterized in that, In step S3, the network is trained using the labeled image dataset. First, loss functions are defined, including L1 loss, perceptual loss, multi-scale structural similarity loss, and adversarial loss, to measure the difference between the model's predictions and the true labels. Then, the Adam optimizer is selected to adjust the network parameters, and the model is continuously optimized through the backpropagation algorithm.

9. The method according to claim 1, characterized in that, In step S4, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the dehazing effect; PSNR measures the image reconstruction quality, and a higher value indicates a higher similarity between the dehazed image and the real image; SSIM evaluates the structural similarity of images; the closer the value is to 1, the closer the generated image is to the structure and visual perception of the real image. These metrics can quantitatively analyze the dehazing effect of the model and compare it with other methods to verify the model's advantages in detail recovery, sharpness, and visual quality. The optimized model will be deployed to real-world application environments to achieve real-time dehazing.

Images