A single image defogging method fusing frequency and content features

Abstract

The present application relates to a kind of fusion frequency and content feature single image defogging method, comprising: first, input image is carried out batch normalization processing, then input to double-flow network containing frequency flow and content flow;Frequency flow uses frequency convolution to extract image high-low frequency feature, and adds the high-low frequency double self-attention mechanism based on Transformer;Content flow is residual embedding structure and extracts image content feature, and adds residual channel attention mechanism;For the problem that image structure information and detail high-frequency information are erased in the defogging process caused by algorithm, it is proposed to accurately extract frequency feature using frequency convolution, and effectively combined with content feature to obtain defogging image finally.The present application can obtain higher quality defogging image for the defogging of single fog image.The present application combines frequency convolution, residual channel attention mechanism and double self-attention mechanism, which can better improve the network defogging performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for dehazing a single image by fusing frequency and content features. Background Technology

[0002] In foggy or hazy weather conditions, dust, smoke, or other floating particles can greatly absorb and scatter light, causing severe degradation of the images acquired by the visual system. This, in turn, affects the detection, tracking, and recognition of targets, making it not only difficult to extract target information from images but also impacting subsequent image processing.

[0003] The purpose of image dehazing is to remove weather-related interference from degraded images, enhance image sharpness and color saturation, and thus maximize the recovery of useful image features. This allows the restored images to be better applied in many computer vision systems such as community monitoring, intelligent transportation, civil aviation assistance, remote sensing observation, and autonomous driving. Therefore, researching how to effectively reconstruct original, clear images from images taken in hazy environments and improve the robustness of vision systems has significant practical implications.

[0004] Image dehazing techniques can be divided into single-frame dehazing and video dehazing. Video sequences consist of consecutive single-frame images, and each frame can be dehazed using single-frame dehazing techniques. Therefore, single-frame dehazing has become a fundamental and popular technique in the field of dehazing. The goal of single-frame dehazing is to obtain a dehazed image from a given hazy image. Existing single-frame dehazing algorithms are mainly divided into two areas: physically based dehazing methods and deep learning-based dehazing methods.

[0005] Dehazing methods based on physical models are generally based on atmospheric scattering models. The dehazing process is ultimately equivalent to estimating multiple unknown parameter variables in the atmospheric scattering model through various prior theories or specific methods, thereby directly deriving a clear image from the fog map.

[0006] With the rise of deep learning, deep learning-based dehazing algorithms have been proposed. Early deep learning-based dehazing algorithms still relied on atmospheric scattering models, making them non-end-to-end dehazing. Cai et al. proposed DehazeNet, which uses a multi-scale convolutional neural network to estimate the atmospheric transmittance map of a foggy image and then inverts the fog-free image based on the atmospheric scattering model. Li et al. proposed the Integrated Dehazing Network Model (AOD-Net), which appropriately deforms the original atmospheric scattering model, turning atmospheric illumination values ​​and light transmittance into a single parameter, thereby reducing the error caused by the accumulation of multiple parameters. End-to-end dehazing algorithms do not rely on atmospheric scattering models but directly learn the correspondence between foggy and clear images, using convolutional neural networks to directly restore the clear, fog-free image from the input foggy image. Engin et al. highlighted a dehazing algorithm based on a GAN network model, combining cycle consistency loss and perceptual loss to enhance the dehazing effect and improve the texture information of the fog-free image. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a single-image dehazing method that integrates frequency and content features. This invention has high operating efficiency, does not require additional training labeled data, and achieves significant dehazing effects while better restoring the color, texture, and other details of the dehazed image.

[0008] The technical solution of this invention is: a method for dehazing a single image by fusing frequency and content features, comprising the following steps:

[0009] S1) Construct a dual-stream neural network architecture that includes a frequency stream and a content stream. The frequency stream is an encoder-decoder architecture consisting of frequency doubling convolutions and residual channel attention blocks. Convolutions are used for downsampling operations, and deconvolutions are used for upsampling operations. A dual self-attention mechanism is used to add and fuse the encoder and decoder path features.

[0010] The content stream is a nested residual structure, including three residual groups. Each residual group consists of three residual channel attention blocks. Each residual channel attention block is constructed using a depthwise separable convolutional layer, a channel attention feature fusion layer, and residual connections.

[0011] S2), for the foggy image X hazy and the corresponding haze-free image Y clean The labels are formatted, and the weight parameters in the dehazing network are trained based on the PyTorch network framework.

[0012] S3) Extract the parameters of the trained neural network and load them into the dual-stream neural network architecture constructed in step S1) to extract image features, calculate the relationships between features, and adjust the data distribution, thereby ensuring the network's dehazing capability.

[0013] S4) There will be a fog test image X hazy In the two-stream neural network architecture input in step S3), Y is obtained respectively. frequency Y content and Y dehaze ;

[0014] Among them, Y dehaze To output the final dehazed image, Y frequency and Y content These are intermediate dehazed images output from the frequency stream and content stream, respectively.

[0015] Preferably, in step S1), frequency doubling convolution is used to extract high and low frequency features in the frequency stream. The frequency doubling convolution divides the ordinary feature map into channels and extracts high and low frequency features respectively. Among them, high frequency features include detailed features such as texture edges.

[0016] Low-frequency features contain the overall structure and contain some redundancy; therefore, low-frequency features are downsampled, reducing their resolution to half that of high-frequency features; frequency doubling convolution is specifically expressed as:

[0017] Y H =Y H→H +Y L→H ;

[0018] Y L =Y L→L +Y H→L ;

[0019] In the formula, Y H Y represents the high-frequency characteristics of the frequency-doubling convolution output. L Y represents the low-frequency characteristics of the frequency-doubling convolution output. H→H and Y L →H Y represents the mapping from input high-frequency features to output high-frequency features and the mapping from input low-frequency features to output high-frequency features, respectively. L→L and Y H→L Let represent the mapping from input low-frequency features to output low-frequency features and the mapping from input high-frequency features to output low-frequency features, respectively. Their specific expressions are as follows:

[0020] Y H→H =f(X) H W H→H );

[0021] Y L→H =Upsampling(f(X) L W L→H ),2);

[0022] Y L→L =f(X) L WL→L );

[0023] Y H→L =f(Pooling(X) H ,2);W H→L );

[0024] In the formula, X H X represents the high-frequency features of the frequency-doubling convolution input. L f(X) represents the low-frequency features of the frequency-doubling convolution input. H W H →H f(X) represents a high-frequency to high-frequency mapping. L W L→L ) represents a low-frequency to low-frequency mapping, Upsampling(·,2) represents an upsampling operation with a sampling factor of 2, and Pooling(·,2) represents a pooling operation with a step size of 2.

[0025] Preferably, in step S1), a channel spatial attention mechanism based on depthwise separable convolution is used to extract frequency features more accurately and fuse them more effectively. First, the feature maps are subjected to global average pooling and global max pooling respectively. Then, channel attention maps are generated through channel attention, and spatial attention maps are generated through spatial attention. Depthwise separable convolution is used instead of traditional convolution, allowing channels to directly use weights. Specifically:

[0026] M c =f(X) a ,X m W c );

[0027] F c =M c *F;

[0028] M s =f(F c W s );

[0029] F s =M s *F c ;

[0030] In the formula, X a X represents the global average pooling feature. m W represents the global max pooling feature. c W represents the convolution kernel for channel attention. s The convolution kernel represents spatial attention, f(·) represents the depthwise separable convolution operation, and M represents the spatial attention kernel. c This represents the generated channel attention map, where F represents the unpooled input feature.c M represents the generated features after channel attention. s Represents a spatial attention map, F s This represents the generated features after spatial attention, and * represents element-wise multiplication.

[0031] Preferably, in step S1), the frequency stream uses a dual self-attention mechanism to enhance high and low frequency features and fuses the corresponding path features of the encoder and decoder.

[0032] To enhance both high- and low-frequency features uniformly, the dual self-attention mechanism first upsamples the low-frequency features, restoring them to the same resolution as the high-frequency features. Then, it performs self-attention calculations to generate a two-channel attention map. After channel splitting, it performs element-wise multiplication with the input high- and low-frequency features respectively, ultimately obtaining the enhanced high- and low-frequency features. The specific expression for the self-attention mechanism is:

[0033] Q = W q I;

[0034] K = W k I;

[0035] V = W v I;

[0036]

[0037] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix of self-attention, respectively, and K... T Let I represent the transpose of the key matrix, and W represent the input features. q W k and W v These are the neural network weights corresponding to the matrices, d k Indicates the feature dimension.

[0038] Preferably, in step S1), the content stream uses a residual nesting structure to extract content features. The residual nesting structure consists of three residual groups with long residual connections, and each residual group consists of three residual channel attention blocks with short residual connections. Through the residual nesting structure, redundant low-frequency information can be filtered out, better restoring content information such as color in the dehazed image.

[0039] Preferably, in step S1), the single-image dehazing method according to claim 1, which fuses frequency and content features, is characterized in that: in step S1), the most basic module of the content stream is a residual channel attention block. The input features are first passed through two layers of 3x3 convolutions, and then through a channel attention block composed of depthwise separable convolutions, accompanied by residual connections, specifically:

[0040] F c =M c *Conv(F in );

[0041] F out =F c +F in ;

[0042] In the formula, F in and F out These represent the input and output features, respectively. Conv(·) represents two 3x3 convolutional layers, and M... c F represents the generated channel attention map. c This represents the features generated through channel attention.

[0043] Preferably, in step S2), all parameters of the dehazing network are obtained through supervised training in the PyTorch framework. The specific training method is as follows:

[0044] Obtaining the foggy image X hazy and the corresponding haze-free image Y clean After labeling, select it as the training set, and define the network optimization objective as:

[0045] arg minΣΘ(F(X hazy ,Q),Y clean );

[0046] In the formula, Q represents all parameters that need to be learned, and X... hazy and Y clean These are the label pairs for foggy and fog-free images in the training dataset; F(X) hazy ,Q) represents the mapping from a foggy image to a fog-free image learned by the neural network, and Θ(·) is the total loss function.

[0047] Preferably, in step S2), the PyTorch framework is used to learn the network parameters with a learning rate of 0.0001. After every 10 iterations, the learning rate becomes 75% of the original rate; the total number of epochs is 100.

[0048] Its total network loss function is as follows:

[0049] L total =L1+λL perceptual ;

[0050] In the formula, L1 is the mean absolute error, L perceptual Let λ be the perceptual loss of VGG, and λ be the weight of the perceptual loss.

[0051] Preferably, in step S3), the weights and biases of the convolutional layer are used to extract image features, and the frequency and content features are verified by visualizing the features output by the frequency stream and content stream; while the parameters in the batch normalization process are used to adjust the data distribution.

[0052] Preferably, in step S4), for the test image X hazy The processing specifically includes the following steps:

[0053] S411) Given a foggy test image X hazy And the already trained weighted dehazing network, first the foggy test image X hazy Normalize the input to a value range of 0-1;

[0054] S412) The normalized input image is passed through a convolutional layer for shallow feature extraction. Then, dual feature streams are extracted through frequency stream and content stream respectively. Frequency features are extracted by frequency doubling convolution and dual self-attention mechanism in the frequency stream, and content features are extracted by residual channel attention in the content stream. After feature extraction and enhancement of salient features, the two feature streams are fused to finally obtain the dehazed image.

[0055] Preferably, step S412) specifically involves:

[0056] The first convolutional layer uses a 3x3 filter to extract shallow features from the input foggy image. The feature map calculation formula for the first layer is as follows:

[0057] X1 = W1X hazy +b1;

[0058] In the formula, W1 and b1 are the weights and biases in the first convolutional layer, respectively, and X... hazy X1 represents the input image with fog, and X2 represents the feature map output from the first convolutional layer.

[0059] After the first convolutional layer extracts shallow features, the features will enter the frequency stream and content stream respectively. The frequency stream first decomposes the ordinary feature map into high- and low-frequency features through a decomposition module, which can be described as:

[0060]

[0061] in, and These represent the high-frequency and low-frequency features obtained by X1 after passing through the feature decomposition module, respectively.

[0062] After the feature decomposition module simply divides the high- and low-frequency features, frequency doubling convolution is used to further extract the high- and low-frequency features effectively and accurately; simultaneously, an encoder-decoder structure is adopted to gradually reconstruct the image. The first layer of the encoder can be represented as:

[0063]

[0064] In the formula, These are the features output by the first layer encoder, Octave(·) is the frequency doubling convolution operation mentioned earlier, and downsampling(·) is the downsampling operation, which is implemented by ordinary convolution with a stride of 2.

[0065] The second layer of the encoder can be represented as:

[0066]

[0067] Similarly, the third and fourth layers of the encoder can be represented as:

[0068]

[0069]

[0070] Accordingly, in the formula, These are the features output by the second, third, and fourth layer encoders, respectively.

[0071] At this point, the encoding process ends, and the decoding process begins. During decoding, in addition to the features from the lower layer, the same-level features from the encoder also need to be fused. These same-level features undergo double self-attention enhancement. The first layer decoding process can be represented as follows:

[0072]

[0073]

[0074] In the formula, Dual_Self(·) represents a dual self-attention mechanism. These represent the enhanced output features of the third-layer encoder, respectively. upsampling(·) indicates an upsampling operation with a sampling factor of 2, and + indicates the addition of the corresponding high-frequency features and the addition of the corresponding low-frequency features. These represent the high-frequency and low-frequency characteristics output from the first layer of decoding, respectively.

[0075] The second layer of the decoder can be represented as:

[0076]

[0077]

[0078] Similarly, the third and fourth layers of the decoder can be represented as:

[0079]

[0080]

[0081]

[0082]

[0083] Accordingly, in the formula, These represent the enhanced output features of the second-layer encoder, the first-layer encoder, and the decomposition module, respectively. These represent the features output by the second, third, and fourth layer decoders, respectively.

[0084] At this point, the decoding process is complete. The frequency stream has effectively extracted high and low frequency features through frequency doubling convolution. To recover the final fog-free image, the high and low frequency components need to be effectively fused. Therefore, the fused feature map generated by the feature fusion module can be represented as:

[0085]

[0086] In the formula, X frequency This represents the final output feature of the frequency stream, and Fusion(·) represents the high- and low-frequency feature fusion module. At this point, the frequency stream feature extraction process is complete.

[0087] The input to the content stream is also the shallow features extracted from the first convolutional layer. The content stream has a nested residual structure, consisting of three residual groups. Each residual group consists of three residual channel attention blocks, for a total of nine residual channel attention blocks. The first residual channel attention block can be represented as:

[0088] F c1 =M c1 *Conv(X1);

[0089] F out1 =F c1 +X1;

[0090] In the formula, X1 represents the shallow features extracted by the first convolutional layer, Conv(·) represents two 3x3 convolutional operations, and M... c1 The channel attention map generated for the first residual channel attention block, F c1 To integrate the features generated by the channel attention maps, F out1 The feature output by the first residual channel attention block through residual connections.

[0091] The second residual channel attention block can be represented as:

[0092] F c2 =M c2 *Conv(F out1 );

[0093] F out2 =F c2 +F out1 ;

[0094] In the formula, F out1 M is the output feature of the first residual channel attention block. c2 The channel attention map generated for the second residual channel attention block, F c2 To integrate the features generated by the channel attention maps, F out2 The feature output by the second residual channel attention block through residual connection.

[0095] Similarly, the representation methods for the attention blocks of the third to ninth residual channels are similar, and the final feature output of the ninth residual channel attention block can be represented as:

[0096] F c9 =M c9 *Conv(F out8 );

[0097] X content =F c9 +F out8 ;

[0098] Among them, F out8 M represents the output feature of the eighth residual channel attention block. c9 The channel attention map generated for the ninth residual channel attention block, F c9 To integrate the features generated by the channel attention maps, X content This is the feature output of the ninth residual channel attention block, which is also the content feature finally extracted from the content stream.

[0099] At this point, both frequency stream and content stream features have been extracted. The features from the two streams need to be fused to obtain the final dehazed image from the network.

[0100] Y dehaze =(W2X frequency +b2)+(W3X content +b3);

[0101] Among them, Y dehaze For the final dehazed image, X frequency and X content W1 and W2 represent the output features of the frequency stream and content stream, respectively. W2 and b2 represent the convolution kernel and bias value when the frequency stream output features are convolved, respectively. W3 and b3 represent the convolution kernel and bias value when the content stream output features are convolved, respectively.

[0102] The beneficial effects of this invention are as follows:

[0103] 1. This invention fully constructs a dual-feature-flow dehazing network, extracting features from different angles to better restore hazy images and preserve details such as color and texture.

[0104] 2. The dual-feature flow dehazing network constructed in this invention can recover the structural information, edge texture, etc. of an image by using frequency features, and can recover the color information of an image by using content features.

[0105] 3. The dual feature stream structure of the present invention can effectively extract frequency and content features. In the frequency stream, a Transformer-based dual self-attention mechanism is added to enhance the high and low frequency features extracted in the encoding stage, and an improved CBAM module is used to achieve effective fusion of high and low frequency components. In the content stream, a residual nested structure is used to filter out redundant low frequency information.

[0106] 4. This invention can better restore background detail information and achieve a cleaner image dehazing effect; in the quantitative comparison of dehazing effect, the two introduced comparison indicators, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), both surpass previous dehazing methods. Attached Figure Description

[0107] Figure 1 This is a schematic flowchart of the defogging method based on the present invention;

[0108] Figure 2 This is a schematic diagram of the dual-stream neural network architecture of the present invention;

[0109] Figure 3 A comparison chart of the scoring of the method provided by this invention and existing algorithms on the SOTS (indoor) dataset;

[0110] Figure 4 The image shows a comparison of the dehazing effects of the method provided by this invention and existing algorithms on synthesized hazy images.

[0111] Figure 5 This is a comparison chart of the dehazing effects of the method of this invention and existing algorithms on real foggy images; Detailed Implementation

[0112] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0113] like Figure 1 As shown, this embodiment provides a single-image dehazing method based on fusion frequency and content features, including the following steps:

[0114] S1) Construct a dual-stream neural network architecture comprising a frequency stream and a content stream. The frequency stream is an encoder-decoder architecture, mainly composed of frequency-doubling convolutions and residual channel attention blocks. Convolutions are used for downsampling, and deconvolutions are used for upsampling. A dual self-attention mechanism is used to fuse the encoder and decoder path features. The content stream is a nested residual structure, including three residual groups. Each residual group consists of three residual channel attention blocks. Each residual channel attention block is constructed using a depthwise separable convolutional layer, a channel attention feature fusion layer, and residual connections. Its structure diagram is shown below. Figure 2 As shown.

[0115] S2) For the dataset containing foggy images X hazy and the corresponding haze-free image Y clean The labels are formatted, and the weight parameters in the dehazing network are trained based on the PyTorch network framework.

[0116] (S201) After obtaining the foggy images and their corresponding fog-free image label pairs, 14,000 synthetic foggy images and their corresponding fog-free labels are selected as the training set, and another 500 image pairs are used as the validation set. The network optimization objective is defined as follows:

[0117] arg min∑Θ(F(X hazy ,Q),Y clean );

[0118] In the formula, Q represents all parameters that need to be learned, and X... hazy and Y clean These are the label pairs for foggy and fog-free images in the training dataset; F(X) hazy ,Q) represents the mapping from a foggy image to a fog-free image learned by the neural network, and Θ(·) is the total loss function.

[0119] The network parameters were learned using the PyTorch framework with a learning rate of 0.0001 and a total of 100 epochs.

[0120] Network total loss function L total Specifically:

[0121] L total =L1+λL perceptual ;

[0122] In the formula, L1 is the mean absolute error, L perceptual Let λ be the perceptual loss of VGG, and λ be the weight of the perceptual loss.

[0123] S3) Extract the parameters of the trained neural network and load them into the dual-stream neural network architecture constructed in step S1) to extract image features, calculate the relationships between features, and adjust the data distribution, thereby ensuring the network's dehazing capability.

[0124] S4) For a foggy image, the test input X hazy Directly input into the network constructed in step S1), Y dehaze The final dehazed image output is as follows:

[0125] The first convolutional layer uses a 3x3 filter to extract shallow features from the input foggy image. The feature map calculation formula for the first layer is as follows:

[0126] X1 = W1X hazy +b1;

[0127] In the formula, W1 and b1 are the weights and biases in the first convolutional layer, respectively, and X... hazy X1 represents the input image with fog, and X2 represents the feature map output from the first convolutional layer.

[0128] After the first convolutional layer extracts shallow features, the features will enter the frequency stream and content stream respectively. The frequency stream first decomposes the ordinary feature map into high- and low-frequency features through a decomposition module, which can be described as:

[0129]

[0130] in, and These represent the high-frequency and low-frequency features obtained by the feature decomposition module for X1, respectively.

[0131] After the feature decomposition module simply divides the high- and low-frequency features, frequency doubling convolution is used to further extract the high- and low-frequency features effectively and accurately; simultaneously, an encoder-decoder structure is adopted to gradually reconstruct the image. The first layer of the encoder can be represented as:

[0132]

[0133] In the formula, These are the features output by the first layer encoder, Octave(·) is the frequency doubling convolution operation mentioned earlier, and downsampling(·) is the downsampling operation, which is implemented by ordinary convolution with a stride of 2.

[0134] The second layer of the encoder can be represented as:

[0135]

[0136] Similarly, the third and fourth layers of the encoder can be represented as:

[0137]

[0138]

[0139] Accordingly, in the formula, These are the features output by the second, third, and fourth layer encoders, respectively.

[0140] At this point, the encoding process ends, and the decoding process begins. During decoding, in addition to the features from the lower layer, the same-level features from the encoder also need to be fused. These same-level features undergo double self-attention enhancement. The first layer decoding process can be represented as follows:

[0141]

[0142]

[0143] In the formula, Dual_Self(·) represents a dual self-attention mechanism. These represent the enhanced output features of the third-layer encoder, respectively. upsampling(·) indicates an upsampling operation with a sampling factor of 2, and + indicates the addition of the corresponding high-frequency features and the addition of the corresponding low-frequency features. These represent the high-frequency and low-frequency characteristics output from the first layer of decoding, respectively.

[0144] The second layer of the decoder can be represented as:

[0145]

[0146]

[0147] Similarly, the third and fourth layers of the decoder can be represented as:

[0148]

[0149]

[0150]

[0151]

[0152] Accordingly, in the formula, These represent the enhanced output features of the second-layer encoder, the first-layer encoder, and the decomposition module, respectively. These represent the features output by the second, third, and fourth layer decoders, respectively.

[0153] At this point, the decoding process is complete. The frequency stream has effectively extracted high and low frequency features through frequency doubling convolution. To recover the final fog-free image, the high and low frequency components need to be effectively fused. Therefore, the fused feature map generated by the feature fusion module can be represented as:

[0154]

[0155] In the formula, X fequency This represents the final output feature of the frequency stream. Fusion(·) represents the high- and low-frequency feature fusion module. At this point, the frequency stream feature extraction process is complete.

[0156] The input to the content stream is also the shallow features extracted from the first convolutional layer. The content stream has a nested residual structure, consisting of three residual groups. Each residual group consists of three residual channel attention blocks, for a total of nine residual channel attention blocks. The first residual channel attention block can be represented as:

[0157] F c1 =M c1 *Conv(X1);

[0158] F out1 =F c1 +X1;

[0159] In the formula, X1 represents the shallow features extracted by the first convolutional layer, Conv(·) represents two 3x3 convolutional operations, and M... c1 The channel attention map generated for the first residual channel attention block, F c1 To integrate the features generated by the channel attention maps, F out1 The feature output by the first residual channel attention block through residual connections.

[0160] The second residual channel attention block can be represented as:

[0161] F c2 =M c2 *Conv(F out1 );

[0162] F out2 =F c2 +F out1 ;

[0163] In the formula, F out1 M is the output feature of the first residual channel attention block. c2 The channel attention map generated for the second residual channel attention block, F c2 To integrate the features generated by the channel attention maps, F out2 The feature output by the second residual channel attention block through residual connection.

[0164] Similarly, the representation methods for the attention blocks of the third to ninth residual channels are similar, and the final feature output of the ninth residual channel attention block can be represented as:

[0165] F c9 =M c9 *Conv(F out8 );

[0166] X content =F c9 +F out8 ;

[0167] Among them, F out8 M represents the output feature of the eighth residual channel attention block. c9 The channel attention map generated for the ninth residual channel attention block, F c9 To integrate the features generated by the channel attention maps, X content This is the feature output of the ninth residual channel attention block, which is also the content feature finally extracted from the content stream.

[0168] At this point, both frequency stream and content stream features have been extracted. The features from the two streams need to be fused to obtain the final dehazed image from the network.

[0169] Y dehaze =(W2X frequency +b2)+(W3X content +b3);

[0170] Among them, Y dehaze For the final dehazed image, X frequency and X content W1 and W2 represent the output features of the frequency stream and content stream, respectively. W2 and b2 represent the convolution kernel and bias value when the frequency stream output features are convolved, respectively. W3 and b3 represent the convolution kernel and bias value when the content stream output features are convolved, respectively.

[0171] To illustrate the advantages of the network proposed in this invention, Figure 3 Line graphs comparing the PSNR and SSIM scores of the method of this invention with those of some existing algorithms on the SOTS (indoor) dataset are provided. SOTS (indoor) is a dataset published by Li et al., and is currently widely used for scoring in the field of individual image dehazing. Figure 3 It can be seen that the method of the present invention obtained the highest PSNR and SSIM values, indicating that the defogging result of the method of the present invention has the highest quality. Figure 4 and Figure 5Comparison images of the dehazing effects of the method of this invention and some existing algorithms on synthetic hazy images and real hazy images are presented. Among them, DCP is the method proposed by He et al., AOD-Net is the method proposed by Li et al., GCA is the method proposed by Chen et al., GDN is the method proposed by Liu et al., MSBDN is the method proposed by Dong et al., and PSD is the method proposed by Chen et al. Through observation... Figure 4 It can be observed that, compared to existing algorithms, the method of this invention achieves the clearest and most accurate restoration of floor details. And... Figure 5 In comparison, DCP and GCA's dehazing results show color deviation; AOD-Net and GDN's dehazing results show artifacts and darker colors; while MSBDN and PSD achieve good dehazing effects, in magnified images of tree trunk details, MSBDN's dehazing result still has obvious fog residue, and PSD's dehazing result has blurry edges on the tree trunk. In contrast, the method of this invention can effectively remove fog from images while preserving more background details, without causing color deviation or edge blurring.

[0172] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A single image defogging method fusing frequency and content features, characterized in that, The method comprises the following steps: S1), a dual-stream neural network architecture comprising a frequency stream and a content stream is constructed, wherein the frequency stream is a coding-decoding architecture composed of frequency multiplication convolution and residual channel attention block, convolution is used for downsampling operation, inverse convolution is used as upsampling operation, and double self-attention mechanism is used to add and fuse the encoder and decoder path features; The content stream is a residual nested structure comprising three residual groups, each residual group comprising three residual channel attention blocks, each residual channel attention block comprising a depth separable convolution layer, a channel attention feature fusion layer and a residual connection; S2), to a foggy image and a corresponding fog-free image The tag pair is formatted and based on The network framework trains the weight parameters in the defogging network; S3), the trained network parameters are extracted and loaded into the dual-stream neural network architecture constructed in step S1) to extract image features, calculate the mutual relationship between the features, and adjust the data distribution, thereby ensuring the network's ability to remove fog; S4), the foggy image The double-stream neural network architecture of input step S3) is tested, and the output is obtained respectively and ; wherein, is the final corresponding dehazed image output, and are intermediate dehazed images for the frequency stream and content stream outputs, respectively. For processing of the test image comprises the following steps: S411), a given fog test image and having loaded the trained weight dehazing network, first input the fog test image Xhazy into the input normalization to the 0-1 numerical range; S412), the input normalized image is subjected to shallow feature extraction through a convolution layer, then dual feature streams are extracted through the frequency stream and the content stream, frequency features are extracted in the frequency stream with the aid of frequency multiplication convolution and double self-attention mechanism, content features are extracted in the content stream with the aid of residual channel attention, after feature extraction and enhancement of significant features, the two feature streams are fused to finally obtain a defogged image; In step S1), the frequency stream uses frequency multiplication convolution to extract high and low frequency features; the frequency multiplication convolution divides the ordinary feature map by channel to extract high and low frequency features; wherein the high frequency features contain texture edge detail features, and the low frequency features contain overall structure, which is redundant; therefore, the low frequency features are subjected to downsampling, and the resolution becomes half of that of the high frequency features; the frequency multiplication convolution is specifically expressed as: ； wherein, represents a high frequency feature of the frequency multiplied convolution output, represents a low frequency feature of the frequency multiplied convolution output, and represent a mapping from an input high frequency feature to an output high frequency feature and a mapping from an input low frequency feature to an output high frequency feature, respectively, and represent a mapping from an input low frequency feature to an output low frequency feature and a mapping from an input high frequency feature to an output low frequency feature, respectively; and the specific expressions are as follows: ； ； ； ； wherein, represents high frequency features of the upsampled input, represents low frequency features of the upsampled input, represents a mapping from high frequency to high frequency, represents a mapping from low frequency to low frequency, represents an upsample operation with a sampling factor of 2, represents a pooling operation with a stride of 2.

2. The single image defogging method of claim 1, wherein: In step S1), a channel spatial attention mechanism based on a depth separable convolution is used, which is specifically: first, the feature map is subjected to global average pooling and global maximum pooling, then a channel attention map is generated through channel attention, a spatial attention map is generated through spatial attention, and a depth separable convolution is used in the middle, so that the channel directly uses weights, and the specific expression is: wherein, denotes the global average pooling feature, denotes the global max pooling feature, denotes the convolution kernel of channel attention, denotes the convolution kernel of spatial attention, denotes the depthwise separable convolution operation, denotes the generated channel attention map, denotes the input feature without pooling, denotes the generated feature after channel attention, denotes the spatial attention map, denotes the generated feature after spatial attention, denotes the element-wise product.

3. The single image defogging method of fusing frequency and content features according to claim 1, characterized in that: In step S1), the frequency stream uses a double self-attention mechanism to enhance high and low frequency features and fuse the corresponding path features of the encoder and the decoder; In order to uniformly enhance the high and low frequency features, the double self-attention mechanism first upsamples the low frequency features to restore them to the same resolution as the high frequency features, then performs self-attention mechanism calculation to generate an attention map with a channel number of 2, performs channel splitting, and finally performs element multiplication operation with the input high and low frequency features to obtain enhanced high and low frequency features, and the specific expression of the self-attention mechanism is: where Q, K and V are the query matrix, key matrix and value matrix of self-attention, respectively, denotes the transpose of the key matrix, I denotes the input feature, , and are the neural network weights of the corresponding matrices, respectively, denotes the feature dimension.

4. The single image defogging method of claim 1, wherein: In step S1), the content stream uses a residual nested structure to extract content features, the residual nested structure comprises three residual groups with long residual connections, and each residual group comprises three residual channel attention blocks with short residual connections, and through the residual nested structure, redundant low frequency information can be filtered out to better restore the color and texture of the defogged image.

5. The method of claim 1, wherein the method is a fusion frequency and content feature based single image defogging method. In step S1), the most basic module of the content stream is a residual channel attention block. The input features are first passed through two layers of 3x3 convolution, and then through channel attention composed of deep separable convolution, accompanied by residual connection in the middle. Specifically, the residual channel attention block can be represented as: wherein, and denote input and output features, respectively, and Conv(·) denotes a two-layer 3x3 convolution operation, denotes the generated channel attention map, denotes the feature generated by channel attention.

6. The single image defogging method of claim 1, wherein: In step S2), all parameters of the dual-stream neural network architecture are learned through supervised training under the pytorch framework. The specific training method is as follows: After obtaining the foggy image and its corresponding fog-free image label pair, select it as the training set, and define the network optimization objective as: ； In the formula, Q represents all parameters to be learned, and are a foggy image and a non-foggy image label pair in the training data set, respectively; represents a mapping from a foggy image to a non-foggy image learned by the neural network, is a total loss function.

7. The method of claim 1, wherein: In step S2), the network parameter learning is performed using the pytorch network framework, the learning rate is 0.0001, and after every 10 iterations, the learning rate becomes 75% of the original; the total epoch number is 100 times; The network total loss function Specifically: ； wherein is the mean absolute error, is the VGG perceptual loss, and λ is the perceptual loss weight.

8. The single image defogging method of fusing frequency and content features according to claim 1, characterized in that: The weights and bias values of the convolutional layer are used to extract image features. By visualizing the features output by the frequency stream and the content stream, it can be verified whether the frequency features and content features are accurately extracted. The parameters in the batch normalization process are used to adjust the data distribution.

9. The single image defogging method of claim 1, wherein: In step S4) the processing of the foggy test image comprises in particular the following steps: The first convolutional layer uses a 3x3 filter to extract the shallow features of the input foggy image. The feature map calculation formula of the first layer is as follows: ； wherein, and are weight and bias values in the first layer convolution layer, respectively, denotes an input foggy image, denotes a feature map of the first layer convolution output; After extracting the shallow features through the first convolutional layer, the features will enter the frequency stream and the content stream. The frequency stream first passes through the decomposition module to decompose the ordinary feature map into high and low frequency features. Specifically, the decomposition module can be represented as: ； wherein, and respectively represent high frequency features and low frequency features obtained by the eigen decomposition module; After the feature decomposition module simply divides the high and low frequency features, frequency convolution is used to further extract the high and low frequency features. At the same time, the encoder-decoder structure is adopted to gradually recover the image. The first layer of the encoder can be represented as: ； wherein are the features output by the first layer encoder respectively, Octave(·) is an octave convolution operation, and downsampling(·) is a downsampling operation implemented by a normal convolution with a stride of 2. The second layer of the encoder can be represented as: ； Similarly, the third and fourth layers of the encoder can be represented as: ； ； Correspondingly, in the formula, are respectively the features output by the second layer encoder, the third layer encoder, and the fourth layer encoder. At this point, the encoding process is complete, and the decoding process begins. During decoding, in addition to the features of the lower layer, the same level features of the encoder also need to be fused. The first layer of the decoding process can be represented as: ； ； In the formula, denotes a double self-attention mechanism, , respectively denote the enhanced third layer encoder output features, denotes an up-sampling operation with a sampling factor of 2, + denotes the addition of the corresponding high-frequency features, and the addition of the corresponding low-frequency features, respectively denote the high-frequency features and the low-frequency features output by the first layer decoding process; The second layer of the decoder can be represented as: ； ； Similarly, the third and fourth layers of the decoder can be represented as: ； ； ； ； Correspondingly, in the formula, respectively represent the output features of the enhanced second layer encoder, the first layer encoder, and the decomposition module, respectively represent the output features of the second layer decoder, the third layer decoder, and the fourth layer decoder. At this point, the decoding process is complete. The frequency stream extracts high and low frequency features through frequency convolution. In order to recover the final fog-free image, it is necessary to fuse the high and low frequency components. Therefore, the high and low frequency components pass through the feature fusion module to generate the fused feature map, which can be represented as: ； In the formula, represents the characteristics of the final output of the frequency stream, Fusion(·) represents a high-low frequency feature fusion module, and thus the frequency stream feature extraction process is completed; The input of the content stream is also the shallow features extracted by the first convolutional layer. The content stream is a residual nested structure composed of three residual groups. Each residual group is composed of three residual channel attention blocks, a total of nine residual channel attention blocks. The first residual channel attention block can be represented as: In the formula, is the shallow feature extracted by the first layer of convolution, Conv(·) is a two-layer 3x3 convolution operation, is the channel attention map generated by the first residual channel attention block, is the feature generated by fusing the channel attention map, is the feature output by the first residual channel attention block through the residual connection. The second residual channel attention block can be represented as: ； ； In the formula, is the output feature of the first residual channel attention block, is the channel attention map generated by the second residual channel attention block, is the feature generated by fusing the channel attention map, is the feature output by the second residual channel attention block through the residual connection. Similarly, the third to ninth residual channel attention blocks can be represented in a similar manner. The feature output by the ninth residual channel attention block can be represented as: ； ； wherein, is the output feature of the eighth residual channel attention block, is the channel attention map generated for the ninth residual channel attention block, is the feature generated by fusing the channel attention maps, is the feature output by the ninth residual channel attention block, which is also the content feature finally extracted by the content stream. At this point, the frequency stream features and the content stream features have been extracted. The features of the dual stream need to be fused to obtain the final dehazing image of the network. ； wherein, is the final defogged image, and are output features of the frequency stream and the content stream, respectively, , denote the convolution kernel and the bias value when the frequency stream output feature is convolved, respectively, , denote the convolution kernel and the bias value when the content stream output feature is convolved, respectively.

Images