Fuzzy-fogging image restoration method based on multi-channel frequency domain modulation

Abstract

The invention discloses a fuzzy combined fog image restoration method based on multichannel frequency domain modulation, and the method is characterized in that a fuzzy combined fog data set is constructed, and a part of fuzzy combined fog images are randomly selected from the fuzzy combined fog data set as a training set; carrying out migration training on the fuzzy combined fog restoration network through the training set of the public data set GoPro and the obtained training set to obtain a trained fuzzy combined fog restoration network; inputting a to-be-restored blurred and fogged image into the trained blurred and fogged restoration network to obtain a restored image; the fuzzy fog recovery network comprises first to sixth convolutional layers, three encoders, three decoders and an output layer; the method has the advantages that the restoration effect of the blurred combined fog image is verified in a blurred combined fog data set GoPro-haze, the PSNR of the method is 26.7 dB and is improved by 6.27 dB compared with an MAXIM algorithm, the SSIM of the method is 0.904 and is improved by 0.15 compared with the MAXIM algorithm, and therefore the restoration effect of the blurred combined fog image is improved.

Description

Technical Field

[0001] The invention relates to an image restoration method, in particular to a fuzzy-fogging image restoration method based on multi-channel frequency domain modulation. Background Art

[0002] Harsh environments or physical limitations of cameras can reduce the quality of acquired images and further affect the robustness of downstream tasks. Image restoration can effectively eliminate various degradations (such as fog, blur, and snowflakes), and plays an important role in autonomous driving, remote sensing monitoring, and diving operations. Current image restoration mainly focuses on single degradation problems, and blur and fog are two common forms of image degradation. There are a large number of scenarios in reality where they co-occur, such as intelligent navigation in low-temperature storage environments and surveillance shooting in bad weather.

[0003] In recent years, the field of image restoration has achieved rapid development with the advancement of deep learning. By introducing a variety of network design methods, such as dilated convolution, residual learning, U-Net, multi-channel, multi-scale and attention mechanisms, a large number of methods based on convolutional neural networks (CNNs) have been developed for various image restoration tasks. However, in image restoration tasks, the pixels are closely connected, which places extremely high demands on the receptive field of the model. The above methods have limited receptive fields and poor long-range dependencies, resulting in limited effects and difficulty in adapting to complex image degradation tasks in actual environments. Recently, many researchers have designed new image restoration methods based on Transformer, and have made progress in image motion deblurring and image dehazing. However, this method mainly performs restoration in the spatial domain, does not fully utilize the frequency characteristics of the image, and has limited effect on restoring image details.

[0004] Some image restoration methods use wavelet transform or Fourier transform to decompose features into different frequency components, and reconstruct the corresponding features by processing each frequency part separately. However, the decoupling method of this type of method is fixed, frequency information cannot be processed flexibly, and the computational complexity is high. To this end, some works have achieved good results by separating frequency features based on variable filters and modulating them separately. However, compared with single degradation, the composite degradation of blur and fog is more significant in the loss of high-frequency information and the range of low-frequency degradation is wider, which requires higher feature extraction ability and receptive field of the algorithm. The above methods ignore the extraction of deep frequency domain information and do not combine the relationship between multiple channels of the image well. The effect of restoring color images in actual environments is average. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a blurred-fogged image restoration method based on multi-channel frequency domain modulation, which improves the restoration effect of blurred-fogged images.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a fuzzy-fogging image restoration method based on multi-channel frequency domain modulation, comprising the following steps:

[0007] Step ①, construct a fuzzy-fogging dataset, and randomly select some fuzzy-fogging images from the fuzzy-fogging dataset as a training set;

[0008] Step ②, performing migration training on the blur-fog restoration network using the training set of the public dataset GoPro and the training set obtained in step ①, and obtaining a trained blur-fog restoration network;

[0009] Step ③, inputting the blurred-fogged image to be restored into the trained blurred-fogged restoration network to obtain a restored image;

[0010] The blur-fog restoration network includes the first to sixth convolutional layers, three encoders with the same structure and function, three decoders with the same structure and function, and an output layer.

[0011] The first convolutional layer is used to extract features from the input image to obtain first shallow features;

[0012] The first encoder is used to extract the first shallow feature to obtain the first deep feature;

[0013] The second convolutional layer is used to extract features from the cascaded first deep features and the input image to obtain second shallow features;

[0014] The second encoder is used to extract the second shallow features to obtain the second deep features;

[0015] The third convolutional layer is used to extract features from the cascaded second deep features and the input image to obtain third shallow features;

[0016] The third encoder is used to extract the third shallow feature to obtain the third deep feature;

[0017] The first decoder is used to decode the third deep feature to obtain a first restored feature;

[0018] The fourth convolutional layer is used to perform a convolution operation on the cascaded first recovery feature and the second deep feature to obtain a first adjustment feature;

[0019] The second decoder is used to decode the first adjusted feature to obtain a second restored feature;

[0020] The fifth convolutional layer is used to perform a convolution operation on the cascaded second restored features and the first deep features to obtain a second adjusted feature;

[0021] The third decoder is used to decode the second adjusted feature to obtain a third restored feature;

[0022] The sixth convolutional layer is used to extract the third restored features to obtain the final restored features;

[0023] The output layer is used to add the input image and the final restored feature to obtain a restored image and output it;

[0024] The encoder and the decoder are both provided with a ResBlock for enhancing frequency information and a SFBlock for extracting characteristic channel-related information and modulating frequency domain information.

[0025] Compared with the prior art, the advantage of the present invention is that the blur-fog restoration network includes the first to sixth convolutional layers, three encoders with the same structure and function, three decoders with the same structure and function, and an output layer. The first to third convolutional layers are used to extract features, and the fourth to sixth convolutional layers are used for convolution operations to adjust the number of channels and feature scales, ensuring efficient transmission and processing of information between layers. The encoder is used for feature extraction, obtaining local features and processing them. The decoder restores low-resolution features to the same resolution as the input image and combines the features extracted by the encoder to enhance the detail performance of the image. ResBlock enhances frequency information, and SFBlock promotes the modulation of different frequency information. , thereby improving the restoration effect of blurred and fogged images, so that the present invention can adaptively process different degrees of blur and fog, and can perform well in a variety of complex scenes; by verifying the restoration effect of blurred and fogged images in the blurred and fogged data set GoPro-haze, it is shown that the PSNR of the present invention is 26.7dB, which is 6.27dB higher than that of the MAXIM algorithm and 3.9dB higher than that of the Restormer algorithm. The SSIM of the present invention is 0.904, which is 0.15 higher than that of the MAXIM algorithm and 0.126 higher than that of the Restormer algorithm. Therefore, the present invention improves the restoration effect of blurred and fogged images.

[0026] Furthermore, the specific operation process of step ① is as follows:

[0027] According to the atmospheric fog model, random fog is added to each blurred image in the public dataset GoPro to obtain the corresponding blurred-fogged image, and a blurred-fogged dataset is constructed. 2 / 3 blurred-fogged images are randomly selected from the blurred-fogged dataset to form a training set.

[0028] The atmospheric fog model is I(x)=J(x)t(x)+A(1-t(x)), where I(x) represents a blurred-fogged image, J(x) represents a blurred image, A represents global atmospheric illumination, and t(x) represents transmittance. When the atmosphere is uniform, t(x)=e -βd(x) , β represents the scattering coefficient of the atmosphere, d(x) represents the depth of field, β ranges from 0.02 to 0.08, and A ranges from 0.4 to 0.8.

[0029] Furthermore, the specific operation process of step ② is: preprocessing the training set of the public data set GoPro and the training set obtained in step ①, inputting the preprocessed training set of the public data set GoPro into the fuzzy-fogging restoration network, and using the Adam optimizer to iteratively optimize the fuzzy-fogging restoration network according to the loss function, until it reaches half of the preset number of iterations, then stopping the iterative process to obtain the pre-trained fuzzy-fogging restoration network, and inputting the pre-processed training set obtained in step ① into the pre-trained fuzzy-fogging restoration network, and using the Adam optimizer to iteratively optimize the pre-trained fuzzy-fogging restoration network according to the loss function, until it reaches half of the preset number of iterations, then stopping the iterative process to obtain the trained fuzzy-fogging restoration network.

[0030] Furthermore, the preprocessing is: firstly cropping the size of each image in the training set to 256×256, and then horizontally flipping each image with a probability of 0.5;

[0031] The loss function is l = l c +λl f , where l c represents the spatial domain loss function, X represents the restored image output by the blur-haze restoration network, Y represents the corresponding reference image, i represents the index of the restored image, S represents the number of normalized elements, ||·|| 1 represents the absolute value error, l f represents the frequency domain loss function, represents fast Fourier transform, λ represents hyperparameter, λ=0.1;

[0032] The preset number of iterations is 2000-2500 rounds.

[0033] Furthermore, the convolution kernel sizes of the first, second, third and sixth convolution layers are 3×3, the convolution kernel sizes of the fourth and fifth convolution layers are 1×1, and the encoder and the decoder are both composed of L ResBlocks and one SFBlock.

[0034] Furthermore, L=16.

[0035] Furthermore, the ResBlock includes two convolutional layers with a convolution kernel size of 3×3 for feature extraction and a dual-path selective frequency module;

[0036] The specific operation process of the dual-path selective frequency module is as follows:

[0037] The input features are segmented along the channel to obtain global features and window features;

[0038] Performing global average pooling on the global feature to obtain first low-frequency information, subtracting the first low-frequency information from the global feature to obtain first high-frequency information, modulating the first low-frequency information and the first high-frequency information by a first learnable weight, and adding the modulated first low-frequency information to the first high-frequency information to obtain a global output feature;

[0039] The window features are divided into four window branch features by data rearrangement;

[0040] Performing global average pooling on the first window branch feature to obtain second low-frequency information, subtracting the second low-frequency information from the first window branch feature to obtain second high-frequency information, modulating the second low-frequency information and the second high-frequency information through a second learnable weight, and adding the modulated second low-frequency information and the second high-frequency information to obtain the first modulated feature;

[0041] Performing global average pooling on the second window branch feature to obtain the third low-frequency information, subtracting the third low-frequency information from the second window branch feature to obtain the third high-frequency information, modulating the third low-frequency information and the third high-frequency information through the third learnable weight, and adding the modulated third low-frequency information to the third high-frequency information to obtain the second modulated feature;

[0042] Performing global average pooling on the third window branch feature to obtain fourth low-frequency information, subtracting the fourth low-frequency information from the third window branch feature to obtain fourth high-frequency information, modulating the fourth low-frequency information and the fourth high-frequency information through a fourth learnable weight, and adding the modulated fourth low-frequency information to the fourth high-frequency information to obtain the third modulated feature;

[0043] Performing global average pooling on the fourth window branch feature to obtain fifth low-frequency information, subtracting the fifth low-frequency information from the fourth window branch feature to obtain fifth high-frequency information, modulating the fifth low-frequency information and the fifth high-frequency information through a fifth learnable weight, and adding the modulated fifth low-frequency information to the fifth high-frequency information to obtain a fourth modulated feature;

[0044] The first modulated feature, the second modulated feature, the third modulated feature and the fourth modulated feature are concatenated to obtain a window output feature;

[0045] The global output features and the window output features are concatenated to obtain the output features of the dual-path selective frequency module. Since the degradation degree and size of fogging and motion blur are different in different images, a larger receptive field can effectively collect the degradation information in the image, which plays a vital role in image restoration. Therefore, the dual-path selective frequency module is used to effectively expand the receptive field of the blur-fogging restoration network. The dual-path selective frequency module has two branches with different receptive fields, namely the global branch and the window branch, which respectively obtain high- and low-frequency information, and use learnable weights to modulate the high- and low-frequency information, effectively expanding the receptive field. In addition, the dual-path selective frequency module does not use convolutional layers to complete frequency decoupling and modulation, has fewer parameters and has little impact on network complexity.

[0046] Furthermore, the SFBlock includes two convolutional layers with a convolution kernel size of 3×3 for feature extraction and a multi-channel frequency domain attention module;

[0047] The specific operation process of the multi-channel frequency domain attention module is as follows:

[0048] The dynamic weight W is obtained by the basic convolutional network for the input feature X, W = Tanh (BN (f 1×1 (GAP(X)))), where C represents the number of channels, H represents the height, W represents the width, GAP(·) represents the global average pooling operation, and f 1×1 (·) represents a 1×1 convolution operation, BN(·) represents a batch normalization operation, Tanh(·) represents a hyperbolic tangent function, k represents the range of the channel;

[0049] Will Reshape According to the reshaped dynamic weights, each channel in the input feature X is interactively adjusted through channel attention to obtain the adjusted feature X′, where the cth channel of the adjusted feature X′ is recorded as c represents the channel index, 1≤c≤C, W c，j represents the value of the cth row and jth column in the reshaped dynamic weight, offset(c, j) represents the relative offset from channel c to its adjacent channel j, X c+offset(c，j) Indicates the channel after offset;

[0050] The adjusted feature X′ is convolved once, twice, three times and four times with a convolution kernel size of 3×3 to obtain four high-frequency information F m , m∈{1,2,3,4};

[0051] The four high-frequency information are added together and sequentially passed through the global average pooling operation and the full connection operation to obtain the intermediate frequency attention weight;

[0052] The intermediate frequency attention weights are fully connected through four fully connected layers to obtain four preliminary frequency attention weights W m ;

[0053] The four preliminary frequency attention weights are concatenated and normalized by the Soffmax function, and the results after the normalization operation are segmented to obtain four final attention weights W m ′;

[0054] Multiply the four high-frequency information with the corresponding final attention weights respectively to obtain four multiplied results;

[0055] The four multiplication results are added together and a convolution operation with a convolution kernel size of 1×1 is performed to obtain the attention frequency map as the output of the multi-channel frequency domain attention module. The multi-channel frequency domain attention module first effectively integrates the useful information on different channel dimensions through the channel attention method, and then further uses the implicit frequency domain attention method to amplify the frequency characteristics of high-frequency elements using a convolution operator with a convolution kernel size of 3×3, extract information at different levels, and aggregate useful frequency components for image reconstruction. It not only effectively combines spatial domain and frequency domain information, but also the number of convolution layers of its frequency domain feature extraction method can be flexibly adjusted to cope with different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0057] Figure 2 Partially blurred images and corresponding blurred-fogged images in the GoPro dataset disclosed in the present invention;

[0058] Figure 3 Schematic diagram of the structure of the fuzzy-fogging restoration network in the present invention;

[0059] Figure 4 A schematic diagram of the structure of an encoder or a decoder in the present invention;

[0060] Figure 5 A schematic diagram showing a comparison of the effects of using the present invention and different existing algorithms for image restoration in the GoPro-haze dataset;

[0061] Figure 6 This is a schematic diagram of the comparison of intermediate features when the blurry-fogging validation set is processed using the present invention and the SFNet method. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below with reference to the accompanying drawings.

[0063] like Figure 1 As shown, a fuzzy-fogging image restoration method based on multi-channel frequency domain modulation includes the following steps:

[0064] Step ①, construct a fuzzy-fogging dataset, and randomly select some fuzzy-fogging images from the fuzzy-fogging dataset as a training set;

[0065] The specific operation process of step ① is as follows:

[0066] According to the atmospheric fog model, random fog is added to each blurred image in the public dataset GoPro to obtain the corresponding blurred-fogged image, and a blurred-fogged dataset is constructed. 2 / 3 blurred-fogged images are randomly selected from the blurred-fogged dataset to form a training set; Figure 2 As shown, the first row of images are partially blurred images in the public dataset GoPro, and the second row of images are the corresponding blurred-fogged images after adding random fog;

[0067] The atmospheric fog model is I(x)=J(x)t(x)+A(1-t(x)), where I(x) represents the blurred-fogged image, J(x) represents the blurred image, A represents the global atmospheric illumination, and t(x) represents the transmittance. When the atmosphere is uniform, t(x)=e -βd(x) , β represents the scattering coefficient of the atmosphere, d(x) represents the depth of field, β ranges from 0.02 to 0.08, and A ranges from 0.4 to 0.8;

[0068] Step ②, performing migration training on the blur-fog restoration network using the training set of the public dataset GoPro and the training set obtained in step ①, and obtaining a trained blur-fog restoration network;

[0069] The specific operation process of step ② is as follows: preprocessing the training set of the public data set GoPro and the training set obtained in step ①, inputting the preprocessed training set of the public data set GoPro into the fuzzy-fogging restoration network, and iteratively optimizing the fuzzy-fogging restoration network using the Adam optimizer according to the loss function until half of the preset number of iterations is reached, stopping the iterative process to obtain the pre-trained fuzzy-fogging restoration network, and inputting the pre-processed training set obtained in step ① into the pre-trained fuzzy-fogging restoration network, and iteratively optimizing the pre-trained fuzzy-fogging restoration network using the Adam optimizer according to the loss function until the preset number of iterations is reached, stopping the iterative process to obtain the trained fuzzy-fogging restoration network;

[0070] The preprocessing is as follows: first, the size of each image in the training set is cropped to 256×256, and then each image is horizontally flipped with a probability of 0.5;

[0071] The loss function is l=l c +λl f , where l c represents the spatial domain loss function, X represents the restored image output by the blur-haze restoration network, Y represents the corresponding reference image, i represents the index of the restored image, S represents the number of normalized elements, ||·|| 1 represents the absolute value error, l f represents the frequency domain loss function, represents fast Fourier transform, λ represents hyperparameter, λ=0.1;

[0072] The preset number of iterations is 2000-2500 rounds

[0073] Step ③, inputting the blurred-fogged image to be restored into the trained blurred-fogged restoration network to obtain a restored image;

[0074] like Figure 3 As shown in FIG. 1 , the blur-fog restoration network includes the first to sixth convolutional layers, three encoders with the same structure and function, three decoders with the same structure and function, and an output layer.

[0075] The first convolutional layer is used to extract features from the input image and obtain the first shallow features;

[0076] The first encoder is used to extract the first shallow feature to obtain the first deep feature;

[0077] The second convolutional layer is used to extract features from the cascaded first deep features and the input image to obtain the second shallow features;

[0078] The second encoder is used to extract the second shallow features to obtain the second deep features;

[0079] The third convolutional layer is used to extract features from the cascaded second deep features and the input image to obtain the third shallow features;

[0080] The third encoder is used to extract the third shallow feature to obtain the third deep feature;

[0081] The first decoder is used to perform a decoding operation on the third deep feature to obtain a first restored feature;

[0082] The fourth convolutional layer is used to perform a convolution operation on the cascaded first restored feature and the second deep feature to obtain a first adjusted feature;

[0083] The second decoder is used for decoding the first adjusted feature to obtain a second restored feature;

[0084] The fifth convolutional layer is used to perform a convolution operation on the cascaded second restored features and the first deep features to obtain a second adjusted feature;

[0085] The third decoder is used for decoding the second adjusted feature to obtain a third restored feature;

[0086] The sixth convolutional layer is used to extract the third restored features to obtain the final restored features;

[0087] The output layer is used to add the input image to the final restored features to obtain the restored image and output it;

[0088] In this embodiment, the convolution kernel size of the first, second, third and sixth convolution layers is 3×3, the convolution kernel size of the fourth and fifth convolution layers is 1×1, and the encoder and decoder are composed of L ResBlocks (Residual Blocks) and one SFBlock (Selective Frequency Block); L=16;

[0089] like Figure 4 As shown, in this embodiment, ResBlock includes a convolution layer 1 with a convolution kernel size of 3×3, a dual-path selective frequency module (DSFM) and a convolution layer 2 with a convolution kernel size of 3×3, which are cascaded in sequence; the input of ResBlock is added to the output of convolution layer 2 as the output of ResBlock;

[0090] SFBlock includes a convolution layer 3 with a convolution kernel size of 3×3, a multi-channel frequency domain attention module (MFAM) and a convolution layer 4, which are cascaded in sequence; the input of SFBlock is added to the output of convolution layer 4 as the output of SFBlock;

[0091] In this embodiment, the specific operation process of the dual-path selective frequency module is as follows:

[0092] The input features are split along the channel to obtain global features and window features;

[0093] Performing global average pooling (GAP) on the global feature to obtain the first low-frequency information, subtracting the first low-frequency information from the global feature to obtain the first high-frequency information (Invert), modulating the first low-frequency information and the first high-frequency information by a first learnable weight, and adding the modulated first low-frequency information and the first high-frequency information to obtain the global output feature;

[0094] The window features are divided into four window branch features by rearranging the data;

[0095] Performing global average pooling on the first window branch feature to obtain second low-frequency information, subtracting the second low-frequency information from the first window branch feature to obtain second high-frequency information, modulating the second low-frequency information and the second high-frequency information through a second learnable weight, and adding the modulated second low-frequency information and the second high-frequency information to obtain the first modulated feature;

[0096] Performing global average pooling on the second window branch feature to obtain the third low-frequency information, subtracting the third low-frequency information from the second window branch feature to obtain the third high-frequency information, modulating the third low-frequency information and the third high-frequency information through the third learnable weight, and adding the modulated third low-frequency information to the third high-frequency information to obtain the second modulated feature;

[0097] Performing global average pooling on the third window branch feature to obtain fourth low-frequency information, subtracting the fourth low-frequency information from the third window branch feature to obtain fourth high-frequency information, modulating the fourth low-frequency information and the fourth high-frequency information through a fourth learnable weight, and adding the modulated fourth low-frequency information to the fourth high-frequency information to obtain the third modulated feature;

[0098] Performing global average pooling on the fourth window branch feature to obtain fifth low-frequency information, subtracting the fifth low-frequency information from the fourth window branch feature to obtain fifth high-frequency information, modulating the fifth low-frequency information and the fifth high-frequency information through a fifth learnable weight, and adding the modulated fifth low-frequency information to the fifth high-frequency information to obtain a fourth modulated feature;

[0099] The first modulated feature, the second modulated feature, the third modulated feature and the fourth modulated feature are concatenated to obtain a window output feature;

[0100] Concatenate the global output features and the window output features to obtain the output features of the dual-path selective frequency module;

[0101] The first to fifth learnable weights are updated through training;

[0102] In this embodiment, the specific operation process of the multi-channel frequency domain attention module is as follows:

[0103] The dynamic weight W is obtained by the basic convolutional network for the input feature X, W = Tanh (BN (f 1×1 (GAP(X)))), where C represents the number of channels, H represents the height, W represents the width, GAP(·) represents the global average pooling operation, and f 1×1 (·) represents a 1×1 convolution operation, BN(·) represents a batch normalization operation, Tanh(·) represents a hyperbolic tangent function, k represents the range of the channel;

[0104] Will Reshape According to the reshaped dynamic weights, each channel in the input feature X is interactively adjusted through channel attention to obtain the adjusted feature X′, where the cth channel of the adjusted feature X′ is recorded as c represents the channel index, 1≤c≤C, W c，j represents the value of the cth row and jth column in the reshaped dynamic weight, offset(c, j) represents the relative offset from channel c to its adjacent channel j, X c+offset(c，j) Indicates the channel after offset;

[0105] The adjusted feature X′ is convolved once, twice, three times and four times with a convolution kernel size of 3×3 to obtain four high-frequency information F m , m∈{1,2,3,4};

[0106] Specifically, the adjusted feature X′ is subjected to a convolution operation with a convolution kernel size of 3×3 to obtain the first high-frequency information F 1 ;

[0107] For the first high frequency information F 1 Perform a convolution operation with a convolution kernel size of 3×3 to obtain the second high-frequency information F 2 ;

[0108] For the second high frequency information F 2 Perform a convolution operation with a convolution kernel size of 3×3 to obtain the third high-frequency information F 3 ;

[0109] For the third high frequency information F 3 Perform a convolution operation with a convolution kernel size of 3×3 to obtain the fourth high-frequency information F 4 ;

[0110] The four high-frequency information are added together and sequentially passed through the global average pooling operation and the fully connected operation (FC) to obtain the intermediate frequency attention weight;

[0111] The intermediate frequency attention weights are fully connected through four fully connected layers to obtain four preliminary frequency attention weights W m , that is, W 1 , W 2 , W 3 , W 4 ;

[0112] The four preliminary frequency attention weights are concatenated and normalized by the Softmax function, and the results of the normalization operation are split to obtain four final attention weights W m ′, that is, W 1 ′、W 2 ′、W 3 ′、W 4 ′;

[0113] Multiply the four high-frequency information by the corresponding final attention weights to obtain four multiplied results, namely F 1 W 1 ′、F 2 W 2 ′、F 3 W 3 ′、F 4 W 4 ′;

[0114] The four multiplication results are added together and a convolution operation with a convolution kernel size of 1×1 is performed to obtain the attention frequency map as the output of the multi-channel frequency domain attention module.

[0115] The present invention randomly selects 2 / 3 of the fuzzy-fogging images from the fuzzy-fogging data set to form a training set, and the remaining 1 / 3 of the fuzzy-fogging images to form a test set;

[0116] During the training process, cosine annealing is used to reduce the learning rate from 1e -4 Reduce to 1e -6 , the training batch size is set to 4;

[0117] The blur-haze restoration network is implemented on an NVIDIA GeForce RTX 4090 GPU using PyTorch. MACs are calculated based on a 256×256 patch size.

[0118] The present invention uses two evaluation indicators, Peak Signal to Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM), for evaluation. PSNR evaluates the image quality by comparing the pixel value difference between the original image and the restored image. The unit of PSNR is dB. The higher the PSNR value, the better the image quality and the closer the restoration effect is to the original image. SSIM evaluates the image quality by comparing the brightness, contrast and structural information of the image. The range of SSIM is 0-1. The larger the value, the higher the similarity of the two images, thus the better the image restoration effect.

[0119] The present invention trains a deblur-dehazing net (BHNet) in the blur-hazing dataset GoPro-haze to verify its restoration effect on blur-hazing images. In order to more objectively reflect the processing advantage of BHNet for blur-hazing degradation, multiple algorithms in the field of image restoration are selected for training in the same dataset, and the training batch size of all algorithms is set to 4. After the same number of training rounds, verification is carried out. The verification results are shown in Table 1. Compared with the MAXIM algorithm, the PSNR of the present invention is improved by 6.27 dB, and the SSIM is improved by 0.15; compared with the Restormer algorithm, the PSNR of the present invention is improved by 3.9 dB, and the SSIM is improved by 0.126.

[0120] Table 1 Comparison of image restoration evaluation indicators of different algorithms on the GoPro-haze dataset

[0121]

[0122] Since deblurring and dehazing are composite tasks of two image restoration tasks, the high degradation complexity leads to great difficulty in training. Therefore, the training is improved by transfer learning. Specifically, BHNet is first pre-trained with a fuzzy data set (Blur) or a foggy data set (Haze). After a certain number of training rounds, it is transferred and trained with the GoPro-haze data set. In addition, while ensuring that the total number of training rounds remains unchanged at 2500, different numbers of pre-training rounds are tried. The results are shown in Table 2. Compared with the PSNR of 26.70dB obtained by BHNet in Table 1, the use of fuzzy data set pre-training increases the PSNR by an average of 1.36dB, and the use of foggy data set pre-training increases the PSNR by an average of 0.48dB. It can be seen that the training effect can be effectively improved by single-task pre-training, and the improvement of the fuzzy data set is more significant. The reason is that the pre-training of the fuzzy data set can make the network pay more attention to the extraction of high-frequency information, which is conducive to subsequent training. Therefore, the public data set GoPro is used in the migration training of the present invention.

[0123] Table 2 The impact of pre-training methods on the results

[0124]

[0125]

[0126] In addition, in order to better reflect the role of transfer training in the GoPro-haze dataset, the Restormer method is also trained in the same way. The results are shown in Table 3. Compared with the classic image restoration algorithm Restormer, BHNet improves PSNR by 4.21dB and SSIM by 0.67.

[0127] Table 3 Comparison of the effects of different algorithms on whether to transfer training

[0128]

[0129] like Figure 5As shown, the first row of images are reference images, the second row of images are input images, and the PSNRs are 17.85, 15.57, 15.14, and 13.32 respectively. The third row of images are the result images obtained using the Restormer method (Restormer-pre) after transfer training, and the PSNRs are 20.69, 24.03, 26.97, and 23.34 respectively. The fourth row of images are the result images obtained using the SFNet method (SFNet-pre) after transfer training, and the PSNRs are 21.49, 25.99, 22.71, and 30.92 respectively. The fifth row of images are The results obtained using the BHNet method have PSNRs of 22.32, 25.14, 25.97, and 29.36, respectively. The sixth row of images are the results obtained using the transfer-trained BHNet method (BHNet-pre), with PSNRs of 29.28, 29.15, 30.49, and 33.01, respectively. The transfer-trained BHNet method can generate images with superior visual effects that are closer to the reference image on the GoPro-haze dataset compared to other methods. The reference image is the original image in the GoPro dataset, and the reference image is taken with a high-speed camera. In addition, BHNet is closer to the SFNet indicator based on frequency domain selection, which reflects the importance of considering frequency domain features in blur-haze tasks.

[0130] In the ablation experiment, we first verify the effectiveness of each module, then study the effect of each module application method, and finally, deeply study the design of DSFM to prove its effectiveness. All studies are verified by the GoPro-haze dataset, and the training method adopts the transfer training method mentioned above. The baseline network is obtained by removing MFAM and DSFM from BHNet.

[0131] As shown in Table 4, compared with the baseline network, DSFM and MFAM improve PSNR by 0.20dB and 0.73dB respectively, introducing only a small amount of computational burden; DSFM is only deployed at a single location at each scale, resulting in significant performance gains, proving the effectiveness of the multi-channel dynamic frequency selection mechanism under complex degradation conditions.

[0132] Table 4 Ablation study of each module

[0133]

[0134]

[0135] As shown in Table 5, the impact of the number of DSFMs in the BHNet model, where number represents the number of DSFM modules in each large module (i.e., encoder or decoder), using more DSFM can continuously improve PSNR from 27.78dB to 28.08dB, and improve SSIM from 0.920 to 0.923, while only introducing 0.02M parameters (Params); due to the small number of introduced parameters and low complexity, DSFM is inserted in each residual block in the large module for frequency learning.

[0136] Table 5 Ablation study on the number of DSFM

[0137]

[0138] The role of channel attention and the impact of the number of convolution operations in the MFAM module are studied. The channel attention is referred to as CA and the number of convolution operations is referred to as DFA.

[0139] CA is compared with two classic channel attention modules used in CBAM and SENet. The contrasting methods only use a single attention weight to refine each channel without absorbing signals from other channels. Table 6 shows that CA is significantly better than ordinary single-channel attention under the condition of comparable complexity. The reason is that there is different information in different channels, so that the attention weights of channel interactions can effectively promote the feature fusion of multiple channels.

[0140] Table 6 Comparison of CA and other channel attention modules.

[0141]

[0142] Different from SFNet, which directly separates high-frequency and low-frequency information through average pooling, DFA separates high-frequency information through convolution. Although it increases the complexity to a certain extent, it can extract deeper information through repeated convolution. The present invention extracts the intermediate features between BHNet and SFNet, such as Figure 6 As shown in the figure, the first row of images represents the reference image, the second row of images is the SFNet algorithm, and the third row of images is BHNet. BHNet can extract more high-frequency detail features. In addition, the relationship between the number of convolution operations and the image restoration effect is further studied. As shown in Table 7, with the increase of the number of convolution operations, the PSNR first increases and then decreases. The best effect is achieved when the number of convolution operations is 4. When the number of convolution operations is too large, BHNet pays too much attention to local information and ignores the global relationship, resulting in a decrease in image restoration quality.

[0143] Table 7 The impact of DFA on image restoration

[0144]

[0145] References in this patent:

[0146] References for the public dataset GoPro: Nah, S., Hyun Kim, T., & Mu Lee, K. (2017). Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3883-3891).

[0147] References for MAXIM: Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y (2022). Maxim: Multi-axis mlp for image processing. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp.5769-5780).

[0148] References for Restormer: Zamir, SW, Arora, A., Khan, S., Hayat, M., Khan, FS, & Yang, MH (2022). Restormer: Efficient transformer for high-resolution imagerestoration. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp.5728-5739).

[0149] SFNet references: Cui, Y., Tao, Y., Bing, Z., Ren, W., Gao, X., Cao, X., ...&Knoll, A. (2022, September).Selective frequency Network for image restoration.In TheEleventh International Conference on Learning Representations.

[0150] CBAM references: Woo, S., Park, J., Lee, JY, & Kweon, IS(2018).Cbam: Convolutional block attention module.In Proceedings of the Europeanconference on computer vision(ECCV)(pp.3-19).

[0151] SENet references: Hu, J., Shen, L, & Sun, G. (2018). Squeeze-and-excitation Networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141).

Claims

1. A method for restoring a blurred and foggy image based on multi-channel frequency domain modulation, comprising the following steps: Step ①, construct a fuzzy and foggy dataset, and randomly select some fuzzy and foggy images from the fuzzy and foggy dataset as a training set; Step ②, transfer training is performed on the blur and fog restoration network through the training set of the public data set GoPro and the training set obtained in step ① to obtain the trained blur and fog restoration network; Step ③, inputting the blurred and foggy image to be restored into the trained blurred and foggy restoration network to obtain the restored image; The fuzzy fog restoration network includes the first to sixth convolutional layers, three encoders with the same structure and function, three decoders with the same structure and function, and an output layer. The first convolutional layer is used to extract features from the input image to obtain first shallow features; The first encoder is used to extract the first shallow feature to obtain the first deep feature; The second convolutional layer is used to extract features from the cascaded first deep features and the input image to obtain second shallow features; The second encoder is used to extract the second shallow features to obtain the second deep features; The third convolutional layer is used to extract features from the cascaded second deep features and the input image to obtain third shallow features; The third encoder is used to extract the third shallow feature to obtain the third deep feature; The first decoder is used to perform a decoding operation on the third deep feature to obtain a first restored feature; The fourth convolutional layer is used to perform a convolution operation on the cascaded first recovery feature and the second deep feature to obtain a first adjustment feature; The second decoder is used to decode the first adjusted feature to obtain a second restored feature; The fifth convolutional layer is used to perform a convolution operation on the cascaded second restored features and the first deep features to obtain a second adjusted feature; The third decoder is used to decode the second adjusted feature to obtain a third restored feature; The sixth convolutional layer is used to extract the third restored features to obtain the final restored features; The output layer is used to add the input image and the final restored feature to obtain a restored image and output it; The encoder and the decoder are both provided with a ResBlock for enhancing frequency information and a SFBlock for extracting characteristic channel-related information and modulating frequency domain information.

2. According to claim 1, a method for restoring a blurred and fogged image based on multi-channel frequency domain modulation is characterized in that The specific operation process of step ① is as follows: According to the atmospheric fog model, random fog is added to each blurred image in the public dataset GoPro to obtain the corresponding blurred foggy images, and a blurred foggy dataset is constructed. 2 / 3 blurred foggy images are randomly selected from the blurred foggy dataset to form a training set. The atmospheric fog model is I(x)=J(x)t(x)+A(1-t(x)), where I(x) represents a blurred foggy image, J(x) represents a blurred image, A represents global atmospheric illumination, and t(x) represents transmittance. When the atmosphere is uniform, t(x)=e -βd(x) , β represents the scattering coefficient of the atmosphere, d(x) represents the depth of field, β ranges from 0.02 to 0.08, and A ranges from 0.4 to 0.

8.

3. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 1 is characterized in that The specific operation process of step ② is: preprocessing the training set of the public data set GoPro and the training set obtained in step ①, inputting the preprocessed training set of the public data set GoPro into the fuzzy fog restoration network, and using the Adam optimizer to iteratively optimize the fuzzy fog restoration network according to the loss function until it reaches half of the preset number of iterations, then stopping the iterative process to obtain the pre-trained fuzzy fog restoration network, and inputting the pre-processed training set obtained in step ① into the pre-trained fuzzy fog restoration network, and using the Adam optimizer to iteratively optimize the pre-trained fuzzy fog restoration network according to the loss function until it reaches half of the preset number of iterations, then stopping the iterative process to obtain the trained fuzzy fog restoration network.

4. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 3 is characterized in that The preprocessing is as follows: first, the size of each image in the training set is cropped to 256×256, and then each image is horizontally flipped with a probability of 0.5; The loss function is l = l c +λl f , where l c represents the spatial domain loss function, X represents the restored image output by the blurred and fogged restoration network, Y represents the corresponding reference image, i represents the index of the restored image, S represents the number of normalized elements, ||.||1 represents the absolute value error, and l f represents the frequency domain loss function, represents fast Fourier transform, λ represents hyperparameter, λ=0.1; The preset number of iterations is 2000-2500 rounds.

5. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 1 is characterized in that The convolution kernel sizes of the first, second, third and sixth convolution layers are 3×3, the convolution kernel sizes of the fourth and fifth convolution layers are 1×1, and the encoder and the decoder are both composed of L ResBlocks and one SFBlock.

6. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 5 is characterized in that L＝16。 7. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 5 is characterized in that The ResBlock includes two convolutional layers with a convolution kernel size of 3×3 for feature extraction and a dual-path selective frequency module; The specific operation process of the dual-path selective frequency module is as follows: The input features are segmented along the channel to obtain global features and window features; Performing global average pooling on the global feature to obtain first low-frequency information, subtracting the first low-frequency information from the global feature to obtain first high-frequency information, modulating the first low-frequency information and the first high-frequency information by a first learnable weight, and adding the modulated first low-frequency information to the first high-frequency information to obtain a global output feature; The window features are divided into four window branch features by data rearrangement; Performing global average pooling on the first window branch feature to obtain second low-frequency information, subtracting the second low-frequency information from the first window branch feature to obtain second high-frequency information, modulating the second low-frequency information and the second high-frequency information through a second learnable weight, and adding the modulated second low-frequency information and the second high-frequency information to obtain the first modulated feature; Performing global average pooling on the second window branch feature to obtain the third low-frequency information, subtracting the third low-frequency information from the second window branch feature to obtain the third high-frequency information, modulating the third low-frequency information and the third high-frequency information through the third learnable weight, and adding the modulated third low-frequency information to the third high-frequency information to obtain the second modulated feature; Performing global average pooling on the third window branch feature to obtain fourth low-frequency information, subtracting the fourth low-frequency information from the third window branch feature to obtain fourth high-frequency information, modulating the fourth low-frequency information and the fourth high-frequency information through a fourth learnable weight, and adding the modulated fourth low-frequency information to the fourth high-frequency information to obtain the third modulated feature; Performing global average pooling on the fourth window branch feature to obtain fifth low-frequency information, subtracting the fifth low-frequency information from the fourth window branch feature to obtain fifth high-frequency information, modulating the fifth low-frequency information and the fifth high-frequency information through a fifth learnable weight, and adding the modulated fifth low-frequency information to the fifth high-frequency information to obtain a fourth modulated feature; The first modulated feature, the second modulated feature, the third modulated feature and the fourth modulated feature are concatenated to obtain a window output feature; The global output features and the window output features are concatenated to obtain the output features of the dual-path selective frequency module.

8. The method for restoring a blurred and fogged image based on multi-channel frequency domain modulation according to claim 5 is characterized in that The SFBlock includes two convolutional layers with a convolution kernel size of 3×3 for feature extraction and a multi-channel frequency domain attention module; The specific operation process of the multi-channel frequency domain attention module is as follows: The dynamic weight W is obtained by the basic convolutional network for the input feature X, W = Tanh (BN (f 1×1 (GAP(X)))), where C represents the number of channels, H represents the height, W represents the width, GAP(·) represents the global average pooling operation, and f 1×1 (·) represents a 1×1 convolution operation, BN(·) represents a batch normalization operation, Tanh(·) represents a hyperbolic tangent function, k represents the range of the channel; Will Reshape According to the reshaped dynamic weights, each channel in the input feature X is interactively adjusted through channel attention to obtain the adjusted feature X′, where the cth channel of the adjusted feature X′ is recorded as c represents the channel index, 1≤c≤C, W c，j represents the value of the cth row and jth column in the reshaped dynamic weight, offset(c, j) represents the relative offset from channel c to its adjacent channel j, X c+offset(c,j) Indicates the channel after offset; The adjusted feature X′ is convolved once, twice, three times and four times with a convolution kernel size of 3×3 to obtain four high-frequency information F m , m∈{1,2,3,4}; The four high-frequency information are added together and sequentially passed through the global average pooling operation and the full connection operation to obtain the intermediate frequency attention weight; The intermediate frequency attention weights are fully connected through four fully connected layers to obtain four preliminary frequency attention weights W m ; The four preliminary frequency attention weights are concatenated and normalized by the Softmax function, and the results after the normalization operation are segmented to obtain four final attention weights W m ′; Multiply the four high-frequency information with the corresponding final attention weights respectively to obtain four multiplied results; The four multiplication results are added together and a convolution operation with a convolution kernel size of 1×1 is performed to obtain the attention frequency map as the output of the multi-channel frequency domain attention module.

Images